Jared Spool: Hello, everyone. Welcome to another episode of Spoolcast. I have to tell you I'm really excited. We have Julie Zhuo with us today. She is the Product Design Manager at Facebook. She has a whole team of folks who are responsible for many of the changes that you see at Facebook.

I had a chance to see her speak last year at An Event Apart. She blew the audience away, bringing out all these examples of how Facebook's design process is currently changing. We have a chance today to talk to her about that and she's going to be at our UIE Web App Masters Tour in San Diego at the end of March. And you'll get a chance to hear her there, too. Julie, welcome.
Julie Zhou: Hey, Jared. Really excited to be here and excited to go to San Diego and talk to a bunch of folks about Facebook.
Jared: Yeah. Well it's very cool. I think what's going to be interesting is listening to the changes that you guys are undergoing. Facebook grew up in this world where they would just put out a brand new design, everybody would get it on the same day. Then, 24 hours later, there would be a group somewhere around 40 percent of the Facebook users complaining that they hate the new design and petitioning that it should be changed back, but you guys are moving away from that.
Julie: Yeah. I think really early on, when we were a college site, we got used to the idea that we'd come up with something new and then we write code and stay up all night and then three days later ship it. All of our college users would be so excited to just see what was new; what was different.

They gave us a lot of suggestions for what they wanted to see and every time something changed they were really, really receptive to that change. Because of our culture of really moving fast and of this idea that "Oh, hey, if we build anything our users will be excited to use it," we got into this mentality that that's the way that we should launch stuff.

We really prided ourselves on moving that quickly. But we did discover over time our user base grew from eight million college users to 400 million users worldwide today. Encompassing everyone from the really tech savvy young person, the teenager, to mothers, grandmothers and all sorts of other people who may not be as comfortable using a computer.

So as a result, we've learned over time and time again that launching something by just pushing a big switch and changing that for everyone and just expecting people to love it is not really the way to go anymore. Nowadays, we still pride ourselves on moving fast and not being scared to really take the risky moves that we think are going to put us in the right direction in the long run, but we're a lot more careful about it.

The ways in which we're more careful mostly have to do with us doing our due diligence and gathering data beforehand. Every time we come out with a drastic new redesign or big change that we think might make some users uncomfortable, or present them something that they're not used to, we make sure to do a lot of user testing. We bring users in at every stage of the process, showing them everything from really rough paper prototype mocks to really basically the product is built. We just want people to come in and sit down and use it and get their reactions and ask them some questions about how the new change made them feel.

Additionally, we've also been doing a lot more A/B testing. It used to be we'd just flip the switch, it goes from zero percent to 100 percent of users who get the change. Now it's more of a dial that we twist slowly. We'll launch it to one percent of users.

We'll monitor all the feedback, we'll listen to what they're saying and we'll even check the metrics to see that they're still sharing and they're able to communicate and that one feature hasn't just completely disappeared off their radar. Therefore, we're seeing like a huge tank in that metric.

We monitor those stats really closely and that tells us a lot about the story of how users are finding our new redesign. Only when we're pretty confident that things are still going well, that people are finding value from it, do we slowly crank that dial up.
Jared: I'm interested in the sort of change that you guys have undergone in the process of moving to this new "test everything first before it goes out." A lot of organizations are dealing with changing that mentality, where they're realizing that they can't just put something new out and change on all their existing users without getting some flak. They want to make sure that everything is an improvement, everything is going better.

Can you talk about sort of the process that you guys went through to do this? I mean, I'm assuming it wasn't an overnight thing that all of a sudden you're doing all this A/B testing. All of a sudden you're doing all this usability testing. Which came first and how did you gear that up?
Julie: I think a funny story we always like to tell each other and the company is the story of when we launched Newsfeed. So we launched Newsfeed in I think September or August of 2006. It was about half a year after I started at the company.

Internally we were really, really excited about the idea of Newsfeed. Keep in mind, back then no one really -- there was not really the concept that you would login to a social service and be able to see what your friends are doing. Right?

It almost seems obvious today because if you go anywhere, that's sort of how you consume information about what people are broadcasting. Back in the day, our home page was really simple. It basically said "Welcome, Julie, to Facebook. And by the way, you have 12 new messages and 29 pokes." That was the entirety of the home page, so no real content there.

We talked about the idea of just having this place where you can login and just see what people are doing. Otherwise, if your friend uploaded a photo album or your other friend went to this great event, you wouldn't know about it. You'd have to go and check their profiles individually in order to see the fact that they had updated their profile or done something interesting.

We're at the company really, really excited about this feature. We've been working for months to get it ready. Then one night we basically flipped it on and then we waited and we waited for people to tell us how great it was, and how exciting.

About an hour later we had about 10,000 pieces of feedback, and only one person liked it. Everyone else was basically like, "Oh my God, this is creepy. This is, you know, invading my privacy. This is like, I can stalk my friends. Like, what's going on? Why did you guys ruin my Facebook experience? I can never come back." It was a really, really strong negative feedback, but two hours later, there was like a protest group outside our office already complaining that we need to get rid of the feature or else they would boycott Facebook.
Jared: People actually showed up at your offices?
Julie: Yes. Yes. People showed up over this.
Jared: Oh, wow.
Julie: The security sent an email to everyone saying, "Hey, if you're going to leave, take the back door because there's people out in the front." You know, and a day or two later there were news cameras outside. This is really big news because at the time I think we had maybe eight or 10 million users and about a million of them joined the group "I hate Facebook Newsfeed, take it away." That's like one-tenth of our user base at the time and that's a huge number.
Jared: Right. I remember it making the New York Times and the Wall Street Journal, like front page articles.
Julie: Right, right. I think that really opened our eyes to the fact that, "Oh gee, we need to think really carefully about launching stuff." It's not to say that if we would have done testing and gotten this user feedback earlier, we would have just said, "We're not going to launch it."

We really believed in the idea of Newsfeed. We believed in its value. But I think had we done a lot of the user testing and sort of gotten a little bit of that reaction earlier, we could have really mitigated the problem.

One example is a lot of people complained about it because of privacy concerns, right? Two or three days later we launched a control panel, basically, a settings page for people to turn off certain things that they didn't want to show up in Newsfeed. And I think that little settings page really helped ease a lot of people's concerns about the Newsfeed.

It helped them think that they had more control over what they were sharing. I think if we had actually gotten all that information beforehand, we could have done that preemptively, prior to launching. I think that maybe the outcry around that feature wouldn't have been so great. This is something that we learned a lesson from, and even then afterwards with every subsequent redesign.

We did have a lot of people -- in general people won't like change, right? And it makes sense. It's like you're used to your desk being a certain way, right? You write letters, you look at photos. And if someone came and rearranged everything at your desk, even if they added a bunch of cool new things, your first reaction is "Why did you change everything? Now I have to relearn how to get stuff done on Facebook and how to find photos and how to find my messages."

We are cognizant of the fact that every time we make a change, the initial user reaction is going to be a little bit negative. That's why listening to feedback really matters. If all of the feedback is basically, "I don't like this change because it's different," then that's maybe a sentiment that will go away once people are used to it.

But if the feedback is, "I don't like this change because now I can't find my applications," or "I can't find chat," or "I can't find messages." Then that's like a real wake-up call for us that we really need to examine this change and see if we've regressed in making it easier and better for users.
Jared: So what was the process of getting usability testing as part of your design practice? Some people have usability testing at the end. You do all the designs and then you bring in a handful of users and you actually see how they use the designs and maybe make some small changes and then release it. Then some people have more participation from the user population earlier in the design process. What did you guys start with?
Julie: I think one of the earliest things that we did was just what you mentioned. When we were getting ready to launch a new redesign or launch a new feature, we'll bring people in. We already have a working version of the site. We can actually have them walk through it as if they were using it at home and witness their reactions, see where they're getting tripped up in the process.

But more and more with some of our bigger releases, it's been really important to do a lot of the user testing way, way earlier in the process. I'll give you an example. Last December we launched a change to privacy and so when you logged into Facebook one day, you got a little privacy dialog that said, "Hey Facebook is making some changes to privacy. Please revisit your privacy settings."

That thing is not going to take that long to build. Right? It doesn't take that long to design, as far as it's just one little dialog. But the process for us getting to that final point was months and months, because we knew privacy is such a sensitive topic for people that we wanted to be absolutely sure that what we were doing people would be comfortable with. It was the right thing to do.

I think maybe four or five months prior to our launch, we were already bringing people in. We hadn't even started building the pod. It wasn't really even designed. We were just showing them sort of a little text dialog with the language that we were going to use and with a lot of different options for how we would present this messaging to them.

These are like paper, low-fi prototypes, you know? Nowhere near what the final product will be. But prior to us even building and getting nice mocks from everyone, we already had at least five sessions with a bunch of users testing about 30 different versions of the language and the messaging for this dialog.

Then after we built it and we actually built like three or four different versions, and we brought another couple dozen users in to -- now we have fewer versions, but we tested with more users. We wanted to make sure that this made sense the young tech savvy user, and the grandmother at home who just wants to communicate with her family and doesn't want the rest of the world to see her family photos.
Jared: It's been a long time sort of moving and changing the way you do your process and trying out new ways of using usability testing. Getting it earlier and earlier into the design process. That's really been working well, it sounds like.
Julie: Yeah. It's definitely still a little bit of attention for us because we want to make sure we're still moving fast. That we're a company that's able to really quickly innovate and get stuff out the door. That's just a really important part of our engineering culture, but also marrying that with the whole user experience process.

So we don't ever launch anything out into the wild, blind, without having a sense of what people are going to say about it and how it's going to do in terms of improving the user experience.
Jared: It's interesting to me that a lot of what you worked on in the privacy stuff was messaging and copy. That it was getting that wording just right was a critical piece of this, which a lot of folks don't think of as part of the design process, interestingly enough.
Julie: Oh yeah. Yeah. We've had a lot of discussions about content and the role that content plays in the design process. Frankly, it's really important in many cases. Especially with something like the privacy wizard where you're messaging a change to the user.

It's really critical the way that it sounds, you know, that we're not putting too much text in front of them because users tend to have a tendency to just "do" rather than "read" all the time. Just finding the balance between design that's intuitive and copy that has the right messaging and the right tone and that can convey what it is we want to express to the users. It's really critical.
Jared: And the team that's putting this together. Is it just visual designers or do you have a multidisciplinary team that's working behind all this stuff?
Julie: Our design team consists of product designers, communication designers, user experience researchers, and content strategists. Product designers are basically designers that work on the core Facebook.com product. That's all of the features on Facebook; photos, or news feed, or platform. That's actually most of our design team, is product designers.

We tend to hire fairly technical product designers. Our product designers sort of span the range of -- each product designer will do the visual design, will do interaction design, will be really involved in the product conversations. We call that product design. Then also be able to make small tweaks to the CSS and the HTML, and occasionally if something doesn't look right, they'll just be able to go in and change it themselves.

Our communication designers work on Facebook branding and messaging. And whenever we launch new features to make sure that users understand the benefits and understand the changes that we're making. Also, every time we host a conference, or we have to make some recruiting posters, or other sort of material that conveys our brand, they're the ones working on that.

We have a small team of user experience researchers. They're working really closely with the product designers, at every stage of the product development process. Being looped in on the conversations about when we should start testing, what kind of tests that we should run with users? What are the open questions that we have about a proposal or a design that we're considering?

Then our Content Strategist also works really closely -- they're looped in really early in the product development process in order to really understand what we're going after and how content might fit into the final design.
Jared: So that's a pretty big team that's working on this stuff. What sort of tools are you using to communicate what you're learning and how things are working, and not working, amongst yourselves? Do you have like a group wiki, or do you use a lot of email?
Julie: This is also one of those things where back then we had four or five designers. It was a non-issue. Everyone sat together. You could just turn around and see what was going on, on people's monitors and have a conversation that way.

Now, the full design user experience team is about 30 people, and we have -- as our organization grew -- had to deal with how do we effectively communicate? On the other side of that, how do we give each other feedback? Because a really big part of design is the feedback process. That's sort of how we make sure that, even though you have 12 different designers working on 12 different parts of Facebook, that the final design still looks unified and consistent.

A lot of the tools that we use are actually basically homemade. One example is a tool that we use called Pixel Cloud. This is something that one of our designers, Alex Roche, just decided one day, "Gee, I don't know what's going on with what a bunch of designers are working on. I wish I could just sort of see their in-progress mocks. Every day, when they're iterating, I just want to see that work."

He basically built a tool called Pixel Could. It has a web component, a download component. If you go in and install the software, then, as you're tinkling around in Photoshop and you basically finish an iteration or you see something interesting, you can do a screen grA/B of that, and then use a quick keyboard shortcut, and it will automatically post it into the Cloud.

You write a few comments about it. You explain what you're working on. You explain, this is really blue sky, but I'm looking for high-level feedback ideas. Then it shows up and other designers are regularly browsing through Pixel Cloud, and not just designers. Anyone at the company can go through and check out Pixel Cloud and see what the design team's working on.

A couple times a day, we'll go check out Pixel Cloud just to see the latest and greatest mocks. Then we can write our comments. We can give feedback. That's actually been a really, really effective tool for designers to share our work. That's something that I think that at least 20 some posts get added to Pixel Cloud, every day. There's always a lot of activity going on over there.
Jared: So it sounds like it's an internal, Flickr-like, image sharing, commenting, discussion thing.
Julie: Yeah, definitely. Pixel Cloud has been really integral to the way that we communicate within our team. But, in addition, we've also established what we call working groups. These are small teams, usually two to three designers, that oversee a specific area of focus.

We'll have a working group for visual design and we'll have a working group for interaction design. Part of the working groups' mantra is that they're going to hold office hours, for about three or four hours every week, where all three members of the working group are going to be in a room. They're gong to be doing nothing but be available to give feedback to other people.

Part of visual standards working group is the idea of consistency. We're all using common components among our designs. Everyone is working with buttons. Everyone is working with tabs. Everyone is working with navigation and menu. We want to make sure that not everyone is redesigning all of the core basic elements. That we're all using the tA/B style that feels the most Facebook. That this is a particular tA/B style that we've decided to use.

The visual standards working group is working on all of these common components. In addition, if you go to office hours and present your work, they can tell you, "Oh, this isn't quite the convention that we sort of see Facebook moving in." They'll give little visual pieces of feedback about how to basically pull the design more in line with the Facebook aesthetic.

Working groups are another way in which people are sharing and receiving really valuable and critical feedback from each other. It's another thing, Facebook, our design team, doesn't have a Creative Director. For the role of the Creative Director, which is to ensure quality and consistency, we're trying to do that ourselves using these tools that we've built and the concept of the working groups to maintain that.
Jared: Well, it's interesting that you don't have a Creative Director, but there's a very consistent feel across the whole site. If there isn't one person who is sort of playing the role of Director on this, how do you guys manage to get that really nice consistency across the design?
Julie: Yeah, I'm glad you think our site is consistent [laughs] .
Jared: Do you not think that? Do you think it has got all sorts of wacky inconsistencies?
Julie: I think, for those of us who are basically looking at every aspect of the site, we definitely feel like there are places where things are inconsistent and where we could be doing a better job. I mean, obviously, this is a lot better than a year or two ago when we first realized this was a really big problem.

We had about five different designs of tabs, across Facebook, just because every designer would do their own. In some cases, it would be really similar, but maybe a few pixels off here and there, in terms of spacing or border colors. Small stuff like that, but called out to us that we really needed to do a better job of standardizing these components, and on talking to each other.

Essentially, because we don't have a Creative Director, it's so important for us to talk to each other. And to really be involved in each other's work, so that we know what's going on. This is also one of the motivating factors for the working groups, right, is that consistency.

If we can get everyone to go to these two or three people, then these two or three people could essentially be -- the visual standards working group could be the ones that say this is consistent or this isn't consistent. Basically, point that out. Point out any issues that they might see.
Jared: It's neat, I think, that you guys have this way of tracking inconsistencies and then you're constantly pushing forward. You mentioned that you're using A/B testing. What's been the history of that at the company? Is that, again, something that sort of picked up after the whole news feed thing? And how did you start? And how has it changed once you got started?
Julie: Yeah, so definitely after Newsfeed -- I think prior to Newsfeed, we really weren't doing that much with user testing at all or A/B testing. I think the group at Facebook that really spearheaded the idea of A/B testing was the growth team.

About two years back we thought, what is the number one goal for the company? For everyone, it was basically growth, right? We're not going to get anywhere if we can't demonstrate the value of Facebook to new users.

Because Facebook's a social network, if you have no friends on it, the service is essentially useless. There's no reason you would really go back. Making sure that people had their friends on Facebook was our number one priority, so we came up with the team that would be dedicated solely to this fact.

This is sort of a very broad problem, it's very ambiguous. But the idea is, how do we get users onto Facebook? And not just get them to register, but how do we get them engaged? How do we get them to learn how to use Facebook and basically become a user that understands its value and keeps logging in every day?

The team, because it's such a broad problem and really we had no idea -- we had some guesses. We thought "Well, of course we should look at a registration flow. First we should look at our new user experience." What this team really did was really try to understand that core question -- how does a user come to Facebook and get engaged?

A lot of the ways in which they did that was through A/B testing, and so you'd have a hypothesis like "Oh, maybe it's the registration flow, right? Maybe if we made that a lot simpler or whatever, then a lot more people would be coming." So it would have like different versions of this registration flow, and they'd put it all out there and A/B test and see which one did the best and which one moved the needle.

Then they would say, oh well maybe it's not... They improved the registration flow, but now maybe we need to focus on the new user experience. Or we need to focus on people inviting their friends. Or maybe we need to focus on showing them features like photos.

People have a lot of ideas for what would get a person engaged on Facebook, and the way to test these ideas was just to run a huge number of A/B tests and try to see which aspects of improvement would really get more people to be engaged in Facebook. Part of that, too, was A/B testing like, "Hey if someone hasn't come back to Facebook for a while, maybe we'll send them an email. What should this email say? Well maybe the email should show them stuff that's been going on with their friends, or maybe the email should just be a simple text block that explains we miss you, come back." And people had a lot of ideas about that.

The growth team would just say, "OK we don't want to just debate this. Let's just put it out there and A/B test it and try it all." That method I think worked really, really well because I think at this point the growth team has a really good understanding of all of the data that they've collected and of what really gets people to join Facebook.

We've tried to adopt that in other parts of the features -- other products as well are trying to model what the growth team did. We usually do A/B testing for smaller features and small tweaks. Generally if we do a home page redesign, we won't have like two different versions of the home page redesign and A/B test both because that's just really expensive to do.

But it works really well for trying out small little features and we have a little bit of a box. You know, what would be the best thing to put in this box to increase engagement? Like, those were the kind of things that we're constantly A/B testing.
Jared: And the variables, the needles you're trying to move, how do you come up with those?
Julie: For every team that's a little bit different. That's actually, prior to us starting any project, we try to really sit down as a team and figure out what exactly are we trying to do here. What's the point of working on this project? For growth, it was always easy because you had a number, right? You could say this is our active user number, and we want to make it grow 50 percent or 100 percent or whatever. Right?

That's why I think A/B testing was a really natural model for the growth team, because everything that was tested, you can always then go back and look at that growth number and see how it performs. For other features, it's a little bit more ambiguous in terms of what exactly we're measuring. More and more we're trying to really nail that down.

If we make improvements to photos we might say, the metric might be, the number of photos that people see every day. The number of albums that they're clicking through or the number of uploads that are being made every day. These are the engagement metrics that we'll try to define for every feature or product improvement prior to us getting really hands down.

Us focusing on the goal really helps this team focus. If people suggest ideas, sometimes it can be hard to really evaluate. Well, is this a good idea or is this something we shouldn't really talk about at this point. Having that goal, having the team sort of centered around a goal is what sort of helps us make decisions about the right things to focus on.

Not everything can be quantified by numbers. Because, you know, definitely a lot of user sentiment, or how much they love something, or how much it delights them. These are the things that are much harder to measure.

That's why you still need the usability testing, you still need to bring people into the lab and get their reactions and see them walk through things. More and more I think it helps us focus if we do have defined metrics and we have a really concrete set of goals that any project is trying to accomplish.
Jared: Well this sounds very, very interesting. I'm looking forward to your presentation at the web app masters tour because you're going to show us examples of things that you've changed and some things that didn't work the way you expected and stuff like that.
Julie: Definitely. Lots of examples, lots of stories. I love sort of talking about the things that we thought and the things that didn't go right and the things that did go right. And so it's going to be a lot of fun.
Jared: Well, I'm very happy that you spent the time talking to us. It's been really interesting hearing how you guys work inside. From the outside it's been a little mysterious, you know. The folks at Facebook, they just do stuff and it just happens. Sometimes it makes us very happy and sometimes it's not as pleasing as we'd like. It's really neat to hear this sort of inside view and what you guys are doing and how you're getting better at this. So thank you for taking the time.
Julie: Oh, yeah. I'm really happy to come here and talk to the world about Facebook.
Jared: And I want to thank everyone again for listening to us. It's always a pleasure and you'll be able to hear Julie at the UIE Web App Masters Tour on March 23rd and 24th in San Diego where she'll be going through a lot of her designs and we're very happy that she's going to be joining us there. So please go to UIEtour.com to find out more about the UIE Web App Masters Tour. Once again, I want to thank you for taking the time. Of course thank you all for encouraging our behavior. We'll talk to you next time.