April 14th, 2011
When you have billions of clicks in your analytics log files, how do you pick out the right ones for making decisions? When do you trust your own well-honed sense of good design and when do you look to what your users are telling you?
At the first stop of the Web App Masters Tour in Philadelphia, Julie Zhuo shared Facebook’s approach to data-informed versus data-driven design decisions. Fortunately for us, Luke Wroblewski took fantastic notes about Julie’s presentation and is allowing us to share them with you. Luke originally posted these notes on his blog in March.
Without further ado, here is Luke’s notes on Julie’s presentation.
Facebook believes in small teams that can act like a start-up. The team on the ground needs be the most knowledgeable about what data to use on their project. Each product usually has a single designer assigned to it. There are only 20 designers and 5 researchers in the company.
Using Quantitative Data
- Facebook uses quantitative data to inform decisions, help understand how people are using the product, and how they can optimize key flows further.
- The company kind of got into the data game late. From 2004 on, the company just kept adding feature after feature. They never took a step back and looked to see what was successful. Some features were doing a lot better than others.
- It wasn’t until 2009 that Facebook went back and updated their Photos service. The last they touched it was in 2006. Photo uploading was hard to do. Facebook used a Java uploader that was built by an outside firm.
- When the team redesigned the photo uploader, they built a solution that required the ability to install a plugin one time and from then on photos could be uploaded. This implementation took about 4 months to build.
- The new photo uploader was rolled out to a small % of users. When they looked at the data, only 34% of users that started were successfully able to upload photos. It was the install plug-in step that was tripping people up. Half of the people dropped off at the install step.
- The team decided they needed to go back to what people were familiar with. They had to throw away months of work and instead stick with what people knew. The second iteration of photo upload made use of native OS controls.
- In the second redesign, of all the people who hit the upload photo button, 87% made it past the create album form, 57% selected photos, 52% make it to upload step, but only 45% of people actually uploaded photos.
- The more standard design increased uploads from 34% to 45% but the team thought they could do better.
- They found 85% of people were only selecting one photo at a time -probably as a result of the native dialog. People needed to know how to use the OS control to select multiple images.
- The Facebook team designed an interstitial to educate users about how to select multiple photos in the OS dialog. Though they usually avoided interstitials, it was important enough to try.
- With the interstitial tip in place, the number of users uploading only 1 photo dropped 40%. Photos uploaded per attempt increased from 3 to 11.
- Another example of using quantitative data to inform design was the “composer” –where people publish status, links, etc. For the composer, the team A/B tested explicit actions for: photos, questions, links and status updates. In this design, Status updates decreased 1%, photo uploading went up 1.5%. But there are way more status updates than photos. So the offset didn’t pay off.
- The second idea they tried was to show a user’s last status update at the top. The idea was if you see an old update you might want to update it. This approach didn’t have any effect on updates.
- Another option included links and an exposed status box. This performed better but only marginally.
- After spending a lot of time tweaking the composer and not getting a lot of results, the team finally stopped working on the problem and moved on to something where more gains could be realized.
Using Qualitative Data
- Quantitative data points to insights. Qualitative data can be used to optimize and sanity check flows.
- Facebook’s first user researcher joined in 2007. In the early days, they tested people going through flow. But because employees at Facebook use the product so much, they spot issues before usability sessions and fix them. As a result this type of qualitative research is not as valuable to them.
- Instead the user research team focused on bigger picture issues.
- Recently, they studied identity and what people feel comfortable sharing through ethnographic studies. Also they studied local businesses to tailor pages product to these smaller businesses.
- The end results of qualitative research are insights on how people feel and what they think. This funnels into higher-level strategy.
Evaluating Design with Metrics
- Some projects at Facbeook use data to judge their success. Not all projects are judged this way but the growth team is an example of a completely data driven team. The growth team was created in 2007 and was the first example of data driven design at Facebook.
- In 2007, Facebook opened up to the world (beyond colleges). A team was formed to tackle growth. A social network is valuable if people you know are on it. Otherwise, you don’t know what it’s for.
- Growth team looked at registration, metrics associated with people that were coming back, etc.
- A simple visual design change on registration page resulted in 9M more people per year joining. 3% increase. For growth, this is a big deal. Deactivation page. Reduced deactivations by 7% (a million more users a year).
- A few months ago, Facebook created a team called engagement –modeled after the growth team. Thought it would operate the same way. First tried quantifying success by looking at number of read/writes. One of the early ideas was comment liking. Saw an increase of writes in the system 7% up when this feature was rolled out.
- Though writes went up 7% it was the same people creating writes. 85% of reads/writes of Facebooks are generated by 20% of users. Same people were doing the activity. Had to go back and reset metrics for the engagement team.
- Current metric for engagement at Facebook: L6/7 number of users that come back to Facebook 6 out of 7 days a week.
- Facebook is wary of being too data driven. Data alone might look good but could have a bad impact on brand.
- It’s difficult for metrics alone to represent what you are trying to build with a company. Decisions at Facebook have to balanced between: qualitative data, quantitative data, strategy, user (individual interests), and network interests, competition, regulatory bodes, and business interests.
- You need to think about all these things as you design.
- Quantitative data helps understand how people use products, how to optimize flows, and if products are successful.
- Qualitative data helps understand user intents and feelings in order to focus strategy.
- Data is one small piece of the pie and lots more comes into the decision-making process.
- True innovation does not come from being lead by numbers.
- When launched, the Facebook newsfeed had an overwhelmingly negative response from users. Metrics plummeted the day after the launch as people were not coming to the site. The team made some modifications to give users more control but never got rid of the idea even though metrics took a hit at first. Now the newsfeed is hugely successful.
- Facebook learned how to take risks and not be surprised by reactions to new products: do research & anticipate what people will do.
- When Facebok was a college site, people loved all the features they released. Now the company needs to communicate why a change is good. They give people a choice to opt in now so a new design is not a shock.
- The greatest risk is taking no risk. Small tweaks are safe. Big ideas are what leads to innovation.
Hear how Julie and her team incorporate data analysis into their design decisions, plus 8 other Masters at the Web App Masters Tour in Seattle or Minneapolis. Other topics at the tour include data visualization, mobile design and strategy, and process best practices.Tweet