Now I look at the basic usability test protocol like it’s a column of wet clay, ready to be molded into exactly the research instrument we need. Over the years, we’ve invented, borrowed, and stolen different variations, all to help us better understand our users and what we’re designing.

In today’s UIEtips, I talk about five of our favorite variations. You’ll see how we bend and twist basic techniques to discover new things about what we’re designing.

Read the article: Bending the Protocols – Useful Variations on Usability Tests

What are your favorite variations on usability testing? Tell us below what you’re doing.

In our research, we found successful design teams have each team member spend a minimum of two hours every six weeks watch real users interacting with either their design or a competitor’s design. The most successful teams have even more frequent exposure hours.

When team members watch someone use the design, several things happen. First, they gain empathy with that person — empathy that makes them sensitive to frustrations and delights the design imparts. That empathy is critical for setting design priorities, as we try to eliminate those frustrations and amplify the delights.

Second, the team picks up the culture of use. They learn the language the users use. They learn how users approach different parts of the design. They learn the goals of the users, and how the design fits into the users’ daily life.

Third, the team develops a design language to describe the differences between good and bad. Having the real experiences of real users as a common understanding breaks each team member of always referring to their own experiences. Instead of saying, “this is how I’d use it,” they can now say, “this is how Mary, who we saw last week, might use it.”

Fourth, because the exposure happens frequently over a long period of time, the team members see how their design attempts are working. It creates a feedback loop in their design process, where they learn when their design changes created the improved user experience they were seeking.

Innovation happens when you add value for the user. Teams with more exposure to how real people use their designs can more easily see opportunities for innovation. They can try new ideas to whittle away the users’ frustration and see how those ideas pan out. This makes them smarter and more informed, so they make better decisions in the long run.

We had been working on a client project, helping their team redesign their checkout process with some new user research and design techniques.

As we were watching seasoned shoppers buy products in the lab, we noticed that people were getting stuck at a screen right before the checkout process, where they had to authenticate their account. Repeat customers couldn’t remember their user ID and password combination. The site used an email address as the user ID, but many of our repeat customers had set up their accounts years before and couldn’t remember which of their email addresses was the ID.

In the lab, the customers who couldn’t authenticate would either give up or request a password reset. However, the password reset required they remember which email address was their ID, which many couldn’t do. We witnessed a remarkable number of abandonments on the password reset screen.

When the password reset was successful, the customer had to go to their email client, find the reset message (often lost in a spam folder), and click on a link in the email. In the lab, we observed that this was a complex process.

All of this led us to ask if this was only happening in the artificial environment of the lab, because we watching users in our space, not using their own machine. We set out to look at the site’s analytics to see if there were clues to this behavior happening in the real world.

The first thing we asked the analytics team was what percentage of visitors to the authentication page ended up on the reset password request screen. Turns out, they had never instrumented either page. We had to wait three weeks while they instrumented it and we collected a reasonable sample size.

We learned a substantial percentage of customers were requesting password reset, approximately 40%. Two out of every five users was getting stuck and needing their password to be reset.

We then wondered what percentage of those people actually came back to finish the transaction after the reset. Again, we discovered the analytics team hadn’t instrumented the return from the reset. That was another three week delay.

We learned that fewer than 25% of the resets were executed — the user clicked on the reset link and returned to the site. Of those who did execute it, fewer than 20% finished their purchases.

A little math and we could calculate out the amount of revenue being abandoned in the carts by all the people who couldn’t authenticate. That’s where the $300,000,000/year number came from.

Once the team implemented a guest purchase capability (which didn’t require authentication to start the checkout sequence), they saw an immediate jump in sales increase of about $6,000,000 in the first week, which remained constant. Password reset requests dropped by about 80% in that first week and remained constant too.

Authentication pages are usually owned by a different group in the company. In this case, they were owned by IT. IT didn’t have the foresight to instrument these pages.

Until we did this research and asked these questions, nobody knew how many people were dropping off at authentication. Because authentication was between the shopping-cart and the Enter-your-shipping-info pages, everyone thought they were getting a much lower percentage of users clicking the Checkout button than they really were. The site had a huge abandonment on authentication that heretofore had gone undetected.

Analytics only work when we know that they are measuring everything correctly. Working with clients, we regularly discover they aren’t capturing the entire picture, leaving out critical information.

In the next UIE Virtual Seminar, Rich Rutter gets you started with JQuery—assuming no prior knowledge—and shows you lots of examples, hints, and tricks. Just 5 minutes into this seminar, you’ll see JQuery in action and have something you can use in your own wireframes.

Employ Simple Show and Hide Techniques

The essence of JQuery is to find something and do something to it. This technique easily shows different page states so your team and test participants can “do things” to your design.

• See, step-by-step, how to put this simple, yet useful example of JQuery in action
• Use modules and plug-ins to make your design to do simple things, without worrying about the performance of production code

Toggle Wireframe Annotations

Add notes to your interactive design.

• Turn your comments on or off depending on who’s viewing your design
• Add lists, comments, or direction for developers and others who need to work with your design

Fake Simple Ajax Interactions

Without creating production level code, get your design to quickly and easily do its thing—click something and change occurs—for your developer or client.

• Replicate what happens when you click something like a “favorite button”
• Fill in all the steps of an Ajax interaction such as a slight delay or adding different page states on a single page

Get Started with JQuery UI Widgets

Rich will introduce a library with options and widgets that you can easily put in place. In many cases you’ll see how to simulate what the full interaction could be.

• Explore modal dialogues, an intrusive piece of interaction and a good example of something you want to test: Do I really need a modal, or is a link better?
• Get more examples: Prototyping calendars, lightboxes, and more.

Rich will show you the power of combining discreet interactions together with a complex interaction.

Regardless of your JavaScript experience, this seminar will be a great way to start using JQuery and take your interactive skills to the next level. JQuery gives us a clean, interactive feel, and can be the difference between a slick design and something annoying or disruptive. It brings rich interactivity to your HTML and CSS3.

Rich will incorporate complex interaction examples along with providing excellent sources of documentation and tutorials for your toolbox. The seminar will keep theory to the bare minimum and focus on getting you started with practical takeaways you can use straight away.

The real power in what you’ll learn is getting very close to a final look and feel of your intended design with just a bit of effort and without having to build the whole application. Get over the initial hurdle of the JQuery learning curve and gain momentum in your design process. Join us for JQuery for UX Designers.

Outsourcing your user research work is like outsourcing your vacation.
It gets the job done, but probably won’t have the effects you were seeking.

I usually say this when someone is asking me to do their user research for them. This is something we did quite a bit in the early days of UIE, but don’t do any longer.

Usually, they are asking us to do this to save time, because they don’t have trained folks, or because they are afraid of bias. All these reasons are rational, but there are better ways to deal with them than hiring someone else to do the research on their behalf.

As I said, I founded UIE as a company to do just this. I felt the rational reasons where why companies weren’t conducting their own research. I thought we could offer cost-effective, inexpensive research services to help. User Interface Engineering, in 1988 (it was our 23rd birthday yesterday!), was one of the first companies to make user research services available to other companies.

However, after working with hundreds of teams and providing their research, we started to looking at how effective we were. Were the teams’ designs getting better? Were they doing more research? Were they creating better user experiences?

We were sorely disappointed with our results. While every team told us they really got a lot out of our work, most weren’t improving their designs. They were appreciative of our reports, but hadn’t read them. They enjoyed our presentations, but weren’t really adopting the recommendations. And, most importantly, their culture didn’t change — they weren’t integrating users into their design process any more than before.

It wasn’t only UIE’s clients with this problem. We reached out to organizations using other outsourced user research services and discovered the same results. Hiring the work out wasn’t getting the job done.

We realized that we were missing an important variable in user research: the team’s direct exposure to their users.

When we take a team on a field research project, we introduce the team members to their users and having them spend time seeing them use the product and doing their work. In doing this, we’ve accomplished 90% of the work of the project.

It’s the exposure that changes the way people work. The same is true for usability testing or interviewing users. The direct exposure is the most valuable part of the project.

When you hire out your user research, even to the most competent of user research professionals, you’re losing 90% of the value. The research becomes a game of telephone, where the “away team” (to steal a Star Trek term) learns all about the users and somehow has to communicate back what they’ve learned. No mount of report writing or presentations can replace that lost experience.

Some UX service companies will tell you that they’ll remain part of the team, integrating the knowledge they learned into the design as the project continues. However, that creates an imbalance, where some people on the team know the users well and others have no idea. Those others, who will eventually own the entire design, are working at a disadvantage and won’t be making their design decisions using this critical knowledge.

This is why we now refuse projects where the team wants to outsource their research. We still do plenty of field visits and usability tests with our clients, but only if they come along to every session. If the client team isn’t there, we won’t conduct the session – there’s no point.

For the folks that think they don’t have time to do their own research: You’re better off taking the money you’ll spend on hiring someone and burning it in the back yard. You’ll get the same value in your product.

Seriously, if you want someone else to do your research because you don’t have time, you’ll need to dedicate twice as much time to spend with the researchers, extracting every little thing they learned about your users. Otherwise, you won’t get the value you paid for. It’s not a time saver to go this route at all.

For the folks who feel they don’t have the skills onboard: That’s an easy problem to fix. Training on user research methods is pretty easy. This is the bulk of our consulting work these days. We use a “Watch one, Do one, Teach one” approach. (We stole it from the medical training world). Most teams pick up the skills pretty quick and do a damn good job in just a few weeks.

And for those folks who feel doing your own research introduces a bias: You’re right, but it doesn’t matter. There’s always a bias in research, even when you get a third party to execute it. There’s nothing wrong with biased research, as long as you understand your biases and how to counter act them.

If there’s anything you can outsource, it could be participant recruiting. However, make sure you work with someone trained in UX recruiting, not market research recruiting. UX trained folks (we use Usability Works – they’re awesome!) know how to deliver the information they learn about your users in the process.

That said, you should even try to resist outsourcing your participant recruiting. You learn a lot when you talk to your potential users, even if they don’t qualify for the study. When you’re outsourcing it, you’re flushing a lot of great source material down the toilet.

Once you’re in the habit of doing your own research, you’ll never want to go back. It’s just too awesomely addicting and useful.

What is a good success rate for a usability test task?
We just conducted user testing on a site map. So we have success rate percentages for each task. They range from 9% – 51% success (in up to 3 tries). Obviously there are problems. (And no, we didn’t create the site map, which makes me feel good.) But what would be considered a “good” success rate? I want to say over 70% for this test. It is only site map, no content, which will limit the success anyway. Maybe I’m aiming too high?

Thinking in terms of % of completion may not be the right approach. (In fact, I’m hard pressed to come up with a time when it is the right approach.)

You haven’t said anything about who the users are or what the site map information contains. But let’s pretend the users are doctors and nurses and the site map contains the necessary information for them to administer drugs safely. If one of those doctors or nurses doesn’t find the information they need, they could improperly administer a treatment which could kill their patient. What would be an acceptable failure rate under these conditions? I’d say 0% — the system needs to ensure success of every user.

Why is your system any less important? Why would you be willing to tolerate any failures?

The real question isn’t “what is an acceptable level of failures?” The question I think you want is “What’s preventing people from succeeding?”

Instead of looking at how many people succeed versus how many fail, what if you were to analyze the failures themselves. Can you rank and categorize all the things that prevent your users from succeeding? Can you assign a classification that helps you understand whether the problems are life and death (as in the example of doctors and nurses I used above), problems that will lose customers, problems that will cost support money, and problems that are annoying without painful side effects?

This will also help you look at the participants you’re recruiting for your study. How similar are they to real users? How realistic are the tasks you’re asking them to complete? How well does the system, if they make a mistake at the site map, help them still succeed by having guidance for common errors on the content pages themselves? (Such as “If you’re looking for x, click here.” type lateral navigation.)

In the end, you really want to understand the problems real users will encounter. That’s the purpose for the studies. Then you want to explore solutions that resolve those problems. In an ideal world, it’s not that you get 100% task completion, it’s that you have addressed and solved all the problems.

The closer you can get your studies to map true in-the-wild user behavior, the more you’ll understand about the problems you’re uncovering and the solutions that will help. Focus on the problems and their resolution and you’ll get the design to where you’d like it to be.

We’re such big fans of the process that I wrote an article about it several years ago. It also happens to be one of our most popular articles.

In our office, we probably conduct a KJ once a month. The KJ Technique is a great way to align everyone to the main objectives and priorities. We often go back to the results during projects to ensure we’re on the right path and help answer additional questions.

In today’s tips, we’re reprinting the article, The KJ Technique: A Group Process for Establishing Priorities. If you haven’t conducted a KJ analysis, you should try it with your team. This article gives you the step-by-step process to conduct your own. For those familiar with it, this article is a nice refresher.

Read the article The KJ-Technique – A Group Process for Establishing Priorities.

In this week’s UIEtips, we reach back into the articles archives and look at some of the challenges we’ve seen users encounter in our usability tests. These are challenges to look out for when users interact with your applications. I hope you enjoy it.

Read the article, 3 Important Usability Challenges for Designing Web Apps.

At UIE, a big part of our research agenda focuses on how to create web applications that delight users. We feel it’s so important that we created a conference focusing on web applications. It’s the Web App Masters Tour.

During the 2 day conference, you’ll hear from 9 Masters on mobile design strategy, data visualization and design best practices. Get all the details on the Seattle and Minneapolis stops at http://www.UIETour.com.

Steve Portigal of Portigal Consulting uses the methods of synthesis and ideation to approach this crucial next step. During his virtual seminar, Steve explains that synthesis is the process of turning field data into insights and then how you move to ideation to turn insights into solutions. So many questions came up during the seminar that Steve ran out of time to answer them all. He tackles the remaining questions in this podcast.

Here’s an excerpt from the podcast.

“…If you were ranking things against something that no one cares about you’re not really deriving value from that process. One thing that I think has been important for us is to align on the factors before we go into the ranking section, so that we know what we’re going to do.

When I say align, I mean that it’s us and the people that are going to be in that session. We’re doing idea generation with a large group, but we’re doing ranking with a very, very small group. My favorite would be a maximum of three to keep it kind of tight.

So I think we’re trying to understand and facilitate discovery with our stakeholders. And then understand their measurement of success. What are their business goals? And I think there are a handful of obvious ones around feasibility and cost and kind of investment and payoff.

I think those feel vaguely standard to me in terms of, how do businesses perform and how do businesses make products. I think more what I would like to get out of that facilitation is to kind of tease out the nuance. I think I gave the example in the webinar of a criteria that one team we worked with had and that criteria is about feasibility.

So, when I hear that, to me that means, can we build it? Is this Star Trek technology or is this something that we can actually build? And when we kind of tease it out, feasibility for them was actually about regulatory stuff. It was about legal feasibility…”

Steve addresses these additional points in the podcast:

• How long does it typically take to get from the first interview in the field to the opportunities?
• How much do you involve the client in the analysis process?
• Should you deliver both good and bad statements so that it’s more palatable to your sensitive stakeholders?
• Who is the top-line report really for?
• Is it dangerous to present the top-line report to the stakeholders?
• Can the opportunities translate into specific user requirements rather than solutions?
• Should collaborative analysis happen with the end users?

Do you have experience analyzing user research data? Share your thoughts with us in our comments section.

Recorded: April, 2011
[ Subscribe to our podcast via ←This link will launch the iTunes application.]
[ Subscribe with other podcast applications.]
[ Transcript Available ]

Now, it feels like we do our best to convince people they should be conducting tests on their own, even those folks who want to pay us. Don’t get me wrong: we love to conduct usability tests. We think they are great fun and we learn something every time.

That’s just it. We learn things the clients should be learning themselves. In a typical project, if we’re lucky, we might communicate half of what we learn about the client’s users, designs, and team members back to the client. In our minds, that’s too much waste of important information, left in our minds and not integrated into the thinking of the client.

In this week’s UIEtips, I look back on an article I wrote in 2007, Usability Testing: Oh, The Things You Can Learn. In the article, I catalogue some of the things a team learns when they put together their own usability tests, starting with recruiting and ending with the analysis. Of course, in a short article, I couldn’t cover everything, but I think I nailed the important stuff nobody ever talks about.

Read the article, Usability Testing – Oh, the Things You Can Learn.

If you’re looking to make the most of the data you get from user research, you’ll want to sign-up for our next UIE Virtual Seminar with Steve Portigal – User Research Analysis Techniques. Steve will show your team how to take the information you’re gathering and take productive steps to moving your design decisions forward. Learn more about the webinar.

Do the recruiting yourself. This gives you bonus research data about your people, and you may learn things you hadn’t anticipated that will influence how you conduct the study. It also starts a relationship with people that gets them invested in taking part. It’s the start of a conversation with you and your organization. They’re more likely to trust your motives and give you a deeper, richer view of their lives.

Focus on behavior, not demographics. If you want people to use your tool to do a particular thing, look for people who already do that somehow. For example, if you want people to use your design to store their photos and music and other content, find people who have a lot of that type of content and who are concerned about losing it. If you want people to use your design to generate invoices, find people who are doing that now and observe what they do to generate invoice when they’re in the mode of doing it. If you want people to use your design to remember to take their medicines at the right time in the right dosage, find people who have persistent conditions that need medication and who have been diagnosed with the type of condition you want to help them deal with.

You’ll see that I never once mentioned age, sex, income, location, education level, marital status – or any of those things that marketers go on. Because it doesn’t usually matter. What matters for UX design is behavior. Do people do the thing you want to make a design for?

There’s a lot more insight in her answer. A must read.

Some other thinking on this topic:

• Avoiding Demographics When Recruiting Participants: An Interview with Dana Chisnell
• UIE Virtual Seminar: Recruiting for Usability Testing: Getting the Right People in the Room for User Research and Usability Tests
• Report: Recruiting without Fear

Our goal, of course, is to learn everything we can. We need to leverage the research time, to ensure we’re filling our brains with the information we’ll need to create great designs.

In today’s UIEtips, I talk about three questions I’ve learned not to ask in sessions. By avoiding these questions, we get to the information we need faster. Read the article Three Questions You Shouldn’t Ask During User Research.

Once you’ve learned what you need from the research, you’ll want to put it in a form that helps you speed through your design process. That’s exactly what scenarios help you do and, coincidentally, why we’ve asked Kim Goodwin to come to UI15. She’ll teach us how to make scenarios work. You can find out about Kim and the other great UI15 experts at UICONF.com.

Explore all the workshop presenters. Register by August 26, 2010 for the lowest rate.

Have you compiled your own questions that you shouldn’t ask? Share your list below.

Today, we have many more options. Portable, inexpensive equipment and software renders formal laboratories obsolete. And the newest of options on the horizon, the remote usability test, is changing the way we think of user research.

In today’s UIEtips, Nate Bolt explores the pros and cons of remote usability testing. He shares his experiences from the projects of his company, Bolt | Peters, where they’ve conducted hundreds of remote studies. I think you’ll find his insights very interesting.

Read the article, Pros and Cons of Remote Usability Testing.

On Thursday, July 15, Nate is delivering an even more detailed look at remote usability testing in his UIE Virtual Seminar. You’ll learn which vendors have the best tools and how to avoid some of the common pitfalls others have experienced. There’s still a few seats left.

Have you added remote usability tests to your toolkit? What’s been your experience with them? We’d love to hear your thoughts below.

Our clients recognize that usability testing is still the most effective way to ensure their designs meet users’ needs. But many teams are still struggling with questions about the tool: How do they ensure they are getting the most out of every test? Are they as prepared as they need be? Are they getting everything they should from the tests results?

In this week’s UIEtips article, UIE alumni Christine Perfetti provides us an excellent article about how to ensure you’re truly prepared for your usability studies. Christine worked at User Interface Engineering for eight years and is a recognized expert in the world of design and user research. I’m delighted to have Christine back to write an article for our readers. I think you’ll enjoy it.

Read the article, Are you Really Prepared for Your Usability Study? The Three Steps for Success.

I also wanted to tell you about a great event focusing on user research and usability testing. This August, Christine will be teaching the 2-day Usability Bootcamp in San Francisco and Boston. In the workshop, Christine shares techniques for conducting user research, developing personas, and running your usability tests.

If you’ve been to one of UIE’s events and seen Christine speak, you already know how insightful her workshops can be. Whether you’ve been conducting user research and usability tests for years or you’re just starting out, you’ll want to attend this event.

Plus, we have a special promotion just for you. If you sign up by July 14 using promotion code “UIEtips,” you can attend the 2-day event for only $795—a savings of $300 off the regular price. And each attendee receives free copies of Rosenfeld Media’s books Remote Research and Prototyping.

What challenges have you had conducting usability tests? How have your testing techniques worked for you? I’d love to hear what you’re doing.

Subscribe to our podcast via

This week we have Jon Wold, an interaction designer calling into the show from Oslo, Norway. Jon’s writing a book on usability testing and brings up a really great point about test results: It’s hard to report on problems without suggesting corrections to the problems you observe in testing.

Both Robert and Jared reflect upon how they’ve handled this situation in the past, when conducting tests along with clients. They both agree that presenting test results without ideas for next steps is a recipe for inaction. So how should you improve your testing methods to set your organization up for success?

Robert and Jared both feel it is critical to involve the product’s designers in the testing. The most important takeaway from the testing process is knowledge about how real people use your stuff. With this knowledge, the team’s designers are equipped to suggest solutions.

Tune in to the podcast to hear Robert and Jared’s thoughts for helping teams make design improvements.

Have a serious UX question? Send it in and Jared Spool and Robert Hoekman, Jr. will answer it with a healthy dose of levity. Please send your deep, vexing questions to us at userability@uie.com. We’d love to feature you on the show!

A little over 3 years ago, we started producing 90 minute online seminars. We call them the UIE Virtual Seminars (or webinars). These seminars give you the chance to hear the latest perspectives in the world of design from the field’s premiere experts.

During the live virtual seminars, we receive many questions. It’s rare that we have enough time to tackle these questions. So to address the unanswered questions from the seminar, we record Q&A podcasts with the presenters.

These podcasts are a wealth of information. It’s like a little secret only a few know about. But we want the world to hear the podcasts. And the beauty of these podcasts is that you don’t need to attend the virtual seminar to follow them. So we thought we’d occasionally feature them in our newsletter.

In today’s UIEtips, we focus on two podcasts. One is with Beth Loring on Effective Moderating for Usability Testing. The other is with Fred Beecher on The Whys, Whats, and Hows of Prototyping. Like all our podcasts, they’re available on our web site and on iTunes (and they are free).

Enjoy these podcasts and let us know what you think below.

Subscribe to our podcast via

Jason Fried

Jason Fried brings a unique viewpoint to any design discussion, one informed by years of success with his company, 37signals. Recently, Jared Spool was able to chat with Jason about 37signals’ current work. If you’re joining us at the UIE Web App Masters Tour in Minneapolis or Philadelphia, Jason will go into even more detail. Among the topics they’ll cover in this podcast include:

• the design and development process behind 37signals ID, their single-sign on feature for their four major web applications which serve more than 3 million customers.
• what they’ve learned about controlling ballooning projects
• their on-going experiment with rotating design and development teams

37signals is known as a organization that does little, if any, testing with users prior to launching a feature. Since UIE advocates user research and usability testing as common components to most design projects, sometimes folks wonder why we’re so interested in the process at 37signals, and why we’re excited to have Jason to speak at our events. We don’t believe there’s only one right way to run a design project, and it’s hard to debate the success Jason and his team have had with their process. In this podcast, Jason discusses their experience with building the 37signals ID, and what role testing played in this project:

…we don’t really do any formal sort of usability testing. I think that usability testing, or bringing outside people in to look at this, it clearly would help in some ways, but… you only kind of really get this when you’re going through the real process.

…you can bring people in to read a screen or to kind of fake go through the process, but it’s only when you’re hurried and you’ve just logged into Basecamp and you need to check a new to-do list, and all of a sudden you see this screen, for real… that’s the only time you can really evaluate whether or not this is clear or not. And so I think it’s really hard to simulate those situations.

After listening to Jason’s thoughts on testing and process, how do you feel they would work in your organization? Share your thoughts on their process and your experiences in the comments. After listening to the podcast, you won’t want to miss Jason’s presentation, Backstage at 37signals, at our Web App Masters Tour in either Minneapolis or Philadelphia. We look forward to seeing you there!

Want to learn more from Jason? See his session – Backstage at 37signals at the Web App Masters Tour

Prototyping is an iterative process. You discover what works, what needs improving, and opportunities for new ideas. The earlier you learn about a design change, the easier it is to implement, and the less costly that change will be. Prototyping allows your team to explore ideas before you invest in them.

In this seminar, Todd Zaki Warfel, a recognized leader in the design-research and usability fields, will explore his Eight Guiding Principles for prototyping. These principles are the foundation for more effective prototyping, and will improve your design process whether you’re a seasoned prototyper or just getting your feet wet.

Register before March 24 to get your free personal PDF copy of Todd’s book, Prototyping, A Practitioners Guide, and lifetime access to Fred Beecher’s seminar, The Whys, Whats, and Hows of Prototyping.

Tell us how prototyping fits into your design process. Do you have an example where something in the design was caught early and saved a bunch of money? Or one where something was identified late and cost money? What is your experience with prototyping, and how do you sell it to the rest of the team? Or your stakeholders? Share your thoughts and experiences below. We’d love to hear them!

Subscribe to our podcast via

Julie Zhuo

When Facebook tweaks anything, it gets coverage across the IT and design realms, and sometimes the Wall Street Journal and New York Times. If the users don’t like the changes, they form protest groups… how can a team operate under such a public microscope?

Julie Zhuo knows. She is the Product Design Manager at Facebook. As the principal designer behind the Facebook Platform and Facebook Connect experiences, she’s contributed to the last two major site redesigns. She also leads the front-end user interface engineering team. On her watch the site grew from 8 million college kids to 350 million people across the globe.

Jared Spool had a chance to chat with Julie recently. The stories she shared, from behind the scenes, are fascinating. When was the last time a throng of people gathered outside your office because you changed a feature on your site? Julie tells that story in the podcast, plus touches upon:

• The early years of innovation and launching features fast, and without testing
• Taking major design risks in front of a large, passionate audience
• How they moved to a strong routine of metrics, A/B tests, usability testing and staged rollouts
• …and much more

Julie talks about the transition from the run-and-gun design strategy that Facebook once used to roll out new features fast, and how it evolved to a more measured approach, while still moving quickly:

We are cognizant of the fact that every time we make a change, the initial user reaction is going to be a little bit negative. That’s why listening to feedback really matters. If all of the feedback is basically, “I don’t like this change because it’s different,” then maybe that’s a sentiment that will go away once people use it regularly.

But if the feedback is, “I don’t like this change because now I can’t find my applications,” or “I can’t find chat,” or “I can’t find messages.” Then that’s a real wake-up call for us that we really need to examine this change and see if we’ve regressed in making it easier and better for users.

They progressed to a test first, launch second strategy, that in some sensitive cases involved a lot of testing.

Last December we launched a change to privacy, and so when you logged into Facebook one day, you got a little privacy dialogue that said, “Hey Facebook is making some changes to privacy. Please revisit your privacy settings.”

That is not going to take that long to build. Right? It doesn’t take that long to design, it’s just one little dialog. But the process for us getting to that final point was months and months, because we knew privacy is such a sensitive topic for people that we wanted to be absolutely sure that what we were doing people would be comfortable with. It was the right thing to do.

Maybe four or five months prior to our launch, we were already bringing people in. We hadn’t even started building the pod. It wasn’t really even designed. We were just showing them a little text dialog with the language that we were going to use and with a lot of different options for how we would present this messaging to them.

These are like paper, low-fi prototypes, nowhere near what the final product will be. But prior to us even building and getting nice mocks from everyone, we already had at least five sessions with a bunch of users testing about 30 different versions of the language and the messaging for this dialog.

So many companies struggle with building the proper amounts of user research into their design process, but so few do it with so many users and so much public attention. Julie’s stories are fascinating case studies that should prove valuable to your own organization.

Julie has many more stories from behind the scenes at Facebook which she’ll share with us in San Diego at the UIE Web App Masters Tour. Join us for her Design Lessons from 350 Million session.

Are you struggling to balance the need to launch features fast with the necessary user research? Let us know in the comments.

Subscribe to our podcast via

Conducting a usability test can be stressful, but you know how important this effort is. Effectively moderating a usability test is a critical part of your user research. It can put the design team on the path to success or failure in the next steps of a product’s design. With a little guidance, and some practice, you can master this art of interacting with you users and get the results your organization needs.
 
Back in October, we asked usability testing expert Beth Loring to present a UIE Virtual Seminar on how to Effectively Moderate Usability Tests. In her presentation, she talks about how to interact with participants and finding that balance between helping them feel comfortable and being too friendly. Beth covers what to do when participants get stuck, or even fail a task. There’s quite a bit of good information that will help you moderate your next usability test. As is often the case, we got lots of great questions from the live audience, but just couldn’t get to them all. I got together with Beth to record this podcast and cover some of the remaining issues. If you find yourself wanting more afterward, don’t forget you can still purchase a recording of the session for another 90 minutes of Effectively Moderating Usability Tests.
 
During the podcast, I asked Beth to explore these questions, and more:

• What is the recommended pathway for learning how to moderate usability tests?
• How much subject matter expertise should a moderator have going into a usability test?
• What’s the impact of using a participant more than once?
• How do you respond to a participant when they’re looking for feedback on how they’re doing in the test?
• What’s the reasonable maximum amount of time for a user test session?

 
Tune in to hear more about designing for facets. Still have questions? Start the discussion in our comments, below

(Oh, and by the way, our last event sold out, so you’ll want to Register your team early!)

When you spend time with your customers, it’s an opportunity to learn how to move your design forward. You don’t want to leave important information “on the table”—information that can give you a more complete understanding of how to move your vision forward. You might act on incomplete detail that creates risk when it forces you to guess what the users need. Worse, the partial insight you have may take your design team in the wrong direction.

User research is an expensive endeavor. Make sure you’re prepared to get the most out of every minute that you’re with your users. Come home with a deep insight into their thinking, their lives, and how you can change their experience for the better.

Steve Portigal will show your team the art of asking the question. You might visit the user in their office or home, have them come to you for a usability test, or even have a chance encounter at a trade show or while waiting for an airplane. Do you know what to ask? Do you know what to listen for, to extract the critical detail of what they can tell you about your design?

Steve will help you prepare your team for any opportunity, be it formal user research or less structured, ad-hoc research. He’ll also give you tips on how to work with your stakeholders and executives, who may also be meeting potential customers and users, so they know what to ask and how to listen—integrating their efforts into the research team. (Wouldn’t it be great if they understood why you’re doing what you’re doing?)

Get your team asking good questions, the right questions, with this fantastic seminar. Honing this skill will be a great addition to their Toolbox. Register your team before January 19, with the promotion code TOOLBOX, and I’ll also send you the link to a fabulous webinar Kate Gomoll did for us, Field Studies: The Ultimate Tool in Your Usability Toolbox.

Are you prepared for meeting someone who could be using your next design? How do you make sure you get into their head, learn what their life is all about, and get the information you need to build something truly innovative and delightful? We’d love to hear your ideas and about your experiences below.

Subscribe to our podcast via

User experience research lives or dies by the appropriateness of the participants in the study. If the participants match the real users, you’re set. But if the participants aren’t like the target users, the project probably dies an ugly death. Worse, the design that is based on the bogus data croaks in public.

Earlier this Fall, I asked usability testing expert Dana Chisnell to present a UIE Virtual Seminar on one of the trickier parts of user research projects—Recruiting for Usability Testing. In her presentation, she talked about how to source and screen participants, how to compensate them, and even why you should do the recruiting rather than outsourcing it. We had more excellent questions from the audience than we could ask during the live event. So, I sat down with Dana to record this podcast and cover some of the remaining issues. If you find yourself wanting more afterward, don’t forget you can still purchase a recording of the 90-minute session, Recruiting for Usability Testing.

During the podcast, Jared asked Dana to dig into these questions, and more:

• Why recruit based on behavior, not demographics?
• How many people should you recruit (as opposed to how many you should test) for a user research project?
• How do you define recruiting criteria for a product that doesn’t exist yet?
• How do you take time zones into consideration when recruiting for remote usability testing?
• How do you word your subject lines in emails for the recruitment process?

Tune in to hear more about recruiting for your usability testing. Still have questions? Start the discussion in our comments, below.

Moderating a usability test is difficult. There’s a lot going on, and you have to keep it all moving. Years ago, when we were privileged to have Carolyn Snyder working for us, she came up with a simple metaphor for facilitating usability test sessions. We called it the “Flight attendant, Sportscaster, Scientist” metaphor and it really helped us simplify how to moderate the session.

In today’s UIEtips, I explain Carolyn’s metaphor. You’ll learn the tricks used by the best test moderators, so you can become one yourself. I know you’ll enjoy the article.

You can learn more tips in our upcoming UIE Virtual Seminar, Effectively Moderating Usability Tests, with Beth Loring. Beth’s an expert in the topic, having co-authored a book on the subject with Joe Dumas. You’re going to love her session.

Also, if you’re a pro at usability testing, then you’ll want to hone your skills at Dana Chisnell’s upcoming workshop, Mastering the Art of User Research at the User Interface 14 Conference. In Dana’s full-day session, you’ll learn advanced techniques and tricks for pushing your team’s research to the next level.

Have you tried moderating usability tests? Post your thoughts and questions below.

Conducting a usability test can be stressful, but you know how
important this effort is. Effectively moderating a usability test is
a critical part of your user research. It can put the design team on
the path to success or failure in the next steps of a product’s
design. Relax, you can do this. With a little guidance, and some
practice, you can master this art of interacting with you users and
get the results your organization needs.

Want to learn more, see Beth’s preview, or register?.

Beth will answer your questions and offer some strategies to
consider. Whether you’ve conducted hundreds of usability tests or
about to do your first one, this seminar is sure to give you
valuable tips to use right away. Sign up today, and learn from one
of the experts in moderating usability tests!

If you have yet to moderate a usability test, but will at some
point, what concerns do you have? If you’ve done them, what tricks
and tips do you have to pass along? Share your thoughts, questions,
and concerns below.

Having the participants that match the design’s audience will give the team feedback on what works well and where the design needs rethinking. By learning from the participants, the team can make informed design decisions on all aspects of the user’s experience.

However, having participants that don’t match the audience can be very problematic. The team may miss learning about critical problems while they spend valuable time and resources fixing design issues that aren’t really important in real use.

In today’s UIEtips, I take you back to an interview from July 2008 with Dana Chisnell, the co-author of The Handbook of Usability Testing. We talk about what happens when teams try to use market research demographics as the basis for recruiting their participants and what the alternatives are.

Read my interview with Dana Chisnell, Avoiding Demographics When Recruiting Participants.

We have two great opportunities for you to get more out of your usability testing. On Wednesday, September 30, Dana is presenting a 90 minute UIE Virtual Seminar - Recruiting  for Usability Testing. Dana will show you the tricks to use to maximize your time and money on the right participants to get the right results.

Dana is also giving a full-day workshop, Mastering the Art of User Research, at this year’s User Interface 14 Conference in November. Learn the user research techniques of the pros. Recruit participants easily. Analyze data faster. Communicate results effectively.

User experience research lives or dies by the appropriateness of the participants in the study.

UX researchers just don’t talk about actively recruiting, do they?  Many researchers ignore it, throwing it over the wall to an agency. It’s complicated, time consuming, and nerve-wracking. In this UIE Virtual Seminar, you’ll learn four strategic steps to make recruiting a fun, useful, and interesting benefit to user research.

If you are involved with user research projects and spend any amount of time worrying about getting the right people in the room, then this UIE Virtual Seminar is for you.

Find out more about Dana’s seminar and register?

Or learn more about our usability testing bundle which includes two seminars and the UIE report, “Recruiting Without Fear.”

Tell us how you source and screen participants? What concerns do you have about the recruiting process? Share your thoughts, questions, and concerns below.

Google made a big investment in the Heathrow Google Space project. However, you don’t need Google’s large bank account to pull this off. It’s really quite simple and inexpensive get great insights by conducting field-based research. 

In today’s UIEtips, we have a great article by author and usability testing expert, Dana Chisnell, explaining how you can easily conduct usability tests “in the wild.” She shares some of the trade offs between field-based testing and more traditional lab-based tests. (And she should know! She wrote THE book on usability testing – The Handbook of Usability Testing.)

By the way, Dana will be one of the great speakers sharing her wisdom and experience at this year’s User Interface 14 Conference. I’m very excited about her full-day workshop, Advanced User Research: Dirty Little Secrets, where she’ll reveal oodles of tricks and techniques that nobody ever talks about. I’ve reviewed the course outline and you’re going to love the advanced techniques she’s covering. 

Have you brought your user research efforts into the wild? What’s worked and what hasn’t? Share your experiences with us below.

Knowing how to sell usability testing will substantially help it get approved and supported by an organization. Most development teams we work with understand the benefits of usability testing, yet still struggle to communicate the value to stakeholders.

In today’s UIEtips newsletter, we look back on an article that former UIE staff member Christine Perfetti wrote in April 2007. The article, Five Techniques for Getting Buy-In for Usability Testing, discusses some of the best techniques for getting stakeholders onboard for testing. I think you’ll really enjoy it.

Read today’s UIEtips article.

If you find this article interesting, I highly encourage you to attend the June 17 UIE Virtual Seminar on Upgrading Your UX Team,with Sarah Bloomer. In this seminar, Sarah will touch on how to get buy in for usability testing. Use the promotion code MYARCHIVE when you register and receive life-time access to the recording of this seminar at no additional charge.

Subscribe to our podcast via

This week’s episode features a compelling question from Will Evans. Usually, this is the point in the post where I say something witty, but this is a question we hear so much, I will cut right to the chase!

Will asked,

When you are creating a web application or website, and you are ready to unleash it to the world, at what point can you say that it is “Usable Enough?”

Tune in for one of the most in-depth Userability Podcasts yet, as Jared and Robert give you advice on how to find that “good enough” launch point.

Have a serious UX question? Send it in and Jared Spool and Robert Hoekman, Jr. will answer it with a healthy dose of levity. Please send your deep, vexing questions to us at userability@uie.com. We’d love to feature you on the show! Till then, if your’re in a situation like Will, let us know how you would handle it in the comments!

So, it’s a nice fifteen minutes of fame when we get picked up by the popular press, in this case, Fast Company magazine:

Cliff Kuang did a nice job describing the research I described in my $300 Million Button article from a few weeks ago:

UIE studied people actually using the site, and it turns out that the prospect of registering was enough to turn some users away; meanwhile, even return users had problems logging in because they didn’t remember the email address or password they signed up with. (45% of users apparently had multiple registrations—a few had up to 10.) Granted, these represented a small portion of users. But for a retailer with $25 billion, even small portions signify huge lost profits.

So UIE redesigned the site, replacing the “register” button with “continue.” They also added a message, saying that registering wasn’t required to checkout, but was optional and might be helpful if you returned.

Sales went up 45%—$15 million in the first month, and $300 million in the first year.

Read Cliff’s write up on FastCompany.com.

Back in October we had the good fortune to host Dana Chisnell’s popular Virtual Seminar, The Quick, the Cheap, and the Insightful: Conducting Usability Tests in the Wild, where she told us you don’t have to run usability tests by the book to get great value out of them. Quite a statement considering she co-wrote the book: The Handbook of Usability Testing, Second Edition.

[If you missed the live seminar, you can purchase lifetime access, for you and your team, to the recording here.]

As happens frequently, seminar viewers sent in more excellent questions than we could answer during the session, so we sat down with Dana afterwards for a quick follow-up.

In the interview, Dana gave me great answers to these viewer questions:

• Is there a middle ground between “classic” testing and “quick and dirty” techniques?
• How many people do you need in these “wild” tests to create enough valuable data?
• How should you screen subjects?
• Should designers observe “wild” tests?
• How do you answer critics who claim quick and dirty testing is not scientific?
• What ethical issues are there with recording test subjects?
• Once you get this quick data, what are the next steps?

During the podcast, Dana & I talked about ways to analyze results and we mentioned the KJ Technique. This is a great way to get a team on the same page about the top priorities that emerge from testing. You can find more about the technique in this article.

Are you going rogue and conducting usability tests that aren’t “by the book”? Tell us your trials and tribulations in the comments!

That’s why Dana Chisnell’s session on quick-and-dirty usability testing resonated with so many of the attendees. They realized they could get a research effort off the ground without having to beg for a huge investment. In many cases, they could do it quickly and inexpensively, under the radar, yielding enough valuable information to make substantial improvements to their design.

In this week’s UIEtips, Dana shares how teams are using quick-and-dirty techniques and getting tremendous insights. I’m betting, after you read her article, you’ll see how you don’t need to follow “the book” to learn things that will improve your designs.

Read the article – Quick & Dirty Usability Testing: Step Away from the Book

Have you tried any quick-and-dirty user research techniques? How did they work for you? We’d love to hear about your experiences below.

It’s not clear when “quick and dirty” became a dirty phrase in the usability world. There are those that believe that testing must be scientific, and that takes time and money — luxuries not often available to many development projects.

Usability testing expert Dana Chisnell knows what it means to work by-the-book – she co-wrote “the book” (The Handbook of Usability Testing, 2nd ed.) with Jeff Rubin. In this seminar, Dana will break down the process of collecting user research data, exploring the must-haves, the nice-to-haves, and the certainly-can-do-withouts. You’ll learn how you can answer your essential design questions using methods that would make MacGyver proud.

This presentation is perfect if you have yet to conduct your first usability test. If you’re experienced with testing, Dana will show you some new ways to inject user research into those tight-on-resources projects that keep cropping up.

Register today at http://www.uie.com/events/virtual_seminars/wild/

What questions do you have about Usability Testing in the Wild? What tools or tricks have you used to maximize the resources available, and still deliver quality results? We’d love to hear your thoughts and experiences.

“Find information about the costs for summer camps in Vermont”

the list member asked if there was a problem with using a scenario like this one:

“You’ve got a nine-year-old boy and an eleven-year-old girl. Pretend you need to find a sleep-away camp for both of them and explore what your options are.”

Asking participants to pretend is always a red flag to me. It typically signals that the overall test plan or the recruitment process isn’t doing what it should.

Years ago (before we started number our web versions with things like 2.0), a client asked us to help them with their tests. The agency they’d hired had been recruiting Wall St. execs and asking them to “pretend you’re interested in Leonardo DiCaprio and find out something you don’t know about him.” Of course, these folks weren’t interested in Leo and didn’t work very hard to discover something to satisfy the test moderator.

This prompted us to study the use of scenarios like this in testing. We found that when a participant is pretending, it’s common for their behavior to be very different than when they are actually doing the task for real.

One of the places we kept noticing this was when we watched people shop online. Asking a shopper to pretend to purchase (“Could you find a pair of shoes you might like to buy and put it in your cart?”) produced extremely different behaviors than when we recruited people who needed the product and gave them the cash to make a real purchase. In the former case, they went through motions and skipped steps that we didn’t see when they were considering and purchasing the product for their own true use.

And here’s the kicker: Had the team changed the site’s design based on the data from the pretend purchasers, they would’ve created a design that would’ve prevented sales from the real shoppers. The behaviors were that different. In other words, listening to people pretending could’ve made the site worse and reduced sales substantially. It could’ve been a huge mistake for the team.

The list member mentioned that her clients were pushing this idea, say that they’d “had success with this technique in the past.” If the client thinks “success” means “we watched people and saw things we didn’t think of before”, that might be a good thing. After all, when clients see the design through the user’s eyes, it helps them inform the decisions they’ll make going forward. However, it could also be a bad thing if it leads them in a direction that could make the site worse for people in the real situations.

To help our clients with this, we developed a technique we call interview-based tasks. Instead of asking the participant to pretend, we recruit participants that would likely interact with the design and we interview them to create their tasks in real time.

Instead of asking the participant to pretend “You just got married in the spring and you’re already thinking about a baby,” you would recruit participants that just got married and are considering a new family. (Any good recruiter can find someone like this pretty quickly. If you don’t know any good recruiters, contact me. I do.)

Then you interview each participant about their situation. During the discussion, you and the participant would collaborate to make a task for the test that would be the same scenario as what you had planned, only it will be for real within the context of their life.

[You can find out about interview-based tasks in this article and in this podcast]

Asking participants to pretend could work just fine, as long as they behave the same when pretending as when they are really in those situations. But it could turn out to be a very bad thing. And you won’t know until you compare against real behaviors.

Knowing how to sell usability testing will substantially help it get approved and supported by an organization. Most development teams we work with understand the benefits of usability testing, yet still struggle to communicate the value to stakeholders.

In this week’s article of UIEtips, we’re featuring a popular article that Christine Perfetti wrote last year, which discusses some of the best techniques for getting stakeholders onboard for testing. I think you’ll really enjoy it.

Read the article – Five Techniques for Getting Buy-In for Usability Testing

If usability testing is a concern for you, I highly encourage you to attend Christine’s full-day seminar, Product Usability: Survival Techniques, at the User Interface 13 Conference this October. In this seminar, you’ll learn valuable tools for incorporating usability testing in your development process.

Are you challenged with selling usability testing within your organization? Is your team struggling to get support and buy-in?  How have you gotten your organization onboard? Join the discussion below.

In this episode of Usability Tools, Brian Christiansen and I continue on how to moderate a usability test. As I mentioned last week, the episode got so long that we decided to break it into two parts. You can find part 1 here. This week’s show focuses upon the step-by-step tasks of running the session with a participant and your observers.

Good moderating is critically important to a successful session. Here are a few points we touched upon in the show:

• Practice and repetition improves your moderation skills. Start the session on the right foot by greeting your user on time and by laying out exactly what will happen during the session.
• Inform your user of their rights as a participant; their comfort is key. Have and follow a testing protocol which will lead you through all the information, and through all the testing steps. It should also govern your observers.
• End your session on time. Respecting the time of your participants and observers is paramount. Walk your user out, both out of politeness and because small talk may lead to critical insights.

There’s much more in the show. If you have questions about the role of the moderator, feel free to ask them in the comments. We’ll try to answer them and may even work them into a future show.

[This show is the first in a series we're going to do on the fundamentals of usability testing. In future shows, we'll cover the entire gamut of testing, from initial planning, through task design, to data analysis and beyond. We want to create a complete resource that you'll share with your entire team.]

In this episode of Usability Tools, Brian Christiansen and I talk about how to moderate a usability test. Turns out, the episode got so long that we decided to break it into two parts. This week’s show focuses upon the different roles a single moderator needs to take on during the session.

The usability test moderator has a lot of influence on the success of the test. Moderating isn’t rocket science, but you’ll need to understand the basics before you sit down with your users.

In this week’s show, I talk about the three roles a moderator needs to play during the test.

First is the scientist. The scientist makes sure your tasks get done, notes get taken, and keeps the show on track.

Then we have the sportscaster. The sportscaster gives play-by-play so the design team members don’t miss anything the user does.

Lastly, there’s the role of the flight attendant. This is the most important role. Keeping your test participant happy and comfortable is your number one job.

Tune in to learn the specifics of each role and how they affect one another.

If you have questions about the role of the moderator, feel free to ask them in the comments. We’ll try to answer them and may even work them into a future show. Stay tuned for the second part of the Moderating show next week.

[This show is the first in a series we're going to do on the fundamentals of usability testing. In future shows, we'll cover the entire gamut of testing, from initial planning, through task design, to data analysis and beyond. We want to create a complete resource that you'll share with your entire team.]

Update: Part 2 is now posted.

This week, I had the great honor of speaking with Dana Chisnell, noted usability expert and principal at Usability Works, a consultancy based in San Francisco. Dana is also the co-author of the recently-released second edition of the Handbook of Usability Testing—a book so fine that I was thrilled when they asked me to write the foreword.

In this interview, I asked Dana what makes the best user researchers better than the pack. Dana suggested that great user researchers dig deeper into who the users are. They don’t just stop after watching novices interact with the design for the first time. Instead, they look to constantly learn about the full range of people who use the design.

Also, the top user researchers look beyond the use of functionality to the entire experience. Dana shared how Enterprise Rent-a-Car spends a lot of time and energy thinking about every interaction they have with their customers. They stand around in their retail outlets and watch individuals getting their cars and turning the cars in, looking at how the experiences fit together.

Dana had a lot more to say about what makes the best stand above the rest. You’ll want to listen to the rest of the podcast to hear her thoughts on the subject of excellence in user research.

[For even more insight, you'll want to attend Dana's full-day seminar, Usability Testing Guerilla Techniques: Collecting User Data on a Shoestring, at our User Interface 13 conference that will take place this October 13-16, 2008 in historic Cambridge, Massachusetts.]

We look forward to your questions and thoughts on this podcast. How are you integrating usability testing into your organization? Let us know what you think in the comments!