Outsourcing your user research work is like outsourcing your vacation.
It gets the job done, but probably won’t have the effects you were seeking.

I usually say this when someone is asking me to do their user research for them. This is something we did quite a bit in the early days of UIE, but don’t do any longer.

Usually, they are asking us to do this to save time, because they don’t have trained folks, or because they are afraid of bias. All these reasons are rational, but there are better ways to deal with them than hiring someone else to do the research on their behalf.

As I said, I founded UIE as a company to do just this. I felt the rational reasons where why companies weren’t conducting their own research. I thought we could offer cost-effective, inexpensive research services to help. User Interface Engineering, in 1988 (it was our 23rd birthday yesterday!), was one of the first companies to make user research services available to other companies.

However, after working with hundreds of teams and providing their research, we started to looking at how effective we were. Were the teams’ designs getting better? Were they doing more research? Were they creating better user experiences?

We were sorely disappointed with our results. While every team told us they really got a lot out of our work, most weren’t improving their designs. They were appreciative of our reports, but hadn’t read them. They enjoyed our presentations, but weren’t really adopting the recommendations. And, most importantly, their culture didn’t change — they weren’t integrating users into their design process any more than before.

It wasn’t only UIE’s clients with this problem. We reached out to organizations using other outsourced user research services and discovered the same results. Hiring the work out wasn’t getting the job done.

We realized that we were missing an important variable in user research: the team’s direct exposure to their users.

When we take a team on a field research project, we introduce the team members to their users and having them spend time seeing them use the product and doing their work. In doing this, we’ve accomplished 90% of the work of the project.

It’s the exposure that changes the way people work. The same is true for usability testing or interviewing users. The direct exposure is the most valuable part of the project.

When you hire out your user research, even to the most competent of user research professionals, you’re losing 90% of the value. The research becomes a game of telephone, where the “away team” (to steal a Star Trek term) learns all about the users and somehow has to communicate back what they’ve learned. No mount of report writing or presentations can replace that lost experience.

Some UX service companies will tell you that they’ll remain part of the team, integrating the knowledge they learned into the design as the project continues. However, that creates an imbalance, where some people on the team know the users well and others have no idea. Those others, who will eventually own the entire design, are working at a disadvantage and won’t be making their design decisions using this critical knowledge.

This is why we now refuse projects where the team wants to outsource their research. We still do plenty of field visits and usability tests with our clients, but only if they come along to every session. If the client team isn’t there, we won’t conduct the session – there’s no point.

For the folks that think they don’t have time to do their own research: You’re better off taking the money you’ll spend on hiring someone and burning it in the back yard. You’ll get the same value in your product.

Seriously, if you want someone else to do your research because you don’t have time, you’ll need to dedicate twice as much time to spend with the researchers, extracting every little thing they learned about your users. Otherwise, you won’t get the value you paid for. It’s not a time saver to go this route at all.

For the folks who feel they don’t have the skills onboard: That’s an easy problem to fix. Training on user research methods is pretty easy. This is the bulk of our consulting work these days. We use a “Watch one, Do one, Teach one” approach. (We stole it from the medical training world). Most teams pick up the skills pretty quick and do a damn good job in just a few weeks.

And for those folks who feel doing your own research introduces a bias: You’re right, but it doesn’t matter. There’s always a bias in research, even when you get a third party to execute it. There’s nothing wrong with biased research, as long as you understand your biases and how to counter act them.

If there’s anything you can outsource, it could be participant recruiting. However, make sure you work with someone trained in UX recruiting, not market research recruiting. UX trained folks (we use Usability Works – they’re awesome!) know how to deliver the information they learn about your users in the process.

That said, you should even try to resist outsourcing your participant recruiting. You learn a lot when you talk to your potential users, even if they don’t qualify for the study. When you’re outsourcing it, you’re flushing a lot of great source material down the toilet.

Once you’re in the habit of doing your own research, you’ll never want to go back. It’s just too awesomely addicting and useful.

What is a good success rate for a usability test task?
We just conducted user testing on a site map. So we have success rate percentages for each task. They range from 9% – 51% success (in up to 3 tries). Obviously there are problems. (And no, we didn’t create the site map, which makes me feel good.) But what would be considered a “good” success rate? I want to say over 70% for this test. It is only site map, no content, which will limit the success anyway. Maybe I’m aiming too high?

Thinking in terms of % of completion may not be the right approach. (In fact, I’m hard pressed to come up with a time when it is the right approach.)

You haven’t said anything about who the users are or what the site map information contains. But let’s pretend the users are doctors and nurses and the site map contains the necessary information for them to administer drugs safely. If one of those doctors or nurses doesn’t find the information they need, they could improperly administer a treatment which could kill their patient. What would be an acceptable failure rate under these conditions? I’d say 0% — the system needs to ensure success of every user.

Why is your system any less important? Why would you be willing to tolerate any failures?

The real question isn’t “what is an acceptable level of failures?” The question I think you want is “What’s preventing people from succeeding?”

Instead of looking at how many people succeed versus how many fail, what if you were to analyze the failures themselves. Can you rank and categorize all the things that prevent your users from succeeding? Can you assign a classification that helps you understand whether the problems are life and death (as in the example of doctors and nurses I used above), problems that will lose customers, problems that will cost support money, and problems that are annoying without painful side effects?

This will also help you look at the participants you’re recruiting for your study. How similar are they to real users? How realistic are the tasks you’re asking them to complete? How well does the system, if they make a mistake at the site map, help them still succeed by having guidance for common errors on the content pages themselves? (Such as “If you’re looking for x, click here.” type lateral navigation.)

In the end, you really want to understand the problems real users will encounter. That’s the purpose for the studies. Then you want to explore solutions that resolve those problems. In an ideal world, it’s not that you get 100% task completion, it’s that you have addressed and solved all the problems.

The closer you can get your studies to map true in-the-wild user behavior, the more you’ll understand about the problems you’re uncovering and the solutions that will help. Focus on the problems and their resolution and you’ll get the design to where you’d like it to be.

• Ways to tailor your UX design process to the culture of your organization
• How to conduct research with minimal time and budget
• Techniques to get useful design feedback from stakeholders
• How to make your case in organizations that don’t prioritize design

You’ll be able to put UX principles into practice in any organization, and learn how to make the case for user experience design with results, not theory.

Register with the code UNDERCOVER and add lifetime access
to the recording of this seminar for no extra cost.

The details for you:
UX Design when Time, Money, and Support is Limited with Cennydd Bowles
Thursday, July 21 at 1:30pm ET
1:30pm ET / 12:30pm CT / 11:30am MT / 10:30am PT
90 minute online seminar
Learn more about Cennydd’s seminar or save your spot now!

And one last piece of good news! Thanks to New Riders, we’re giving away copies of Cennydd’s book, UNDERCOVER User Experience Design, to random attendees. Winners will be notified within 24 hours of the live seminar. Join us!

Today, we have many more options. Portable, inexpensive equipment and software renders formal laboratories obsolete. And the newest of options on the horizon, the remote usability test, is changing the way we think of user research.

In today’s UIEtips, Nate Bolt explores the pros and cons of remote usability testing. He shares his experiences from the projects of his company, Bolt | Peters, where they’ve conducted hundreds of remote studies. I think you’ll find his insights very interesting.

Read the article, Pros and Cons of Remote Usability Testing.

On Thursday, July 15, Nate is delivering an even more detailed look at remote usability testing in his UIE Virtual Seminar. You’ll learn which vendors have the best tools and how to avoid some of the common pitfalls others have experienced. There’s still a few seats left.

Have you added remote usability tests to your toolkit? What’s been your experience with them? We’d love to hear your thoughts below.

Subscribe to our podcast via

At UIE, we receive a steady flow of questions about user research. There is a staggering amount of information out in the world, just waiting to guide your designs. Best of all, it’s nearly free for the taking. We’re happy to reveal the key to unlocking this information:

Ask the right questions.

Getting out into the world and actually interacting with real people who use, or potentially could use, your product or service is incredibly valuable. We tell our clients this constantly: the organizations who are most successful are the ones who are on intimate terms with how and why their customers use their product. But how? To answer that question, we invited our friend Steve Portigal, principal of Portigal Consulting, to conduct the UIE Virtual Seminar, “Deep Dive Interviewing Secrets: Making Sure You Don’t Leave Key Information Behind”

Steve’s specialty is informing design decisions by getting on the ground and speaking directly with customers. And sharing how you can do the same. Today, we release the interview Jared Spool conducted with Steve after his seminar, following up with a number of additional audience questions. You can enjoy this interview without first seeing the Virtual Seminar, but afterwards, you’re going to want to see it. You’re in luck. You and your team can still access the recorded seminar.  (See special offer below)

Jared asked Steve, in the end, what does the interview process really boil down to for it to be effective? Steve’s answer?

You have to really, really listen.

Jared and Steve discuss several points about the interviewing process, drawing on stories from both their experiences. Here are some brief highlights:

• Understand the why behind what people are saying. This improves designs.

• Transcribe your interviews. They can be used as a deliverable for a client and they allow you to critique your interviewing technique. Steve uses Chromolume Transcription, but there are many options (see note at the bottom of this post).

• How do you deal with uncomfortable situations, like when an interviewee’s supervisor wants to observe your interview? Steve thinks this is often an indication of a failure in the planning process. When you’re in the field, there are a couple of tricks you can use to help steer the situation towards productivity.

• How do you deal with interviewees who ramble? Try not to interrupt, Steve suggests. This may be their natural inclination and you should try to roll with it. Try to keep them on track with very focused questions.

• How do you end an interview when you discover you’re talking with someone that won’t be helpful? Hopefully your planning will avoid this, but even so, you may find later that the person’s insights help you more than you thought. Stay in the moment.

• Can you interview well into the product design stage? Sure, you can even bring prototypes. Ask, “how does this work for you?” “How would you teach your (parent/significant other) how to use this?”

• How do you deal with difficult interviewees? There’s no saving some interviews, but you should attempt to build a friendly rapport that exudes professionalism. Many times they’ll open up.

Have a listen to the podcast. You’ll certainly pick up some tips for the next time you’re planning research or are out in the field.

Special offer:

For a limited time, you can gain lifetime access to this seminar at a special price. Purchase the recording before May 21 for just $99 (that’s $50 off the regular pricing). Anyone in your organization will be able to watch Steve’s seminar whenever they want, as often as they want.

Note, as an aside,

Steve mentions having had some trouble with the transcription service, CastingWords. For what it’s worth, we use CastingWords at UIE for our transcriptions, and aside from the occasional issue, we have been happy with them. Also, they now offer time stamps, for an additional fee.

When we do our jobs well, important decisions are made correctly. Designs are improved. Experiences transition from frustrating to delightful. Assuming we do our jobs well.

Doing our jobs well is very hard work. A thousand details need to line up just perfectly. If we don’t get things just right, important decisions are made wrong. Designs regress. Experiences frustrate even more.

As user experience professionals, it’s all about the assumptions we make. If we assume correctly, things go well. It’s when we make false assumptions that problems occur. How do we know when our assumptions are any good?

In this week’s article, we look back to an article originally published in 2006; Interview-Based Tasks: Learning from Leonardo DiCaprio. In the article, I address the assumption question head-on by looking at a testing technique known as interview-based tasks. This non-traditional approach to usability tests helps work around the assumptions built into standard task design, allowing teams more flexibility and insight into what users actually need from the design.

When using interview-based tasks, the art of asking the question is critical. How you prepare for the interview, build rapport with the interviewee, and how to work with varying levels of experience and expertise will determine how successful the interview-based task is completed.

That’s where Steve Portigal comes in. Our next UIE Virtual Seminar is on Deep Dive Interviewing Secrets: Making Sure You Don’t Leave Key Information Behind. This is a not-to-miss-seminar if you want to know more behind the art of the question.

Have you tried interview-based tasks? What insights did you gain from it? How else have you checked the assumptions that go into your work? Join the discussion by submitting a comment below.

(Oh, and by the way, our last event sold out, so you’ll want to Register your team early!)

When you spend time with your customers, it’s an opportunity to learn how to move your design forward. You don’t want to leave important information “on the table”—information that can give you a more complete understanding of how to move your vision forward. You might act on incomplete detail that creates risk when it forces you to guess what the users need. Worse, the partial insight you have may take your design team in the wrong direction.

User research is an expensive endeavor. Make sure you’re prepared to get the most out of every minute that you’re with your users. Come home with a deep insight into their thinking, their lives, and how you can change their experience for the better.

Steve Portigal will show your team the art of asking the question. You might visit the user in their office or home, have them come to you for a usability test, or even have a chance encounter at a trade show or while waiting for an airplane. Do you know what to ask? Do you know what to listen for, to extract the critical detail of what they can tell you about your design?

Steve will help you prepare your team for any opportunity, be it formal user research or less structured, ad-hoc research. He’ll also give you tips on how to work with your stakeholders and executives, who may also be meeting potential customers and users, so they know what to ask and how to listen—integrating their efforts into the research team. (Wouldn’t it be great if they understood why you’re doing what you’re doing?)

Get your team asking good questions, the right questions, with this fantastic seminar. Honing this skill will be a great addition to their Toolbox. Register your team before January 19, with the promotion code TOOLBOX, and I’ll also send you the link to a fabulous webinar Kate Gomoll did for us, Field Studies: The Ultimate Tool in Your Usability Toolbox.

Are you prepared for meeting someone who could be using your next design? How do you make sure you get into their head, learn what their life is all about, and get the information you need to build something truly innovative and delightful? We’d love to hear your ideas and about your experiences below.

User experience research lives or dies by the appropriateness of the participants in the study.

UX researchers just don’t talk about actively recruiting, do they?  Many researchers ignore it, throwing it over the wall to an agency. It’s complicated, time consuming, and nerve-wracking. In this UIE Virtual Seminar, you’ll learn four strategic steps to make recruiting a fun, useful, and interesting benefit to user research.

If you are involved with user research projects and spend any amount of time worrying about getting the right people in the room, then this UIE Virtual Seminar is for you.

Find out more about Dana’s seminar and register?

Or learn more about our usability testing bundle which includes two seminars and the UIE report, “Recruiting Without Fear.”

Tell us how you source and screen participants? What concerns do you have about the recruiting process? Share your thoughts, questions, and concerns below.

Knowing how to sell usability testing will substantially help it get approved and supported by an organization. Most development teams we work with understand the benefits of usability testing, yet still struggle to communicate the value to stakeholders.

In today’s UIEtips newsletter, we look back on an article that former UIE staff member Christine Perfetti wrote in April 2007. The article, Five Techniques for Getting Buy-In for Usability Testing, discusses some of the best techniques for getting stakeholders onboard for testing. I think you’ll really enjoy it.

Read today’s UIEtips article.

If you find this article interesting, I highly encourage you to attend the June 17 UIE Virtual Seminar on Upgrading Your UX Team,with Sarah Bloomer. In this seminar, Sarah will touch on how to get buy in for usability testing. Use the promotion code MYARCHIVE when you register and receive life-time access to the recording of this seminar at no additional charge.

Robert Nyman does just that with the Nokia and SonyEricsson UK sites and finds some surprising results. It’s quite a good write up.

Very much worth a read followed by the obvious next step: checking your own site out the same way.

[Hat tip to NortyPig]

It’s not clear when “quick and dirty” became a dirty phrase in the usability world. There are those that believe that testing must be scientific, and that takes time and money — luxuries not often available to many development projects.

Usability testing expert Dana Chisnell knows what it means to work by-the-book – she co-wrote “the book” (The Handbook of Usability Testing, 2nd ed.) with Jeff Rubin. In this seminar, Dana will break down the process of collecting user research data, exploring the must-haves, the nice-to-haves, and the certainly-can-do-withouts. You’ll learn how you can answer your essential design questions using methods that would make MacGyver proud.

This presentation is perfect if you have yet to conduct your first usability test. If you’re experienced with testing, Dana will show you some new ways to inject user research into those tight-on-resources projects that keep cropping up.

Register today at http://www.uie.com/events/virtual_seminars/wild/

What questions do you have about Usability Testing in the Wild? What tools or tricks have you used to maximize the resources available, and still deliver quality results? We’d love to hear your thoughts and experiences.

“Find information about the costs for summer camps in Vermont”

the list member asked if there was a problem with using a scenario like this one:

“You’ve got a nine-year-old boy and an eleven-year-old girl. Pretend you need to find a sleep-away camp for both of them and explore what your options are.”

Asking participants to pretend is always a red flag to me. It typically signals that the overall test plan or the recruitment process isn’t doing what it should.

Years ago (before we started number our web versions with things like 2.0), a client asked us to help them with their tests. The agency they’d hired had been recruiting Wall St. execs and asking them to “pretend you’re interested in Leonardo DiCaprio and find out something you don’t know about him.” Of course, these folks weren’t interested in Leo and didn’t work very hard to discover something to satisfy the test moderator.

This prompted us to study the use of scenarios like this in testing. We found that when a participant is pretending, it’s common for their behavior to be very different than when they are actually doing the task for real.

One of the places we kept noticing this was when we watched people shop online. Asking a shopper to pretend to purchase (“Could you find a pair of shoes you might like to buy and put it in your cart?”) produced extremely different behaviors than when we recruited people who needed the product and gave them the cash to make a real purchase. In the former case, they went through motions and skipped steps that we didn’t see when they were considering and purchasing the product for their own true use.

And here’s the kicker: Had the team changed the site’s design based on the data from the pretend purchasers, they would’ve created a design that would’ve prevented sales from the real shoppers. The behaviors were that different. In other words, listening to people pretending could’ve made the site worse and reduced sales substantially. It could’ve been a huge mistake for the team.

The list member mentioned that her clients were pushing this idea, say that they’d “had success with this technique in the past.” If the client thinks “success” means “we watched people and saw things we didn’t think of before”, that might be a good thing. After all, when clients see the design through the user’s eyes, it helps them inform the decisions they’ll make going forward. However, it could also be a bad thing if it leads them in a direction that could make the site worse for people in the real situations.

To help our clients with this, we developed a technique we call interview-based tasks. Instead of asking the participant to pretend, we recruit participants that would likely interact with the design and we interview them to create their tasks in real time.

Instead of asking the participant to pretend “You just got married in the spring and you’re already thinking about a baby,” you would recruit participants that just got married and are considering a new family. (Any good recruiter can find someone like this pretty quickly. If you don’t know any good recruiters, contact me. I do.)

Then you interview each participant about their situation. During the discussion, you and the participant would collaborate to make a task for the test that would be the same scenario as what you had planned, only it will be for real within the context of their life.

[You can find out about interview-based tasks in this article and in this podcast]

Asking participants to pretend could work just fine, as long as they behave the same when pretending as when they are really in those situations. But it could turn out to be a very bad thing. And you won’t know until you compare against real behaviors.

In this episode of Usability Tools, Brian Christiansen and I continue on how to moderate a usability test. As I mentioned last week, the episode got so long that we decided to break it into two parts. You can find part 1 here. This week’s show focuses upon the step-by-step tasks of running the session with a participant and your observers.

Good moderating is critically important to a successful session. Here are a few points we touched upon in the show:

• Practice and repetition improves your moderation skills. Start the session on the right foot by greeting your user on time and by laying out exactly what will happen during the session.
• Inform your user of their rights as a participant; their comfort is key. Have and follow a testing protocol which will lead you through all the information, and through all the testing steps. It should also govern your observers.
• End your session on time. Respecting the time of your participants and observers is paramount. Walk your user out, both out of politeness and because small talk may lead to critical insights.

There’s much more in the show. If you have questions about the role of the moderator, feel free to ask them in the comments. We’ll try to answer them and may even work them into a future show.

[This show is the first in a series we're going to do on the fundamentals of usability testing. In future shows, we'll cover the entire gamut of testing, from initial planning, through task design, to data analysis and beyond. We want to create a complete resource that you'll share with your entire team.]

In this episode of Usability Tools, Brian Christiansen and I talk about how to moderate a usability test. Turns out, the episode got so long that we decided to break it into two parts. This week’s show focuses upon the different roles a single moderator needs to take on during the session.

The usability test moderator has a lot of influence on the success of the test. Moderating isn’t rocket science, but you’ll need to understand the basics before you sit down with your users.

In this week’s show, I talk about the three roles a moderator needs to play during the test.

First is the scientist. The scientist makes sure your tasks get done, notes get taken, and keeps the show on track.

Then we have the sportscaster. The sportscaster gives play-by-play so the design team members don’t miss anything the user does.

Lastly, there’s the role of the flight attendant. This is the most important role. Keeping your test participant happy and comfortable is your number one job.

Tune in to learn the specifics of each role and how they affect one another.

If you have questions about the role of the moderator, feel free to ask them in the comments. We’ll try to answer them and may even work them into a future show. Stay tuned for the second part of the Moderating show next week.

[This show is the first in a series we're going to do on the fundamentals of usability testing. In future shows, we'll cover the entire gamut of testing, from initial planning, through task design, to data analysis and beyond. We want to create a complete resource that you'll share with your entire team.]

Update: Part 2 is now posted.

The quality of the weights will depend on what information the team already has. In the case of the project I described in my presentation, the first time that team sat down to create their matrix, they had done very little user research (as in virtually none) to work from. So, to some extent, they were going to guess on the weights. However, they had been in business for years with a web site that was attracting millions of visitors a month, so they knew something about their users and the users’ needs. It’s with this information that they’d start the process of assessing weights.

Whether the team has good research to back up their assessment, or whether they are just guessing from their experience, the process is basically the same. Once we have our list of differences (as I explained in yesterday’s post), we assess the weights.

We like to use a facilitator for this process — someone who isn’t going to contribute to the weights and isn’t going to push an agenda. It can be a team member, but they need to understand that they are abstaining from giving their own opinions. (It’s ok if a team member has an agenda they want to push, they just shouldn’t be the facilitator.)

For each difference we’ve identified, the facilitator asks for the votes. The easiest way to make this happen is for everyone in the room to raise their two hands, displaying between zero and ten fingers. The facilitator then calls out each number around the room. (“7, 5, 6, 3, 8, 7″)

Then comes the fun part: The facilitator picks someone who had a vote that was very different from the rest and asks them to explain their rationale. So, if almost everyone was sixes or sevens, the facilitator would ask the person with the lowest vote, say a three, to explain why they rated it so low. Then, the facilitator would ask someone with an opposite vote to explain their rationale. The goal is to get a small debate going to bring out the differences in thinking.

The facilitator can decide when the group has heard enough of the debate. We don’t want to fixate on this, since there is still a lot to do, so moving quickly is a good thing.

After ending the debate, a second vote is taken, again by holding up fingers. Again, the facilitator reads off everyone’s fingers and then declares what the final number should be. We use the “olympic scoring method” by throwing out the highest and lowest scores and averaging the rest.

We like this three-step process (vote, discuss, vote again), because it keeps the proportionate to the differences of opinions. When the team really agrees, the debate is really short. Only when there is real disagreement does the debate take time, but with the help of a good facilitator, it still can go quickly.

(One of the most fun moments for us is when, in the first vote, everyone goes in one direction except for a single dissenting team member. That member shares their rationale and everyone goes, “Oh, of course” and then votes with that person. It’s fun to see a team have a group-wide “aha!” moment like that.)

In Part 3, I’ll talk about how we assess the scores for each design alternative.

The process we’d use is to compare the designs side-by-side and list the differences. A method I’m fond of is to do the comparisons with two sites at a time. In this case, we’d probably start with the current design and alternative #1. We’d put them side-by-side and ask, “What makes these designs different?” Once we’d exhausted our thinking, we’d replace alternative #1 with alternative #2 and repeat the questioning, looking for new differences to add to the list.

To be complete, after comparing all the alternatives to the current design, we might try comparing them to each other. However, we can probably just eyeball the different designs to catch the last few differences that we may have missed on the first pass of comparisons. In my experience, 95% of the differences are discovered on the first pass. For most projects, that’s good enough.

Given five alternatives, we’d allocate an hour to generate this list.

Tomorrow, we’ll talk about how we would assess the weights for each difference.

Here’s what the session was described as:

What’s the Best Way to Compare Multiple Design Alternatives?

Good design practice suggests you create multiple sketches, which eventually evolve into full-out design alternatives. However, once you have those alternatives, what’s the best way to decide which one wins?

I will review one approach and talk about the pitfalls and advantages to evaluating multiple designs at once.

Because of An Event Apart and our Virtual Seminar, I didn’t have as much time to prepare as I would’ve liked. So, I ended up creating the entire presentation while sitting in the back of the room, waiting my turn to present.

Here’s how it turned out:

| View | Upload your own

I should mention that the subject of this talk was inspired by the UIEtips article, A Counter-Intuitive Approach to Evaluating Design Alternatives, published on May 19. Thanks to Pauric for inviting me to the meeting to present the idea.

This week, I had the great honor of speaking with Dana Chisnell, noted usability expert and principal at Usability Works, a consultancy based in San Francisco. Dana is also the co-author of the recently-released second edition of the Handbook of Usability Testing—a book so fine that I was thrilled when they asked me to write the foreword.

In this interview, I asked Dana what makes the best user researchers better than the pack. Dana suggested that great user researchers dig deeper into who the users are. They don’t just stop after watching novices interact with the design for the first time. Instead, they look to constantly learn about the full range of people who use the design.

Also, the top user researchers look beyond the use of functionality to the entire experience. Dana shared how Enterprise Rent-a-Car spends a lot of time and energy thinking about every interaction they have with their customers. They stand around in their retail outlets and watch individuals getting their cars and turning the cars in, looking at how the experiences fit together.

Dana had a lot more to say about what makes the best stand above the rest. You’ll want to listen to the rest of the podcast to hear her thoughts on the subject of excellence in user research.

[For even more insight, you'll want to attend Dana's full-day seminar, Usability Testing Guerilla Techniques: Collecting User Data on a Shoestring, at our User Interface 13 conference that will take place this October 13-16, 2008 in historic Cambridge, Massachusetts.]

We look forward to your questions and thoughts on this podcast. How are you integrating usability testing into your organization? Let us know what you think in the comments!

I’ve posted the slides above and have synched it up with audio from the conference. (Unfortunately, there was a mic-input problem during the recording and they ended up using the built-in mics instead of the sounds system. So, the recording is noisy and unintelligible in places. Sorry about that.)

Here’s the description of the talk:

Journey to the Center of Design

User-centered design was born in the 1980s, amidst a world filled with frustration with blinking VCR clocks and computer command lines. Up until this time, developers focused on making the devices work, giving little heed to how they’d be used. Terms like “user friendly” and “easy to use,” buzzwords for the UCD movement, soon became as common as “new and improved” on laundry soap.

We guarantee a journey that shouldn’t be missed.

| View | Upload your own

You can download the slides (without audio). On the Slideshare site, you can view this presentation full screen to see the details.

What do you think of this presentation?

In this week’s article for our email newsletter, UIEtips, I share some of the challenges we’ve seen users encounter in our usability tests of web applications. These are challenges you’ll want to look out for when users interact with your applications. I hope you enjoy it.

You can read my article here.

At UIE, a big part of our research agenda focuses on how to create web applications that delight users. In the upcoming months, we’ll share our tips and techniques in our weekly Usability Tools podcast.

What are some of the challenges you’ve had to address when building web-based applications?

Duration: 22min | File size: 12.5 MB
[ Subscribe to our podcast via iTunes. This link will launch the iTunes application.]
[ Subscribe with other podcast applications. ]
[ Text Transcript ]

Each week in our Usability Tools Podcast, we will be sitting down to discuss tips and tools for improving your site’s user experience. The goal of our weekly podcast is to share some of the most important findings from UIE’s research on web design and usability.

In this week’s podcast, Brian Christiansen and I continue exploring usability techniques for web-based applications. Web-based applications are different from content-based web sites because the users are involved in a transaction. When we’re researching the usability of a content-based site, we’re focused on how users will find and react to the information. However, with web-based applications, there are many other considerations that we need to account for.

If you missed the first part of the show, you can listen to it here:
Usability Tools Podcast: Useful Web App Usability Techniques, Part 1

This week, we explore the usability technique toolbox, focusing on those methods that help us with web-based applications.

In this episode we start with the basic usability test, move onto variants, then talk about field studies. In each case, we explore the web-app specific advantages and talk about how we get the information we need to make informed design decisions.

We talked about several books in this episode:

• The Handbook of Usability Testing—by Jeff Rubin
• A Practical Guide to Usability Testing—by Ginny Reddish and Joe Dumas
• Paper Prototyping—by Carolyn Snyder

As always, we’re very interested in hearing from you. Do you have questions or comments about this episode? We love to create shows based on your questions. Please leave a comment below or email us directly at mailbag@uie.com

UIE’s Latest Research: If you’re interested in the topics we discuss in the podcasts, I highly suggest you sign up for our free newsletter, UIEtips, to read our latest usability and design research as soon as we publish it. We’ll also notify you in UIEtips when we publish new podcasts.

Joshua Porter (formerly of UIE, but now doing great work on his own at Bokardo Design) recently described much of the latest online debate about the need to develop personas when designing. Josh got a lot of things right, but he got one thing very, very wrong. And, unfortunately, he bases a lot of his argument on that one thing.

Josh said this:

Definition, please?

But while all of this arguing is going on, nobody is really defining what personas are. This, of course, is a big part of the problem. What most definitions don’t say is that personas are a document. They might be a poster, a word file, or a PDF. But they are a document that represents an archetypical person that is passed around design teams. Ok, just wanted to make that clear.

Personas are not a document. Persona descriptions can be a document (or a movie or any other practical rendering). But, those are just renderings of what happened during the persona creation process.

Here’s the way to think of it:

Personas are to Persona Descriptions as Vacations are to Souvenir Picture Albums.

While people who didn’t go on the vacation can look through the album and think, “Boy, that must’ve been fun,” they’ll never get the full experience of what the actual vacation experience was. The album is just a remnant.

In the UX community, many folks are now saying, “I’ve looked at these documents and they just don’t do anything for me. I don’t think personas are valuable.” Unfortunately, they are judging the value of creating and using robust personas based on the quality of the paper deliverable. If the vacation pictures aren’t compelling, did the vacation itself suck?

To be fair, I think there could be a lot of improvement in the ways people document their personas. Todd Zaki Warfel, over at MessageFirst has some interesting diagrams, though he admits they take a bit of explanation.

Todd Zaki Warfel’s persona description with his Persona DNA chart

Stuart Karten is also doing interesting stuff with his ModeMapping work.

Stuart Karten’s ModeMapping deliverables

However, these are just the final souvenirs, after the team has gained the real value. That value comes when the team visits and observes their target audience, absorbs and discusses their observations, and reduces the chaos into patterns, which then become the personas.

What’s in the team’s head, as they are designing, is what will make a difference in the final design. The persona descriptions are just there to remind everyone what happened.

If you’re amongst those who insist on judging the value of personas on their descriptions, I suggest you cancel your next vacation and just order one of these gift boxes. I’m sure you’ll find it just as valuable as the vacation itself.

Jen wrote:

I enjoyed your December 10th article, on assessing UX skills, so I sent it around to my colleagues. It was favorably received, but the folks on the alias found that one core skill was missing: the creation of UI mockups, particularly interactive mockups.

The places where I saw the creation of mockups implied were: under Interaction design, “creating design deliverables such as wireframes and design priority descriptions.” and in the methods section, “Team members need to understand how to integrate their work with development approaches, such as Agile techniques.”

However, you never come right out and say it’s a skill …

It is indeed a skill. Actually, several, I think.

You need to work with the tools available, whether it be HTML, Flash, paper, or any new tools coming down the pike, like Adobe Thermo. Knowing which tools is best for the situation your in is an important skill.

You also need to know what makes a good prototype and how to get the most from it. Prototypes should be smoke and mirrors — not full running versions. Knowing what functionality to omit from the prototype is a skill.

Another critical skill is knowing how to collect usability data from a prototype. Prototypes that users can actually use (versus just look at and say, “Yah, I don’t like the fonts”) is an important technique during design.

I’d bundle all of these skills under Interaction Design.

A while back I wrote a piece about personas where I talked it about why it’s a bad idea to base your personas on a single real individual. Using a real person as a substitute persona might seem like a good idea since you know that everything about that user is for real. The problem, though, is that one individual comes with quirks that may not be representative of the market as a whole. If you interview enough people, you can eliminate their biographical peculiarities by focusing your personas on the characteristics common to the group.

Thanks, Bruce!

The session went well, but we ran tight on time and I didn’t get to all the great questions. Brian and I were just talking and we’re planning a followup podcast on some questions.

Meantime, one attendee, Pamela, couldn’t wait to ask me some of her questions. Here they are and my responses:

Do you have tips on how to convince subject-matter managers (decision makers) why you have adopted personas and what the benefits of using them are? The use of personas as a methodology for the creation of new products is a new concept for [our organization]. This is my biggest challenge at the moment.

My first reaction is to avoid trying to convince anyone of their benefit. If the “subject-matter manager” (a term I’m not completely familiar with) believes they know everything there is to know about how users will use the product, then there is probably nothing you can do to convince them otherwise.

Instead, I’d look to finding someone who is feeling pain because they are discovering they don’t know everything they should’ve known, usually because users aren’t reacting to the product’s design they way they’d hope. Typically, when you have a poor experience, it reflects somewhere in the organization’s bottom line (lost revenue, increased expenses, and work being redone are just a few symptoms). Find the person responsible for reversing those bottom-line effects and you’ve got your champion for doing something different.

I wrote about this in more detail in The Cost of Frustration.

Is the use of personas useful for product review? Alan Cooper tells us that throwing away original code/design is unlikely. Therefore, what is the best option for reviewing products that were not originally designed with user goals in mind?

One thing you can do with a set of well-defined personas is walkthrough the design and role play their behavior. (“How would Mary respond to this question?”, “Is Pierre likely to remember his password at the login screen? What if he doesn’t?”) This can quickly point out many potential risk areas on the current design.

It can also help you argue for more usability testing. When a walkthrough shows many potential risks, usability testing with people who match those riskiest persona types will likely yield a ton of insight into the design and potential improvements.

Given that the gap continues to grow between the power user and the non-technical user, will the power user in your opinion become growingly frustrated by having products primarily designed with the non-technical informant in mind?

For most scenarios, there is little use between what is referred to as power users and non-technical users. They will be happy with the same designs.

What we typically mean when we talk about the difference between power users and non-technical users are two things:

1) The power users have very different current knowledge — they know more about the tool and its usage. (More about this in my article, What Makes A Design Seem Intuitive? and a Virtual Seminar on the same topic.)

2) The power users may need an interface that’s optimized more for efficiency than learning.

In these cases, developing personas for your power users may be an important tool. It’s often possible to create designs that adequately service both types of personas (but not always). Having both types covered in two (or more) separate personas will help experiment with potential designs and keep the team’s awareness of the distinctions.

In the set of questions developed for ethnographic interviews (Stage 1 of your seminar [conducting field interviews]), is it important to include questions asking informants what their user goals are and what their vision of the perfect product description is? Or, are these questions answered by the design team based on final persona descriptions (Stage 3 [Building the Personas and Scenarios])? Having a vision of what the final product will look like prior to product development, I would think, would be beneficial to serve as a benchmark for product developers who will be able to determine when the product is “done”?

User’s goals have to do with their personal and business needs. For example, a user may be tasked with keep track of changes in consumer purchasing patterns, so they can inform their organization’s strategic planning committee on shifts in the marketplace. Their goal may be to someday be part of the that committee, so accurate data collection and projections are a key part of their needs.

We’ve found it’s not useful to ask them what their vision of the perfect product is. At best, they can only tell you what they have seen elsewhere. (If I were to ask you what improvements you’d like to see in cars coming off the production line in 2012, what would you tell me that wasn’t part of today’s existing technology or some part of popular science fiction?)

Instead, you want to watch them do their work and learn about their goals. From that, you can figure out a vision on how to optimize the user experience to better meet their needs. You know what the technology and team’s potential are — you can match that to the needs you observe.

Also, I’d like to suggest there is no such thing as a “final product” (unless the entire development team is going to suddenly die or quit). Once developed, products continue to live on in one form or other. Therefore, you can’t have a vision of a final product.

You can, however, have a vision of what the product will be in the future, say 5 years from now. That’s an extremely useful exercise to help the team focus and understand what the short-term priorities should be.

I’ve written about this in these places:

Solid Vision: A Critical Element to UX Design
The Experience Vision
The 3 Steps for Creating An Experience Vision

We’ll post more answers to the great questions we received shortly. Stay tuned.

Personas provide a fabulous way for people who understand their audience/users to communicate their understanding to people who do not understand that audience in a rapid and meaningful way. They are also a great way for people who have different understandings of their target audience (users, customers, etc) to reach common ground on who they are designing for, particularly if the objective is to expand into new audiences rather than simply to serve an existing audience. Neither of these uses of personas requires any original actual hard field research to be done. All it takes is a few good conversations and collaborations around a white board, and a little running around to gather existing documentation.

In my mind, Christopher is clearly confusing Personas with User Descriptions. User descriptions are what-we-think-we-know-now writeups of who uses our design and why. Personas, on the other hand, are carefully researched and crafted personalities we create to focus the design energy.

I believe both are very useful in the design process. User descriptions help us see where our thinking is, help new team members come up to speed, and help us identify where we may have made assumptions that could turn out false. Personas helps us get past the this-design-is-for-every-breathing-being problem and help us focus our attention on the needs of three to seven specific individuals. Many teams regularly do both.

That said, I think confusing them would be a bad idea. It’s akin to using the term apple when talking about a tomato. They both are fruits, round, red, and have seeds in the middle. Yet, someone thinking they are getting an apple is going to be very disappointed on that first bite if they received a tomato instead (and vice versa).

A few years back, in a project studying the work of master wood craftsmen, we learned there are dozens of types of saws. All of them cut things (mostly wood, but not always). Interestingly, the best craftsmen knew the individual names of each type of saw they owned and were really miffed when you called a tool by the wrong name. To them, the subtle distinctions were very important, since knowing to use the right tool at the right time in the right way was a big part of what distinguished them from other, less capable, craftsmen.

So, I recommend we call things by their names and not try and bunch different types of design activities and deliverables under one name. (And don’t even get me started with the folks who often refer to usability tests as focus groups.)

We don’t use personas. We use ourselves. I believe personas lead to a false sense of understanding at the deepest, most critical levels.

Every product we build is a product we build for ourselves to solve our own problems. We recognize our problems aren’t unique. In fact, our problems are probably a lot like your problems. So we bundle up the solutions to our problems in the form of web-based software and offer them for sale.

We recognize not everyone shares our problems, our point of view, or our opinions, but that verdict’s the same if you use personas. Making decisions based on real opinions trumps making decisions based on imaginary opinions.

I’ve never been a big believer in personas. They’re artificial, abstract, and fictitious. I don’t think you can build a great product for a person that doesn’t exist.

There was a lot of discussion on Jason’s blog, with many sentiments similar to this one from Mimo:

I never heard of Personas before. Now I read it on wikipedia. The idea sounds interesting. But I think at the end of the day it is crap. The product is always shaped by two things. YOUR experience (your present ego and YOUR idea (your future ego).

So, to add into the fray, here’s my thoughts on using personas:

It takes virtually no skill to build something crappy

No one is going to make you use personas. If you create a design without using personas, I’ll promise you the sun will continue to rise on schedule, without variation. The universe will remain intact.

However, how do you know you’re actually meeting the needs of your users? After all, that is why you were designing in the first place, right?

Some products, like the tools built by 37Signals, don’t need personas. Not because the folks at 37Signals have any special powers, but because they themselves are the personas they want to build for. They build tools they like to use themselves. For them, that will work great.

Not all teams have that luxury. A hospital IT team, building software systems used by critical care nurses in the hospital’s pediatric intensitve care unit, are not building tools they will use themselves. They are building tools used by others whose education, experience, goals, contexts, and tasks are extremely different.

A well-built, robust persona set can help educate the IT design team on what it’s like to be a critical care pediatric ICU nurse and the things they need to deal with. This information will inform their designs. And good personas help inform the design process.

Oh My God! They’re Made of People!

In his post, Jason says:

Personas don’t
Personas don’t talk back. Personas can’t answer questions. Personas don’t have opinions. Personas can’t tell you when something just doesn’t feel right. Personas can’t tell you when a sentence doesn’t make sense. Personas don’t get frustrated. Personas aren’t pressed for time. Personas aren’t moody. Personas can’t click things. Personas can’t make mistakes. Personas can’t make value judgements. Personas don’t use products. Personas aren’t real.

People do
People talk back. People answer questions. People have opinions. People can tell you when something just doesn’t feel right. People can tell you when a sentence doesn’t make sense. People get frustrated. People are pressed for time. People are moody. People click things. People make mistakes. People make value judgements. People use products. People are real.

It’s clear that Jason hasn’t used robust personas, because, when designed well, they do all these things. Jason hasn’t had the Soylent Green moment to realize that well-designed and researched personas are made of real people — real people who you can ask questions of, observe their frustrations, and discover their true goals.

I can see where Jason’s coming from. Recently we conducted a study of several dozen organizations who claimed to use personas. Less than 5% actually conducted field research to inform their personas. The remaining 95% just made up the descriptions from internal guesswork.

If you’re just going to guess on the personas, why bother? Just design for yourself, like the 37Signals team does.

However, when you do the field studies, you create relationships with the people in your research. You can return to those people and ask them questions. You can learn about the things they do.

The persona becomes a package for containing what you’ve learned from your field research. A package that is transportable to everyone on the team, so they can have the same benefits of knowing the users as you have.

Once you have well-designed, robust personas, you can take advantage of the benefits that come from them:

• Preventing Grounding
• Using the Oral Tradition
• Role Playing

In our research, teams that utilize robust personas find they create better designs, especially for things they wouldn’t normally use themselves.

I’m going to talk in great length today about building robust personas in our latest virtual seminar. See the description for more information. (It’s available live today and there’s still plenty of room. There will be a recording available shortly.)

Recently, Cheryl got back to me about her company’s reaction:

A few months back you kindly responded to my inquiry about facilitating a KJ Analysis session. The team did a great job and I was relieved as a couple of the team’s members could have easily hijacked the agenda. As you pointed out, by not allowing any discussion until the end of the process, the team — the entire team — was able to talk about the important stuff.

Within a week of facilitating that exercise, I was contacted by a several colleagues who were going to be leading a similar team exercise and had heard through the grapevine how well our activity went using the KJ Technique. I shared what I had learned from your presentation and my own facilitator notes (basically index cards with keywords so I would remember what to do next). I didn’t think to much about until yesterday, my Senior Manger made a comment about the KJ technique sweeping the company. I guess a lot of teams have adopted it as a means to brainstorm and prioritize without spiralling into the black hole of fruitless discussion.

So, thanks Jared! You get another (because I must assume you receive so many) pat on the back. Once again I was able to take something from one of your virtual seminars and put it to work!

Thanks, Cheryl!

Have you had good (or not-so-good) results from something you’ve learned from us? We’d love to hear about it.

Each week in our Usability Tools Podcast, I will be sitting down with UIE’s Managing Director, Christine Perfetti to discuss tips and tools for improving your site’s user experience. The goal of our weekly podcast is to share some of the most important findings from UIE’s research on web design and usability.

This week, Christine Perfetti asked me about one of UIE’s most popular usability testing techniques, Interview-based Tasks. A couple of years ago, I wrote an introductory article about interview-based tasks, and we’ve been getting questions about the technique ever since.

In usability tests where we incorporate interview-based tasks, the participants interests are discovered, not assigned. Unlike traditional task design methods, the test facilitator and participant negotiate the tasks during the tests, instead of proceeding down a list of predefined tasks. Because each task is drawn from the experience and interest of each participant, no two participants perform exactly the same tasks. In this podcast, Christine and I discuss:

» How interview-based tasks successfully engage users
» How to create interview-based tasks
» Why User Interface Engineering’s researchers moved away from traditional task design
» How to recruit users with this technique
» How to combine interview-based tasks with other types of tasks
» Where to learn more about task design: The excellent book by Ginny Redish and JoAnn Hackos, User and Task Analysis for Interface Design

As always, we’re very interested in hearing from you. Do you have questions or comments about this episode? Do you have suggestions for future episodes? We want to know. Please leave a comment below or email us directly at mailbag@uie.com.

UIE’s Latest Research: If you’re interested in the topics Christine and I discuss in the podcasts, I highly suggest you sign up for our free newsletter, UIEtips, to read our latest usability and design research as soon as we publish it. We’ll also notify you in UIEtips when we publish new podcasts.

New: Survey and listener drawing!
We would like to give you and your co-workers free admission to our next Virtual Seminar program, with full, lifetime access to the archived program as well! All you need to do to be eligible is give us your feedback on your podcast listening experience. Fill out the following survey and each week we’ll randomly send one survey participant a free admission to the next UIE Virtual Seminar and Archive, a $169.00 value! We appreciate your input!

Participate in our survey to win!

As designs and product visions become more complex, there is an increasing need to find ways that effectively convey just how people will use the product and integrate it into their lives. Recently, more and more organizations and design teams have turned to comics to communicate difficult product and design concepts to large and diverse audiences.

Yesterday, we published a podcast recording of a wonderful chat I had with Yahoo!’s Kevin Cheng. I think Kevin’s work integrating tools like comics into the development process is remarkable. I highly suggest you check out our podcast. In this week’s issue of our email newsletter, UIEtips, we’re publishing an excerpt of my conversation with Kevin. Kevin shared a lot of great information with me and I think you’ll find the conversation enlightening.

Read today’s article with Kevin Cheng.

Is your organization considering the use of comics to convey product concepts? What techniques are you using in your organization to communicate how users interact with your designs? Share your thoughts below.

Daniel asked:

Would the result be the same if the participant sat in a different room from the 30 observers?

Are there benefits by having them in the same room?

On initial reads this seems scary for the participant (especially when walking in to see 30 people)

In my experience, the result is different for both the observers and the participant when they are in different rooms.

For the observers, when there’s a window or a video link separating them from the participant, that distance sends a message. I’ve found observers are less likely to pay attention and more likely to make jokes and derogatory comments when they are isolated.

Participants aren’t dumb and know someone is watching. They can see the mirrors and the cameras and know they are being observed. But, because they can’t see their observers, they can only imagine what those observers are doing. I’ve noticed participants tend to assume the worst and the mirror/video-link unnerves them more than having the observers right in the room.

Another benefit comes from direct interaction. Having the observers in the same room as the participants means they can interact. I’ve found developers and stakeholders come away from sessions they attend this way with a solid connection to the users they are trying to design for.

Chris said:

I would be terrified to have my back to a room of 30 stakeholders and experts.

Chris, I’m assuming your fear may be justified because of your organization’s culture. In my experience, most of the time, this fear is completely unfulfilled and everyone behaves nicely, getting a lot out of the study. However, if you don’t want to have the stakeholders behind you, there are two options:

1. Rotate the moderator and participant station 90 degrees, so the stakeholders are not behind you, but on one side of you. (You’ll be positioned between the participant and the observers.) This way you can keep an eye on both. I’ve done this plenty of times and it works well.
2. Get a buddy to help you with “crowd control”. Either you or your colleague sit with the observers and, through careful passing of notes, help them understand what they are seeing. Don’t underestimate the power of a well-timed hand-written note.

Dr. Pete asked:

Do you find that doing observation on an actual subject has more value than just taking the 30 observers through a simulated session? Do you throw that subject’s data out?

Dr. Pete, it sounds like you think I’m doing this only for demonstration purposes. In fact, these are real tests that the observers are participating in. No data is thrown out. The entire series of tests use the same setup.

The last time I did this was for a top-10 e-commerce site (who averages approximately $1.2 billion in sales each year). We observed 24 users this way, collecting the data from each one. Each session was filled with 30 folks (we had to put together an elaborate signup and waiting-list system to handle it), often with senior executives — directors, VPs, and even the CEO. Most observers came to more than one session and would cancel other meetings to attend.

As a result, changes were made that look like they’ll realize about $250,000,000 in additional revenue this year. I don’t think we could’ve had the impact if I didn’t have the 30 observers in each session.

Each week in our Usability Tools Podcast, I will be sitting down with UIE’s Managing Director, Christine Perfetti to discuss tips and tools for improving your site’s user experience. The goal of our weekly podcast is to share some of the most important findings from UIE’s research on web design and usability.

This week, Christine Perfetti asked me about one of UIE’s most valuable usability testing techniques, Inherent Value Tests. Inherent Value Testing gives the team important information about how well a web site communicates the inherent value the designers are putting into the site. In this podcast, Christine and I discuss:

» How inherent value tests help measure how your site communicates your product’s value
» How inherent value tests are different than traditional usability testing techniques
» How to recruit users with this technique
» How to combine inherent value tests with other types of tasks

As always, we’re very interested in hearing from you. Do you have questions or comments about this episode? Do you have suggestions for future episodes? We want to know. Please leave a comment below or email us directly at mailbag@uie.com.

UIE’s Latest Research: If you’re interested in the topics Christine and I discuss in the podcasts, I highly suggest you sign up for our free newsletter, UIEtips, to read our latest usability and design research as soon as we publish it. We’ll also notify you in UIEtips when we publish new podcasts.

New: Survey and listener drawing!
We would like to give you and your co-workers free admission to our next Virtual Seminar program, with full, lifetime access to the archived program as well! All you need to do to be eligible is give us your feedback on your podcast listening experience. Fill out the following survey and each week we’ll randomly send one survey participant a free admission to the next UIE Virtual Seminar and Archive, a $169.00 value! We appreciate your input!

Participate in our survey to win!

Larry is co-founder of Constantine & Lockwood, the pioneer of usage-centered design, and author of the landmark book, Software for Use.

September 27th UIE Virtual Seminar: Don’t Panic — Design and Usability Under Impossible Pressure ($129—includes handout)

[Larry is also going to present the full-day seminar, Interaction Design in an Agile World, at this year's UI12 Conference. You'll learn how to use the agile development process to implement a shorter lifecycle and encourage team collaboration.]

We’ve found it’s critical to the adoption of testing in the organization for people to quickly see the benefits. We’ve also found there’s no better way to show the benefits of testing than to observe a test firsthand.

Because there’s a large number of design agents and stake holders involved and often a small number of available tests to observe, we can find ourselves in a situation where we need to have many observers in one session. It’s not unusual to have 30 or more observers sitting in a single usability test. Here’s how we pull it off.

First, get a large room

We love training rooms. They work really well. (We’re not a big fan of usability labs — they distract and hurt more than they help, in our opinion.)

You want a room where you can set up the seats for the observers, a projector, an audio system, and a small station for the participant and facilitator.

Often, we’ll set up the room to look like this: (Thanks to Dana Chisnell for supplying this image from the upcoming revision to the Handbook of Usability Testing.)

The participant and session facilitator will sit up front. The user (and, if you can get a mixer, the facilitator) will have a mic, which plugs into the speakers, so everyone can hear what the participant says (or even mumbles).

We project the participant’s screen onto the large screen so all the observers can watch what the user is doing.

(If you want to get super fancy, you can put a video camera plugged into a monitor below the projected screen to see the user’s expressions. We don’t do this because we rarely find it adds anything for the extra effort and stress on the user’s part. If you do decide to use it, don’t do a picture-in-picture display on the big screen — in inevitably obscures an important part of the interface. )

Brief the observers

We’re hold a 15 minute meeting with all observers before each participant shows up. In this meeting, we go over the rules for observing. We’re quite strict about these, since our number one objective in a usability test is to lookout for the stress of the participant. We hammer in how a session like this can be stressful on the participant and perfect behavior is absolutely necessary to have a successful study.

We don’t allow observers to come or go during the session. They have to arrive before it starts and stay for the entire duration. (You’d be surprised how often other things can be rearranged to accommodate this when you’re strict about it.)

We also explain how noises affect the participant’s behavior, so we tell them they are not allowed to laugh, sigh, or make any other noises while the participant is focusing on their design.

We explain how questions will work (I’ll get to that in a bit). Finally, we go over the tasks (which we’ve provided a copy for each observer) and the test protocol, including the objectives.

Brief the participant

Outside the session room, we greet and brief the participant. In this session, we walk through the usual pre-test protocol of a test: explaining what the purpose is, what their role will be, how we want them to behave, and how they are in control (they can stop or take a break at any time).

At this point, we’ll explain how there will be 30 (or however many there actually are) members of the design team observing, what the observer’s role is, and how the participant can practically ignore their presence. We explain the observers will see (on the big screen) and hear (thru the speakers) everything the participant does and how this will help us create simpler and more delightful designs in the future.

Introduce the observers to the participant

We’ll then bring the participant in the room. At this point, the facilitator will say something like, “Everybody, say hi to Lisa” to introduce the participant. The facilitator will then review the pre-test paperwork to “introduce” the participant and familiarize the observers with the participant’s background and experience.

Conduct the Test

Once the facilitator has explained the protocol to the participant (we do this in the room so the observers can hear it again), we start on the test. In addition to ensuring the participant has a stress-free experience, the facilitator’s role becomes that of sportscaster — making sure the subtle things happening with the participant are observed by the spectators.

If the participant makes faces, non-verbal utterances (like sighs or mumbles), or other things the observers wouldn’t pick up on, the facilitator will make a comment, such as “I noticed you seem to be frowning. Is something not right?”

If the observers have questions for the participant, they can either wait until the Q&A period we hold after each task, or they can write their question on a note and pass it to the facilitator. The facilitator has the discretion to ask the question or hold it (if they think it will disturb the participant’s flow or gives away too much information).

If observers want to talk to each other, we recommend passing written notes. (After years of doing this, we’ve found participants rarely pay attention to notes being passed. They do pay attention to whispers and typing, so we ask observers to refrain from whispering, instant messenger, or taking notes online.) Observers have to be careful not to make a joke in the notes that might cause a colleague to laugh. (Participants will almost always assume the laughing has something to do with them, even though it rarely does.)

When the end-of-task Q&A session starts, any observer can ask a question. (During the pre-test briefing, we often explain not to ask “Design” questions — questions like, “How would you order the options in the menus?” since these questions rarely yield interesting answers and take up valuable time while the participant makes a stab at answering. Instead, we suggest they ask behavioral questions, “Can you tell me what is confusing about the menus?”)

Saying Goodbye

Once the participant has completed the test, we ask the observers to thank them for their time. We then have the facilitator escort the participant out of the building. This gives the participant a chance to give any feedback that they weren’t comfortable giving before the entire group. (This rarely happens, but it has happened enough we allow for it.)

Debriefing Observers

After the session, before the observers disperse, we ask them to share important things they learned. Often, this won’t take much prompting.

It’s in this point where we can help them form the best inferences. (Inexperienced observers often jump to the first inference that comes to mind, often overlooking serious alternative causes of the problems. If left unchecked, they may rush to make changes that won’t improve the experience and may break something else.)

Do participants get more stressed with 30 observers?

With proper preparation of both the observers and the participants, we find it’s just like any other study. The key takeaways from doing this are:

• Make sure the room is set so each observer can clearly see and hear what’s going on. No matter how many you have, when observers can’t follow along, they get bored and fidgety, which leads to problems.
• Make sure the participant is not surprised upon entering the room by the crowd. Talking to them before they walk in will help tremendously. If you can warn them when talking to them on the phone the day before, that’s even better.
• The facilitator needs to have both crowd-control skills and comfort-giving skills. They’ll be tending to the observers and participants’ needs simultaneously. In our experience, this is something that gets better with practice.
• The observers need to understand from the outset that a usability test is a structured process with a defined protocol and agenda. Once the study starts, things must progress as planned, so if they have specific concerns, they need to address them during the test planning stage, before the first participant arrives on site.

It’s hard enough creating usable designs when there is ample time and plenty of resources, but what do you do when that’s just not an option for your design team? One of the most common reasons designs fail users is because the design team didn’t have the time or resources to focus on user research or conduct usability testing. That’s why one of our main goals at User Interface Engineering is to disseminate some of the quick-and-dirty techniques available to designers and usability professionals.

One of our most valuable techniques is the 5-Second Test. This technique turns out to be a quick, flexible testing method that gives teams fascinating insights about the design. In this week’s issue of our email newsletter, UIEtips, we re-printed an excellent article that describes the 5-Second Test technique and discusses its advantages and disadvantages for measuring the effectiveness of web site content pages. I think you’ll enjoy it.

Read today’s UIEtips article.

Christine Perfetti and I also recently discussed the topic in our Usability Tools Podcast Series. If you would like to learn more about the techniques, I highly suggest you give it a listen.

Also, if you’re struggling to come up with the time or resources to conduct user research, you’ll really want to take advantage of the latest live presentation in UIE’s Virtual Seminar Series. On September 27th, my good friend, Larry Constantine, will present the online seminar, Don’t Panic: Design and Usability Under Impossible Pressure, sharing a wide range of field-proven techniques for conquering design and usability problems while in crunch mode. You really shouldn’t miss it.

Each week in our Usability Tools Podcast, I will be sitting down with UIE’s Managing Director, Christine Perfetti to discuss tips and tools for improving your site’s user experience. The goal of our weekly podcast is to share some of the most important findings from UIE’s research on web design and usability.

This week, Christine Perfetti asked me about one of UIE’s most popular usability testing techniques, 5-Second Tests. A couple of years ago, Christine Perfetti wrote an excellent introductory article about 5-Second Tests, and we’ve been getting questions about the technique ever since.

The 5-Second Test Method shows users a single content page for a quick 5 seconds to gather their initial impressions. Five seconds may not seem like a lot of time, but users make important judgments in the first moments they visit a page. In this podcast, Christine and I discuss:

» Why 5-Second Tests should be used primarily to test a site’s content pages
» Why 5-Second Tests aren’t effective on most Home Pages
» How to conduct this test with your users
» What some of the common mistakes design teams make when conducting a 5-Second Test
» How to recruit users with this technique
» How to combine 5-Second Techniques with other types of tasks

As always, we’re very interested in hearing from you. Do you have questions or comments about this episode? Do you have suggestions for future episodes? We want to know. Please leave a comment below or email us directly at mailbag@uie.com.

UIE’s Latest Research: If you’re interested in the topics Christine and I discuss in the podcasts, I highly suggest you sign up for our free newsletter, UIEtips, to read our latest usability and design research as soon as we publish it. We’ll also notify you in UIEtips when we publish new podcasts.

New: Survey and listener drawing!
We would like to give you and your co-workers free admission to our next Virtual Seminar program, with full, lifetime access to the archived program as well! All you need to do to be eligible is give us your feedback on your podcast listening experience. Fill out the following survey and each week we’ll randomly send one survey participant a free admission to the next UIE Virtual Seminar and Archive, a $169.00 value! We appreciate your input!

Participate in our survey to win!

These days, we’re seeing more and more design teams successfully adopt personas into their design process. Personas allow teams to delve right into the attitudes, behaviors, and goals of their target audience. They’re also a great consensus building tool, helping every member of the design team understand exactly who they’re designing for.

In this week’s UIEtips, we’re publishing an interview I recently conducted with Steve Mulder after he presented the UIE Virtual Seminar, The User is Always Right: Making Personas Work for Your Web Site ( http://tinyurl.com/23w9sz .) In the interview, we talk about how to create effective personas based on user research as well as valuable tips for convincing your organization to adopt personas into the design process. I think you’ll really enjoy it.

Read today’s UIEtips article.

Has your organization adopted personas into the design process? We’d love to hear your thoughts. Join the discussion below about this week’s topic.

Want to learn more about personas? Check out the UIE Virtual Seminar: The User is Always Right — Making Personas Work for Your Web Site, presented by Steve Mulder. In this 90-minute presentation, you’ll learn about the entire persona creation, and understand how personas can help you uncover the goals, attitudes, and behaviors of your target audience.

In the method, teams brainstorm on potential answers to the focus question, group the answers, name each group, then vote on the group names that best answer the question. The method, in less than 45-minutes, allows teams to come to a democratic consensus on an answer, avoiding endless discussion for elements that turn out to be unimportant.

Our friend Cheryl recently decided to try the technique with her team and ran into a little confusion. Here’s what she wrote us:

After taking your “Making Sense of Usability Test and Field Study Data” webinar, I decided to try out the KJ Analysis with a team. I went t the UIE website and read the paper you published in May 2004. I ran a pilot with a group and ran into some problems.

I stumbled when it came to [the "Naming each group"] step and [the "Voting on the most important groups"] step in the technique. Folks had used different names for each of the groups of post-its. When I asked them to rank the groups, folks recorded their stars on their on group post-it — so a group may have had 9 stars, but it appeared as 3 stars on each of the three group post-its.

How do you resolve different names for groups? I’m a bit desperate — I need to figure things out by then end of the week. The “real” KJ process will occur early next week.

The problem Cheryl had is common for people trying this for the first time. The reason for these two steps is to work out the problems she encountered, so let me share some guidelines:

• When people are naming each group, they should be reading through the stickies and looking for a “theme”. The name will represent what they think the theme will be.


Naming Each Group: Group names are on the smaller orange and green stickies.

• Sometimes, groups form with multiple themes. If this is the case, you want to split the groups up. (You don’t want to discuss this — just instruct anyone who discovers this to go ahead and do it.)
• Similarly, you may discover, in the course of identifying the themes and naming the groups that more than one existing group share the same theme. In that case, you can merge them. (Again, no discussion necessary — just do it.)
• Two people, reading the same group, may see different themes and give them different names. Perfectly natural and it works out just fine. In fact, we instruct the team that they should attempt to give their own name to every group. The only reason they can skip naming a group is if someone else has “already used exactly the same words” they would’ve used to describe it. This will often encourage 2-3 names on many groups.
• When it comes time to vote for their personal “top groups” — the ones they think are most important to addressing the focus question (the focus question is the question being answered by the exercise) — it’s ok if team members put votes on different names within the same group. They should put the votes on the group name that best represents their thinking of the focus question.


Voting on Groups: Each team member chooses the group names that best address the focus question.

• When you collect up the groups to rank overall, you collect up all the group names with votes on them. Don’t worry about keeping group names from the same group together at this point. Just rank them as if they were from different groups.
• During the “Combining Like Categories” step, you’ll ask the question, “Are these two groups the same?” While they may have come from the same initial groups, they may actually mean different things. Only if you can easily substitute one for the other, do you combine their votes. So, if the group has complete consensus they mean the same thing, the fact they started with separate names will come out in the wash. If someone believes they mean something different, you’ll have an important discussion as to why they are different and then let them stand on their own in the final ranking.


Ranking Top Groups: Team combines the votes of groups names that are true synonyms.

The result of this is that group names that are true synonyms get resolved and their votes are combined. Group names that turn out to be different perspectives on the same results end up standing on their own. You’ve allowed these different perspectives to emerge with just the right amount of discussion based on their importance to the task (it doesn’t matter if they don’t get anyone’s vote) and the team is still spending the most “discussion time” talking about the most important outcomes of the exercise — not discussing the un-important edge conditions (which is where many teams get trapped).

At User Interface Engineering, we’ve spent two decades researching the usability and design techniques employed by both successful and unsuccessful development teams. We’ve focused our research agenda on identifying the key techniques successful teams utilize to consistently produce usable products on time and within budget.

Because we think these techniques are essential tools for all designers and usability professionals, we teach them to everyone who attends UIE’s most popular course, Product Usability Survival Techniques. I think these techniques are crucial for every development team’s success.

In this week’s article, one of our most popular instructors, Christine Perfetti, describes several of the techniques we share with design teams in our courses and consulting work. I think you’ll really enjoy her insights.

Read today’s UIEtips article.

What usability techniques do you utilize in your own work? At what stage of development does you team begin usability testing? Do you take advantage of paper mockups? I’d love to hear what you’re doing. Join the discussion below about this week’s topic.

Want to learn more about the five survival techniques for creating usable products? At this year’s User Interface 12 Conference, Christine Perfetti will teach the half-day UIE Showcase Seminar: Product Usability Survival Techniques.

(Over at Yahoo!, they use something similar for the personas they create.)

A Flickr set with more images of these cards

Here’s an image of a set I made here at UIE, which I modeled after baseball player cards. These don’t need to be fancy, in fact I made this set in about ten minutes. Here’s how we do it:

1. We start with images of our users. We snap these during our field visits or when people come in for testing. We assure them that the images won’t be used for anything commerical.
2. Second, we use the OmniGroup’s excellent OmniGraffle to layout the cards. If you don’t have OG, you could use something from your Office suite, for instance PowerPoint would be fine. You could simply divide a slide into 6ths to create your cards.
3. We dropped in our images.
4. We added labels our cards with as much or little info as necessary. Here we used their “user number,” which is how we tracked them through testing and their first name. We wanted the cards to be as easy to read from a medium distance as possible, so a group could discuss them while strewn about the conference table or pinned to the wall. Sometimes we’ve played games with them, quizzing each other by holding up two cards and asking what user needs they have in common, or contrast how their needs are unalike.
5. We printed them out on card stock. Snip, snip.

(Extra credit would be to make two-sided cards with more stats about the user, if that’s helpful for you. You could accomplish that by producing two cards per person, print, then cut and paste them together. )

The next step is deployment of the tool: determining how many copies of which cards to make and letting your collaborators decide how to keep themselves reminded. Some groups stick them around their displays or cube walls, for example.

In each part (part 3 is in the works,) Leisa states a way you might make qualitative research “look more scientific,” and then discusses some reasons why trying to make qualitative research more quantitative is a waste of time, money, and energy.

1) Use a relatively large sample size

The richness of the information and insight you receive even from this small sample size makes the return on investment enormous – and the small sample size makes it an activity that almost any project can incorporate into their timeline and budget. At the end of the day – those things are far more important than scientific validity.

2) Ensure that your test environment doesn’t change

If you want to quickly weed out problems with your site/application/prototype – then I recommend that you fix the problem and move on to spend your valuable research time learning about things you don’t already know about. It will certainly keep you awake as you’re researching, you’ll get rapid return on investment and excellent bang for buck as far as research techniques go.

3) Ensure that your test approach doesn’t change (don’t change the script, and stick to it)

Perhaps you can use her arguments when someone asks you about the validity of your qualitative research.

You can read part 1 here: Embracing the Un-Science of Qualitative Research Part One – Small Sample Sizes are Super

You can read part 2 here: Embracing the Un-Science of Qualitative Research Part Two – Ever-Evolving Prototypes are Ace

Part 3 is on the way.

Question: How does usage-centred design help designers and developers focus on what is important?

Answer: The whole nature of usage centred design is all advised to focus on those relatively few things that have the greatest impact on getting a good user interface design. The fact that it starts out as the user role means that you have already narrowed your focus from users in the broader sense to the relationship with the system.

The task modelling using task cases keeps your attention on what it is your users are attempting to do from their perspective. If you do this modelling process it leads you straight forward from nowhere to a design that will work well for your users.

Question: How do developers go about learning good user interface design?

Answer: I think there are lots of different ways and it depends on the developer. I think its good to get hands on training from a university or one of the commercial organisations but then you have to practice. That means ultimately doing designs that see the light of day and getting them tested and deployed to real users. Doing exercises is good for a start.

Not everybody is cut out for visual interaction design, just like not everyone is cut out to code in C#. People who are good at visualising things and visual problem solving have a head start on doing user interface design.

You can read the full interview here: Developer Interview — Larry Constantine

The most valuable asset of a successful design team is the information they have about their users. When teams have the right information, the job of designing a powerful, intuitive, easy-to-use interface becomes tremendously easier. When they don’t, every little design decision becomes a struggle.

While techniques, such as focus groups, usability tests, and surveys, can lead to valuable insights, one of the most powerful tools is the field study. Field studies get the team immersed in the environment of their users and allow them to observe critical details for which there is no other way of discovering.

While field studies are one of the most expensive techniques to implement, the value they return is tremendous. We’ve never come back from a study thinking we’ve wasted our time and resources. A quality study can produce enough information to keep a team busy for months.

In this week’s feature article, UIE’s Ashley McKee has conducted an excellent interview with Kate Gomoll, a recognized Field Research expert. In the interview, Kate shares how she and her team at Gomoll Research & Design conduct field studies.

Read today’s UIEtips article.

Has your design team conducted field studies? How have they worked for you? As always, I’d love to hear what you’re doing. Join the discussion below.

[Whether you're new to the benefits of performing field research, or a seasoned researcher looking to brush up on your research techniques, you'll definitely want to get your copy of our latest UIE Fundamental Report: The Field Study Handbook -- A Common Sense Approach for Discovering User Needs, written by Kate Gomoll, Ellen Story Church, and Eric Bond.]

Our Product Management team at iRise is constantly reaching out to our potential customers as part of an ongoing “Customer Needs Analysis” campaign. We engage usability experts, interaction designers, information architects, business analysts and other software definition participants in observation sessions that let us get closer to our potential customers’ needs.

If you are interested in allowing someone from our Product Management team to spend a day unobtrusively observing you and talking to you over lunch, then please contact me at peter@irise.com. We are willing to consider any location in the U.S.

If we end up engaging you for an observation session, we will provide you with a gift certificate to Amazon.com or your favorite restaurant and a trial license for our iRise Studio simulation product.

I bet Peter and his team learns some great stuff.

Field studies aren’t hard or expensive. Are you thinking about them?

Improvements happen when the design truly eliminates frustration. Organizations benefit from those improvements when the eliminated frustration increases sales (due to a better user experience) or reduces expenses (due to decreased support or development costs).

One problem we often see, however, is when the user researchers get overzealous in their improvement suggestions. They start to make recommendations in cases where there may not be any clear evidence the change will eliminate frustration or improve the design in a measurable way.

The worst case scenario is when someone issues a poorly thought out recommendation, which turns out to make the design more frustrating. When this happens, it hurts the reputation of the user research effort and puts into question other recommendations.

At the recent CHI conference, our good friend Meghan Ede, who runs the user research effort at Symantec, told us she’s instructed her team of researchers to ask the following of every recommendation they write: “Would you bet your life savings on this recommendation improving the design?” They remove any recommendation which doesn’t meet this criteria from the final presentation.

Meghan reports, since she’s instituted the policy of asking this question for every recommendation, the number of recommendations has dramatically decreased, but the quality of their results have substantially improved. The researchers are now more confident when they report their results and the teams are less argumentative.

What do you think of Meghan’s approach?

Before the user arrives for a usability test, we gather the design team around and ask them to predict what they think that test participant is going to do.

We share with the team any information we know about the participant, often supplied from our recruiter. We talk about the tasks we’ll ask the participant to perform. Then we ask the team to tell us what they think is going to happen. Where will the participant get stuck? Where will they glide through the design? What do we think will delight them?

By asking these questions up front, we give the team members something to focus on. When a participant gets stuck somewhere they predicted, it stands out. They can take pleasure in knowing they identified a potential landmine.

In our experience, it stands out even more when the participant gets stuck someplace nobody expected. These opportunities allow us a chance to learn something new about how users approach our designs.

Exploring the similarities and differences between the team’s predictions and what happens during the testing can yield important insights for the team, leading to a much improved design.

I particularly took away the idea of the Elastic User. This is where stakeholders make statements about what ‘users’ want, what ‘users’ do, what ‘users’ prefer, and because the ‘user’ in that context is so undefined and broad, they are able to say almost anything they like and there is no real way to contradict that opinion.

The creation of personas means that user groups are much more defined, so broad sweeping statements about what users want can actually be tested against something. Rather than having a free pass to do anything to the requirements or design by just using the word ‘user’, these assertions can now be tested and validated against more closely defined user characteristics and goals.

You can read Leisa’s full article here: Yes, you should be using personas.

Steve’s book, The User Is Always Right: A Practical Guide to Creating and Using Personas for the Web, is a must have on your design bookshelf.

If your team is considering personas or would like to know how to make them better, you’ll want to gather everyone around for Steve’s seminar on the 24th. It’s a cost-effective way to get a good discussion going.

May 24th UIE Virtual Seminar: The User is Always Right: Making Personas Work for Your Website ($129—includes handout)

What’s the difference between science and art? Is usability testing more of a science, that produces predictable, repeatable results? Or is it more of an art form, where the skill and talents of the people involved will have tremendous influence on the outcome? These are critically important questions, as we try to bring techniques like usability testing into the mainstream.

In 1999, Rolf Molich challenged the usability world with the CUE-2 study. In this landmark experiment, he had 9 separate usability teams each evaluate the same interface: Microsoft’s Hotmail.

If usability testing is a science, we would expect that every team would produce essentially the same results by finding the same problems and reporting them essentially the same way. Certainly, as with any scientific process, we could expect that the teams would generally agree on the most serious usability problems and rate problems with similar scales.

But that isn’t what Rolf found. Instead, each team essentially had their own methods and, surprisingly, found their own set of problems. 75% of the problems reported were only found by a single team — many of them very serious problems with the interface.

When everybody does something differently, it means we’re closer to art than to science. And being close to art, it means that the people involved have more impact than the process or methods chosen.

Is this bad? Not really. However, I do think there is something good about moving our craft to be more scientific. And it’s research like the CUE studies that will help us do just that.

In this week’s UIEtips, we’re re-printing an article I wrote back in 2005 about how Rolf’s work can help us learn about usability testing best practices and hone our own capabilities.

As always, I’m interested in your views on this. How do you learn to do your work better? Have you come up with ways to learn new techniques and tricks? I’m always interested to hear what you’re doing to improve your results. Leave your thoughts and join the discussion below.

Read today’s UIEtips article.

[If you want the opportunity to compare your work to practitioners taking part in Rolf's CUE projects, then you should seriously consider attending his full-day seminar at UI12, Advanced Methods for Usability Testing. Under Rolf's careful direction, you'll walk through all aspects of conducting a usability test, from test scenario creation to reporting, to see what some of the best practitioners have done. You can read more about Rolf's session here: Advanced Methods for Usability Testing]

Livia Labate, over at I think, therefore IA, wrote a post on her experience creating a comic strip to demonstrate the core elements of a movie plot to other people. She got the idea for the exercise from Scott McCloud’s book, Making Comics, and thought she could use the technique in her information architecture practice.

I bought Scott McCloud’s Making Comics as soon as it was published and dug in right away. It’s always a joy to read his stuff, but this volume in particular struck a cord since it’s exclusively about storytelling and using comics format and techniques to convey a story. I love comics and graphic novels, and have long been interested in how I could use comics’ storytelling techniques in the information architecture practice.

You can read Livia’s entire post here.

If you have any experience using comics in your work, I’d love to hear about them.

Baecker, R. and Buxton, W. Readings in Human-Computer Interaction: A Multidisciplinary Approach. Los Altos, CA: Morgan Kaufmann, 1987.

Bauersfeld, P. Software by Design: Creating People Friendly Software. New York, NY: M&T Books, 1994.

Carroll, J. (Ed.) Scenario-Based Design: Envisioning Work and Technology in System Development. New York, NY: John Wiley & Sons Ltd., 1995.

CHI Conference Proceedings. Reading, MA: Addison-Wesley. (Proceedings published annually since 1983 by ACM’s SIGCHI.)

Hackos, J. and Redish, J. User and Task Analysis for Interface Design. New York, NY: John Wiley & Sons Ltd., 1998.

Holtzblatt, K. and Beyer, H., (Eds.) Communications of the ACM. May 1995, Vol. 38, No. 5. (Theme of the issue is “Requirements Gathering: The Human Factor.”)

Gardiner, M. and Christie, B. Applying Cognitive Psychology to User-Interface Design. New York, NY: John Wiley & Sons Ltd., 1987.

Gomoll, K. “Some Techniques for Observing Users” from The Art of Human-Computer Interface Design, Brenda Laurel, editor. Reading, MA: Addison-Wesley Publishing Company, Inc., 1990.

Greenbaum, J. and Kyng, M., (Eds.) Design at Work. Hillsdale, NJ: Lawrence Earlbaum Associates, 1991.

Laurel, Brenda, (Ed.) The Art of Human-Computer Interface Design. Reading, MA: Addison-Wesley Publishing Company, Inc., 1990.

Muller, M. and Kuhn, S., (Eds.) Communications of the ACM. June 1993, Vol. 36, No. 4. (Theme of the issue is participatory design.)

Norman, D. Design of Everyday Things (formerly Psychology of Everyday Things). New York: Basic Books, Inc. Publishers, 1988.

Stone, D., Jarrett, C., Woodroffe, M., and Minocha, S. User Interface Design and Evaluation. San Francisco, CA: Morgan Kaufmann Publishers, The Open University, 2005.

Schuler, D. and Namioka, A., (Eds.) Participatory Design: Principles and Practices. Hillsdale, NJ. Lawrence Earlbaum Associates, 1993.

Wixon, D. and Ramey, J. Field Methods Casebook for Software Design. New York, NY: John Wiley & Sons Ltd., 1996.

Present for the call were Jared Spool, DeWayne Purdy, Lyle Kantrovich, Kyle Pero, and Nate Bolt. You can meet the crew here.

You can fin the first episode and more about what’s in this episode here.

Here’s a feed that iTunes likes.

We’d love to hear what you think. Leave your comments on this page or you can write us at SpoolCast@uie.com.

Present for the call were Jared Spool, DeWayne Purdy, Lyle Kantrovich, Kyle Pero, and Nate Bolt. You can meet the crew here.

You can fin the first episode and more about what’s in this episode here.

Here’s a feed that iTunes likes.

We’d love to hear what you think. Leave your comments on this page or you can write us at SpoolCast@uie.com.

To improve the designs we’re creating today, we know that teams do best when they have all of the essential information about their users to make informed decisions.

In our experience, one of the most powerful ways to gather important insights about users is the field study. By making direct observations, design teams can identify opportunities they may have never discovered if they had only conducted usability tests, focus groups, or surveys.

I think it’s essential for all designers to really understand how to conduct a field study and learn how to gather critical information about users. That’s why in this week’s UIEtips we’re reprinting an article I wrote back in 2003, where I discuss the unique power of field studies.

Read today’s UIEtips article.

Has your design team conducted field studies? How have they worked for you? I’d love to hear what you’re doing. Leave a comment and join the discussion below.

[We are so excited about this topic that we've invited Kate Gomoll, a recognized Field Research and User Experience expert, to present UIE's next Virtual Seminar on March 20th. If you haven't had a chance to hear Kate describe the process of conducting a field study, I highly suggest you and your team attend. Space is limited so sign up today]

We’ve invited Kate Gomoll, one of my most favorite people in the entire world, to talk on a topic she knows tons about: conducting field studies.

The virtual seminars have been a great way to get your team introduced to important topics and this is no exception. Kate will talk about the field studies process and its effectiveness for all types of projects. You’ll have a chance to ask her your most pressing questions. All at a very reasonable price (especially since it is the same price when you bring your entire team to listen to her)!

The session is filling up fast, so I highly recommend you plan on registering soon.

Details are here.

Present for the call were Jared Spool, DeWayne Purdy, Lyle Kantrovich, Kyle Pero, and Nate Bolt. You can meet the crew here.

In this episode, the SpoolCast crew convened to discuss:

• the impact of the pending Target Suit
• mixing automated and in-person testing techniques
• multi-variate testing
• designing tasks for testing
• moderated vs. unmoderated testing methods
• using Flash and AJAX on home pages
• the importance of validating inferences
• the interface paradigm of the Nintendo Wii
• the impact of new devices, such as the TiVo, Wii, and Guitar Hero on future interface design

We’ve divided the recording into three parts to make it easier to digest…
Part 2 is available here.
Part 3 is available here.

(Here’s a feed that iTunes likes.)

Sorry it’s taken so long to get this one out. It’s been crazy ’round here!

Production assistance on this SpoolCast from Brian Christiansen.

We’d love to hear what you think. Leave your comments on this page or you can write us at SpoolCast@uie.com.

In 1988, UIE started with the goal of conducting usability tests for clients. For many years, we made quite a good business of this.

Now, it feels like we do our best to convince people they should be conducting tests on their own, even those folks who want to pay us. Don’t get me wrong: we love to conduct usability tests. We think they are great fun and we learn something every time.

That’s just it. We learn things the clients should be learning themselves. In a typical project, if we’re lucky, we might communicate half of what we learn about the client’s users, designs, and team members back to the client. In our minds, that’s too much waste of important information, left in our minds and not integrated into the thinking of the client.

In this week’s UIEtips, I catalogue some of the things a team learns when they put together their own usability tests, starting with recruiting and ending with the analysis. Of course, in a short article, I couldn’t cover everything, but I think I nailed the important stuff nobody ever talks about.

Read today’s UIEtips article.

Do you conduct your own usability tests? Have you learned important things from your tests? We’d love to hear your insights. Leave a comment and join the discussion below.

[If you're thinking about conducting your own usability tests, you'll definitealy want to take part in Friday's UIE Virtual Seminar, Demystifying Usability Tests: Learning the Basics, by Christine Perfetti. It's the perfect way to introduce testing to your team. Space is limited so sign up today ]

His initial solution was to hire a consultant to run some usability tests, gather the essential information, write a report, and present it back to the team.

I had a different idea: I suggested we train the development team to do their own testing. In my 28 years of experience of doing this work, I’ve found there is no single experience more educational than conducting usability tests.

Sitting next to a real user while they do real work is always an enlightening experience. I have yet to sit in on a test where I don’t learn something, even if what I learn is we’ve done a good job at nailing the design to meet the user’s needs. I learn the words the user’s like to use, the way they like to approach the problem, and where the design succeeds and fails at helping them.

Watching 10 real users is always an eye opener. Seeing patterns become enlightening. Seeing what doesn’t happen is just as useful. (“Look, none of the 10 users showed any interest in that fancy widget we spent so much time on!”)

Once you have a little training, conducting tests are fairly simple and straight forward. (As I say often, usability testing is not rocket science. We know this because NASA is one of our clients and they are very specific as to what they call rocket science. This is not it.)

Plus, once testing is embedded into the team’s regular process, it becomes a great way to try out new ideas and collect some actual data, instead of the usual opinion wars. There’s nothing like having a test scheduled and closing off an endless design debate with “Let’s see what the users say on Wednesday.”

I’ve seen a lot written about usability testing over the years, but I can’t recall seeing anyone talk specifically about it’s value as a training tool, to bring the entire team on the same page about who the users are and what they are trying to accomplish. Stretch your thinking of usability testing as a design validation tool or an idea generator into a team education technique. You won’t regret it.

The following is a list of sample questions we created for a project exploring new development tools:

Goals:

• What is the user trying to accomplish?
• How will the user know when they are done? What will be different?
• How does the user describe their goals?
• How do the user’s actions fit into the objectives of the organization?
• Who established the goals for the user? Were they self anointed or were they assigned by someone else?
• Are the user’s immediate goals part of a larger scope? (For example, the new point-of-sale application is one piece of delivering an entire new line of business.)

Process:

• What are the steps the user will follow?
• Who defined the steps?
• How prepared is the user for each step? (Do they have it all laid out or does it seem to be ad-hoc?)
• How does information flow from one step to the next?
• What are the various roles (such as creator, contributor, editor, or approver) that are involved?
• How long does the process take?
• What artifacts (such as design documents, emails, or whiteboard drawings) are used?
• How do the various team members communicate with each other?
• What other tools are used during the process?

Inputs & Outputs:

• What materials and information will the user need to successfully use the interface?
• From whom will they get that information?
• What do they do when the information isn’t complete?
• What will they need from the interface to continue with their overarching goals?
• To whom do they give those results?
• What happens after they’ve turned them over? (Does the user move on to something else or do they have more interactions?)

Experience:

• What similar things has the user done in their past?
• Is this something that repeats itself or is the use a first-time occasion?
• What journals or magazines do they read?
• What kind of “organizational memory” helps the user avoid mistakes of the past?
• How has the organization survived without this design in the past?
• What competitors systems have users taken advantage of?
• How will the user learn how to use the tool?
• What training has the user received?
• What conferences has the user attended?

Constraints:

• What physical, temporal, or financial constraints are likely to affect the user’s work?
• What ideals are subverted by reality as the work progresses?
• What constraints can the user predict in advance? What can’t be predicted?

Physical Environment:

• How much room does the user have to work?
• Do they have a place to store the documentation?
• What materials are on their desk?
• What access do they have to necessary information (such as user manuals)?
• What have they taped to their monitor?

Tools In Use:

• What hardware and software does the user currently use?
• Do they participate in online forums?

Relationships:

• What are the interactions between the primary user and other people affected by the tool?
• Does the user interact with other people who use the tool?

Of course, you’d want to change these questions to fit your project’s needs. After every two or three visits, we come back and revisit the questions, often updating them to include new areas we’d like to learn more in future visits. (We call these meetings focus redefinition sessions and we schedule them when we’re putting together our visit schedule.)

Bonus: Once we have our questions, we play a little game. Before we visit a study participant, we gather all the information we collected through the recruitment process about them and guess what the answers to the questions will be. Then, when on the visit, we compare our guesses to what we discover during the visit.

By playing this game, anything different from our guesses really jumps out at us. Interestingly, anything that matches our guesses also jumps out at us. It helps us see how real people are different from our expectations (a good thing — it justifies the site visits) and where we’re good at predicting what is happening in the real world.

I guess that’s what interested me when I read this in yesterday’s New York Times article, Stirring Up the Cubicles at eBay (Registration may be required, the story may hide behind their pay-wall in a few weeks.):

[John Donahue, deputy to eBay's CEO and president of eBay Marketplaces, one of eBay's most important divisions] accompanied two members of eBay’s research group to the San Jose apartment of Kanvasi Tejasen, a 30-year-old Lockheed Martin engineer who had agreed to have her online buying habits studied by the company in exchange for $200.

With Mr. Donahoe (who makes $800,000 a year and has received around $10 million worth of eBay stock) sitting on her sofa taking notes, Ms. Tejasen shopped for a TV tuner and visited rival sites like Amazon and Google. In one crucial moment, she plugged the term “4G iPod Nano” into the eBay search engine and received 1,700 results, which she said she found confusing. That set Mr. Donahoe scribbling furiously.

“We have to do a better job getting her what she wants,” he said afterward. “If we improve search efficiency even 1 percent, it’s worth hundreds of millions of dollars.”

What would happen if your CEO went on a few site visits?

“Lots of people were talking about how wonderful usability studies were” Rolf points out. “I tended to agree with them because I had seen myself the enormous political impact a usability study can have. But then I was curious and said, ‘Are they reproducible?’”

He shared his thoughts on a usability list serv, got four volunteers, then launched CUE-1. This study looked at a Windows application, having the participants test it in any way they saw fit to try and identify usability problems.

The results were presented at UPA 1998. Though “everyone in the crowd thought it was interesting,” there was also some resistance, something that Rolf would become used to over the years. He notes, “People said, ‘This just can’t be true. The results just can’t be this diverse.”

…

Undoubtedly, the most important finding of the CUE studies has been the diversity of the results. In explanation, Rolf points to the simple number of usability problems in the average system: “If you have 500 problems to choose from on a non-trivial website, and the usability test basically finds 40 problems more or less at random, then it’s no longer a surprise when you get these different problems.”

What’s the solution? Interestingly, Rolf doesn’t see it as increasing the number of users, or other methodological changes. “The solution is a much more radical one,” he states firmly. “It’s prevention. Many of the usability problems we have seen should have not occurred on the website in the first place.”

Read the article.

When we show a work-in-progress (like an alpha release) to the public, press, a client, or boss… we’re setting their expectations. And we can do it one of three ways: dazzle them with a polished mock-up, show them something that matches the reality of the project status, or stress them out by showing almost nothing and asking them to take it “on faith” that you’re on track.

The bottom line:
How ‘done’ something looks should match how ‘done’ something is.

This is very similar to what Bill Buxton said last month, when he talked about sketches and how they change the feedback the designer collects.

Carolyn Snyder’s book on Paper Prototyping is a required read on this topic too. (And the topic of an upcoming UIE Virtual Seminar.)

I think they are extremely useful, but not for all people in all situations. In specific, I’ve found personas to be very important under the following conditions:

1. The design team is an actual team, with more than a single individual working the entire process from ideation through implementation (and beyond).
2. The team members are different from their users (which is most of the time).
3. The team members do not have constant interaction directly with the users, particularly getting feedback on how the relevant artifacts are used.
4. Different users will interact with the artifacts differently because they have different intentions, context, knowledge, skills, or experience.

The well-executed persona description helps the team work “on the same page,” when it comes to understanding who their users are. It can eliminate the confusion and wasted efforts that come when team members are walking around with different ideas of who their users are. (See Yahoo’s Approach to Keeping Personas Alive.)

The well-executed persona description enables successful role-playing and story-telling for intra-team communication and for inter-group understanding of the design goals and objective. (See Three Important Benefits of Personas.)

The well-executed persona description helps the team members fit the design solution against the attributes which make one persona different from another, to ensure they’ve not excluded activities or impaired actions because they were ignorant (or forgot) about a subtlety of use. (See 5 Things to Know About Users.)

I see a major role for personas to be dissemination of information about users to others in the organization. When well executed, the entire organization understand who the design is for and the subsequent design rationales.

One of the greatest challenges we face in the design field is communicating design decisions to other stakeholders (e.g. Business unit, Marketing, Engineering). We’re often forced to attempt this through a requirements document. Personally, I’ve yet to come across a requirements document that is usable and doesn’t take a couple of days to get everyone on the same page. So, we use something different – a task analysis grid.

Each column starts out with a scenario, describes a task and is followed by all the sub-tasks necessary to complete the task. The sub tasks are colour-coded and prioritized from 1 (must haves) to 4 (some day in the future).

His premise is an interesting one:

It’s hard to describe what design is because it crosses so many boundaries: fashion, architecture, interaction, and mechanical, to name a few.

But it’s easy to talk about what all those different types of design have in common. One thing is sketching. While a clothing designer is trained very differently from an architect or an industrial designer, they all learn to use sketches as basic starting point.

Buxton asserted sketching was a fundamental activity to ideation. It is a quick way to play with an idea. And it communicates the proper stage of the idea to the viewers. Early, rough sketches just scream, “This is an idea! I’m not done!”

Buxton talked about a study of sketching for traditional design disciplines that showed all had common attributes:

• They are quick to make and timely to talk about the idea
• They are inexpensive and easy to dispose of (making designers less “wedded” to a particular idea because of investment)
• They are plentiful (designers should bring many different ideas-as-sketches to the table, not just one)
• They have a clear vocabulary (such as drawing through the endpoints to show the “unfinishedness” of the idea)
• They use no higher resolution than necessary (so they don’t waste designer’s time and effort in preparation)
• Their resolution does not suggest they are further along than they really are (to avoid giving the impression of being more done than reality)
• They suggest and explore instead of confirming (to support ideation, instead of forcing decisions)

Buxton then suggested “Since Experience Design is a type of Design, it too must have sketching. However, traditional sketching doesn’t work well to represent interactions, so what would sketching for interactions look like?”

Ideation vs. Evaluation

Buxton suggested ideation is a different process than evaluation. In ideation, the goal is to come up with many different ideas, using each idea to suggest others. In evaluation, the goal is to narrow down the choices of ideas, honing in on the best idea. He suggested today’s usability process is evaluation, while ui design is ideation, an idea I agree with.

He made it clear that both ideation and evaluation were necessary.

“It’s like saying there are both girls and boys. One isn’t necessarily better than the other, but both are required and it’s important to know the distinction.”

Ideation has to come first. You generate ideas, which you will subsequently evaluate. So, Buxton suggested that sketching has to come before prototyping.

Sketching vs. Prototyping

The attributes of each are different:

The idea is you create many sketches at first, then narrow them down, until you’re ready to build your prototype to evaluate with. Because sketches are more lightweight and cheaper than prototypes, they are easy to play with and throw away. When you’ve explored the idea space sufficiently, then you eliminate ideas to a basic few, which you then prototype out with the rigor necessary to evaluate.

“It’s about making many good mistakes. I want to have brilliant mistakes.”

This was just a subset of the great ideas in the 90-minute presentation, but I thought the idea of sketching was a brilliant take on the ideation process I hadn’t heard before. (At least, not quite this way.) If you get a chance to hear Bill speak on this subject, I highly recommend it.

Update: Nick shared this link where you can watch a recording of Bill’s presentation.

Present for the call were Jared Spool, DeWayne Purdy, Lyle Kantrovich, Rashmi Sinha, Nate Bolt, and Josh Porter. You can meet the crew here.

You can find the first episode and more about what’s in this episode here.

Here’s an RSS feed that iTunes likes.

We’d love to hear what you think of this episode. Leave your comments here or send us an email at SpoolCast@uie.com.

Present for the call were Jared Spool, DeWayne Purdy, Lyle Kantrovich, Rashmi Sinha, Nate Bolt, and Josh Porter. You can meet the crew here.

You can find the first episode and more about what’s in this episode here.

Here’s the final part of this episode. (Here’s an RSS feed that iTunes likes.)

We’d love to hear what you think of this episode. Leave your comments here or send us an email at SpoolCast@uie.com.

Present for the call were Jared Spool, DeWayne Purdy, Lyle Kantrovich, Rashmi Sinha, Nate Bolt, and Josh Porter. You can meet the crew here.

More about what’s in this episode here.

The third part is available here. (Here’s an RSS feed that iTunes likes.)

We’d love to hear what you think of this episode. Leave your comments here or send us an email at SpoolCast@uie.com.

Present for the call were Jared Spool, DeWayne Purdy, Lyle Kantrovich, Rashmi Sinha, Nate Bolt, and Josh Porter. You can meet the crew here.

In this episode, the SpoolCast crew convened to discuss:

• Individual and group card sorting techniques
• Incorporating group elements into individual usability testing
• Collaborative recruitment and social collaborative tools
• Accommodating testing on a social scale
• The state of UX organizations across the country
• Whether World Usability Day is helpful or harmful to the community in the long run

We’ve divided the recording into four parts to make it easier to digest.
Part 2, Part 3, and Part 4 are now available.
(Here’s a feed that iTunes likes.)

Production assistance on this SpoolCast from Brian Christiansen.

We’d love to hear what you think. Leave your comments below or you can write us at SpoolCast@uie.com.

The usability affinity grid is comprised of 4 levels. Each level builds on the others, moving from business goals to usability goals. Talking about a project in terms of a usability grid helps large or dispersed teams and their stakeholders get on the same page, agreeing on the value provided by a focus on usability in the organization.

Here are the levels, and how they build on each other.

1. Business Goals
   Business goals are the goals that the business needs to reach in order to be successful. These are often very straight-forward, but difficult to achieve. One big goal of many businesses is repeat revenue, getting revenue from folks on a recurring basis. Magazine subscriptions are a great example of recurring revenue occurring on a yearly basis.
2. Issues
   Issues are the problems that arise during daily operation and directly affect business goals. In a call center, for example, the biggest issue is the hold time for incoming calls. As hold time increases, customer satisfaction goes down, and customers become frustrated and angry, making business goals more difficult to attain. We’ve all had the experience of being on hold and having a pseudo-pleasant voice promising us “Your call is important to us”. Argh!
3. Business Objectives
   Business objectives are objectives that, if reached, will solve the outstanding issues of the organization. In the call center example, the obvious business objective is to reduce customer queues. This objective doesn’t lead to revenue directly, but indirectly.
4. Usability Objectives
   If one of your business objectives is to reduce calling queues, then a usability objective might be to enhance the productivity of the call center operators. Designing to support efficiency of use of the call center operators would directly reduce the time it took to handle each call, which would directly address the issue and thus the business goals in the end.

As Sarah and Susan point out, this grid is simply a tool to help design teams within an organization. Some teams already use this sort of reasoning implicitly, without mapping out these levels explicitly. But for those teams who are still struggling with communicating the value of usability, the usability affinity grid can prove invaluable.

This was a really fun conversation. Some of my favorite bits are:

Kyle Pero’s idea for her dream panel at a conference

I think we need to ask ourselves if we’re growing and improving our service based on what the clients want – or what we think they want – doing a little bit of our own user research I guess. I believe that our clients, and not the industry, should definitely be setting our standards, and I don’t know if we’re doing that. A panel of clients discussing their needs would be quite interesting to me, this is just my opinion. I know every one of us works on different products – some of them offline, some of them online – but basically, the need is the same. Can the products be used easily to accomplish the task? I think if we just take a moment in one of these conferences and just stop listening to each other for a change and start listening to who we are servicing I think it would be pretty interesting and different.

Rashmi Sinha on whether usability studies are scientific or not

Rashmi: But anyone who thinks that this is science, hasn’t done science! I have done science, and this ain’t science, by any stretch of the imagination. I disagree with the whole notion of trying to make it scientific because it isn’t scientific. Second, you have a big problem…

Jared: Making what scientific? The study itself, or the field?

Rashmi: The field is not scientific, and now people have this big notion from experimental psychology, but really it’s not. Experiments have – there’s a way of doing them, and it’s very hard to do that in the field.

Lyle Kantrovich pipes in on why Rolf Molich’s CUE studies are really interesting

Jared: But this brings us back to what Kyle Pero were saying. Because, I think, what is a key element of this is that the client doesn’t realize that they have different results depending on which team comes through. We’re not pitching it that way. Let me put it another way. When Ralph ran CUE-4, he had 17 teams look at, I’m trying to remember, oh it was…

Kyle: Hotel reservations.

Jared: Hotel reservations system, the U Penn system. Not the Penn, but the Hotel Pennsylvania or whatever it was, I don’t remember what the hotel was. But he had people look at this hotel thing that was done by the iHotelier folks, and the 17 teams, approximately half of them, did heuristic evaluations, half of them just happened to work out to do usability tests. They found, across all the teams, they found 61 errors, 61 problems with the design, that the client, the iHotelier people, thought were critical designs, critical problems. Things that they said, these are things we definitely have to fix. But each one of those 61 problems was only reported by one team. So in order to have collected all 61 problems, if the iHotelier people, Jim Whitney and the folks at Webvertising, were to hire a group to do it, they’d have to hire all seventeen teams to get those results.

Lyle: Well, this is why I think those studies are pretty interesting: I think that, by doing these kinds of studies, we will learn what helps us find more problems, what helps us find more critical problems, what we mean by those things. But also, frankly, if you’re iHotelier and you hire only one team, and you find, say, ten critical problems, aren’t you still in a better place? So I don’t think it brings in a question of value or the validity of these studies; but these will help us refine our art, our craft. I don’t think it’s a science: I think what we do is craft; it’s part science and part art. I think it gets better.

Kyle: What the studies really show is that it really does matter whom you hire.

Jared: Exactly.

Rashmi and Nate Bolt discuss more about the scientific nature of our work

Nate: Well I was just thinking about the discussion we were just having and it almost sounded like there was an implicit assumption that if you were a scientist that you follow certain rules and that you should come up with the same results as everyone else. But that would also lead to the idea that all scientists are created equal and we know that that’s not true because scientists like Darwin and Kepler, Galileo, they all had something that no one else had. They had this sort of insight to translate what they were seeing and the math behind it into something new. I would just add to that, before we ultimately kill it, I would just add that all scientists aren’t created equal either and we really can’t get away from the person that’s doing the process in the discussion that we’re having.

Rashmi: Correct. When you teach courses and you do even basic scientific experiments and you have every student in the class replicate that, you do find deviations. They might be deviations around some kind of thing that might form a normal distribution that there’s this one most likely result and then there’s deviations from that. But the method and the way that people carry out the method is different every where. And also in science there’s an explicit concept of the meta kind of research where you look a bunch of the different studies on a topic and you make some logical conclusions. So that once again acknowledges that not every result done addressing the same exact question is going to come up with exactly the same answers.

Josh Porter on the Facebook Controversy

Josh: Well I think Rashmi makes a great point, that this is kind of a new sort of social design issue. The really ironic thing that I just read about this morning was that the actual implementation of the feature itself caused its own downfall. Because as you were looking on your Facebook profile, you could see that all the other people, all the other people in your network, were signing up for the protest petition. So that’s how that petition was signed by like 700,000 people in 24 hours, because the new feature itself propagated it so fast, so I found that really interesting. But one thing that I don’t think any of us have mentioned as well is that the privacy settings of everyone’s information actually stayed the same. So when, if you were seeing information in your news feed, that was information that was already available to you. The only difference is you had to work much less hard to see it. It’s kind of like moving from looking at HTML home pages to going to RSS. There’s just that much less effort in seeing information.

Jared: The MoBuzz TV people, said that this is like people who, somebody who stands naked in their window every morning, who gets upset because someone handed them a photograph of them standing naked in their window.

Read the entire transcript here.

If you watch users long enough, you’ll notice that they rarely move their mouse until *after* they’ve decided what they want to click on. So, if the rollover contains any information they would need to make the right choice, it’s likely they’ll choose wrong.

Test two designs: one with rollovers and one without (with the same content always visible on the page). Almost always, the users will perform and prefer the latter.

More info: Users Decide First; Move Second

Andrew Boyd from Humane Information Architecture responded:

I’ve spent a lot of time watching people use web interfaces, and some of them definitely do use the mouse pointer as a guide when they are browsing around the page – they do look for rollovers and tooltips prior to making a click decision. Not all of them, but enough so that meaningful tooltips are worthwhile. On three separate occasions in the last week-and-a-half on the road watching other people test a design I have contributed to, we’ve had comments to the effect that the tooltips were not meaningful enough. This is all anecdotal, of course, but I’ve seen people move the mouse prior to making click decisions (rollovers or no).

I agree you’ll see some people move their mouse. However, It’s important we don’t confuse observation with inference.

Andrew’s observation (based on what he wrote) is he’s seen users move the mouse over items to explore them. He’s also observed users commenting that they felt tooltips weren’t meaningful enough.

Andrew’s inference (also based on what he wrote) seems to be he believes the problem is the tooltips need improvement and this would benefit those users who explore with their mouse.

I don’t disagree with Andrew’s observations. I wasn’t there and have no way of knowing what users did or didn’t do. Plus, I too have seen some users move their mouse around the screen, waiting for the browser to give them the “finger” and looking for additional clues.

However, I do think his inference (as I interpreted it) could be faulty. From what he wrote, I don’t think he asked enough questions:

• Why were the users looking for rollovers and tooltips in the first place?
• Do those same users always look for rollovers and tooltips regardless of the page they’re on?
• Or is it the function of the page’s design that suggests they should do that?
• If you increased the effectiveness of the on-screen design, would it eliminate those users’ desire to move their mouse?
• What happens for those users who need to scroll and the mouse is needed to control the scroll bar?
• Do they have more problems making the right choices than users who end up moving their mouse?

It could be that his users are more likely to move their mouse (thereby suggesting he invest more in improving tooltips and rollovers).

It could also be that all users are more likely to move their mouse on his page, because his design demands it to succeed (thereby suggesting he invest more in the on-screen page design and reduce the need for tooltips and rollovers).

What observations could Andrew have collected to figure out which it is? How could he test to see which is more likely?

It’s important we ask the right questions before we draw our inferences from the observations. Different inferences will lead us to different design solutions. We need to ensure our inferences are leading us in the right direction.

Present for this recording were Jared M. Spool, DeWayne Purdy, Kyle Pero, Lyle Kantrovich, Rashmi Sinha, Nate Bolt, and Joshua Porter. You can meet the crew here.

More info about what’s in this episode here.

(Here’s an RSS feed that iTunes likes.)

We’d love to hear what you think. Leave your comments here or you can write us at SpoolCast@uie.com.

Present for this recording were Jared M. Spool, DeWayne Purdy, Kyle Pero, Lyle Kantrovich, Rashmi Sinha, Nate Bolt, and Joshua Porter. You can meet the crew here.

More info about what’s in this episode here.

I’ll have Part 4 available tomorrow. Part 4 is now available here.
(Here’s an RSS feed that iTunes likes.)

We’d love to hear what you think. Leave your comments here or you can write us at SpoolCast@uie.com.

Present for this recording were Jared M. Spool, DeWayne Purdy, Kyle Pero, Lyle Kantrovich, Rashmi Sinha, Nate Bolt, and Joshua Porter. You can meet the crew here.

More info about what’s in this episode here.

Part 3 is available here. (Here’s an RSS feed that iTunes likes.)

We’d love to hear what you think. Leave your comments here or you can write us at SpoolCast@uie.com.

Present for this recording were Jared M. Spool, DeWayne Purdy, Kyle Pero, Lyle Kantrovich, Rashmi Sinha, Nate Bolt, and Joshua Porter. You can meet the crew here.

In session, the SpoolCast Crew convened to discuss a variety of topics:

• What would our dream panel at a conference be?
• Do the Comparative Usability Evaluation studies really tell us anything about the discipline of usability research?
• Is usability practice an engineering discipline or a craft?
• Is there value to conducting heuristic evaluations?
• Is it worth discussing how many users should be in a usability test?
• What can we learn from the recent Facebook controversy?

I’ve divided the recording into four parts to make it easier to listen to while commuting.
Part 2 lives here.
Part 3 lives here.
Part 4 lives here.
(Here’s a feed that iTunes likes.)

The transcript is available here.

I think I’ve solved the volume problem, having discovered how dynamic range compression can be my best friend. I’m sure you’ll let me know if I got it wrong.

We’d love to hear what you think. Leave your comments below or you can write us at SpoolCast@uie.com.

Gerry is an excellent interviewer, but it’s not hard to get Donna to talk about the subtleties of this age-old technique. Some of the more interesting bits are when she talks about why she never does closed sorts to validate the IA (she’d rather ask the users to find information in the hierarchy) and why she doesn’t like dendrograms and cluster analysis (she thinks if you’re doing your job right, these techniques add little value).

If a card sort is in your future, this 15 minute podcast is a must-listen! I can’t wait until her book is done.

To prepare:

Collect up 1 blank index card for each observer. On each card, write down one of these four sentences, such that you have an almost distribution across the number of cards (so, if you have 12 people being trained, you’ll have 3 cards with each sentence):

• Group 1: Things that are blue
• Group 2: Things that are round
• Group 3: Things that roll
• Group 4: Things that make noise

You’ll need a flip-chart or whiteboard and appropriate markers during the training session.

To use:

1. During the training, pass out one card to each person. Ask them to look at it, but not share what it says with anyone else.
2. Tell them that, for the next 2 minutes, they are to look around the room and write down everything that matches the description on the card. They aren’t to share their lists with anyone else.
3. Give them the 2 minutes to write down their lists.
4. Divide a flip-chart page (or the whiteboard surface) into four boxes. In each box, write the numbers 1 thru 4.
5. After their two minutes, tell the group that you’re going to play a game. The people in Group 1 are going to list items they observed without divulging the category of the objects. Everyone else has to guess what the category was by the objects they observed.
6. Starting with group 1, encourage them to shout out their objects as you write them down on the flip chart. Once you’ve gotten a 4 or 5, ask the others what they think the category is. Repeat with groups 2-4.
7. For the wrap up to the exercise, ask the entire group if there were things on their list they hadn’t paid attention to until they were looking around the room. Then ask them if there were things on other’s list they didn’t notice, until they were collected on the flip-chart.

The point is to show them how, when you have the right focus questions, observing specifics is much easier. When you don’t have any focus questions, you’re likely not to see something you need to see.

Next weekSoon, I’ll post a templateframework we use to develop our focus questions. Here is the framework.

Happy Labor Day! (For non-USers, Labor Day is a US holiday to celebrate the “working man” and, oddly, a day when we don’t work.)

Carolyn Snyder’s Paper Prototyping Book

This is a must-have book for anyone thinking about conducting testing. Carolyn’s done a great job talking about how to conduct usability tests, both with and without paper prototypes. She also walks through the steps to create a paper prototype, which we think is the best way to start any design project.

More details on the book here.

Other Paper Prototyping Resources

Using Paper prototypes to Manage Risk: An article by Carolyn Snyder about the steps to prototyping.

Five Paper Prototyping Tips: Quick tips about making prototypes out of paper.

Looking Back on 16 Years of Paper Prototyping: An article I wrote talking about our history with the technique and why we still think it’s a great way to do design.

Workshop: Product Usability Survival Technique: Christine’s very popular one-day workshop where you actually build and test a full working prototype. The team that has the most usable interface wins fabulous prizes!

5-Second Tests

5-Second Tests: Measuring your Site’s Content Pages: An excellent article Christine wrote on how to conduct a 5-second test.

Inherent Value Testing

Inherent Value Testing: An article I wrote describing the benefits of this technique.

Conducting Inherent Value Testing: Apparently, I got long-winded when I originally wrote the article above, so we broke it into two parts. This part walks, step-by-step, through the process of conducting an inherent value test.

Field Studies

Field Studies: The Best Tool to Discover User Needs: An article I wrote discussing the value of conducting field studies.

Designing Products That Work the Way People Work: This is a great interview that Christine did with Kate Gomoll a few years back, talking about how they use field studies in their work.

Book: User and Task Analysis for Interface Design: A great book by JoAnn Hackos and Ginny Redish (both former UI Conference speakers) about conducting field studies to determine who your users are and what they do with your design.

Book: Contextual Design: Karen Holtzblatt and Hugh Beyer’s (also both former UI Conference speakers) landmark book about contextual inquiry — a more formal field study methodology.

Book: Rapid Contextual Design: Karen’s follow-on book. (I haven’t read it yet, but several people have told me it’s excellent.)

Category Agreement Analysis (CAA)

The CAA: A Wicked Good Design Technique: A quick summary of what a CAA is.

Personas

Personas: Matching a Design to the Users’ Goals: Christine’s first article on how to design with personas.

Three Important Benefits of Personas: An article I wrote talking about some oft-ignored benefits that personas bring to the organization.

Getting from Research to Personas: Harnessing the Power of Data: An article by Kim Goodwin describing how to take research data and use it to build personas.

Book: The Persona Lifecycle: Keeping People in Mind Throughout Product Design: An excellent book by Tamara Adlin and John Pruitt describing the process of building personas and utilizing them in the design process.

Recruiting Participants for Studies

Report: Recruiting without Fear: A report we wrote last year to help you manage the recruitment process.

One last book:

Observing the User Experience: Ex-Adaptive Pather, Mike Kuniavsky, has done an excellent job of compiling many of the most important techniques into a single volume. Another must-have for the bookshelf.

Christine also mentioned during the presentation that we like to use Techsmith’s Morae. It’s not the only way to record a usability test, but it’s a darn good solution.

According to the article, Michael DeHaven, who is in charge of the site’s ecommerce marketing efforts, used A/B testing to try out two different versions of the new design.

Version “B” looked like this:

Version “C” looked like this:

What it came down to was this (from Version “B”):

versus this (from Version “C”):

The article reports how everybody on Mr. DeHaven’s staff who saw version “C” said it was too crowded and people would feel overwhelmed with links, so they expected version “B” to win the A/B Test. As we would’ve predicted based on our work with link-rich home pages, version “C” should do better.

And we were right. Version “C” won more customer accounts than “the cleaner, more graphical design”.

But what really interested me about the article was the reason that Mr. DeHaven gave for the surprise success of version “C”:

It seems that business executives prefer to look at fairly plain textual content online rather than cheerful graphical interfaces. Plus, they prefer vertical to horizontal groupings of options and longer, wordier textual click links.

“After I thought about it awhile, it made total sense,” explains DeHaven. “Users are trained to allow their eye to scan down something that looks like search results. And what we have here now is something that looks an awful lot like search results. It’s a good quick scan. Graphics and images are not what the eye’s trained for online.

“It was a HUGE finding. Really exciting.”

Wow! What an inference! Mr. DeHaven got the observation right — people bought more when presented with version “C” — but I don’t agree with his inference at all.

Jumping to poorly formed inferences can get us into trouble. Now that he believes that (a) people prefer things that look like search results and (b) users’ eyes’ are not trained for graphics or images, he’s likely to make design decisions based on those beliefs. And, while many of the resulting designs are likely to work, many probably won’t.

Part of the problem comes from exclusively using tools like A/B tests. A/B tests tell you that one version performs better than another, but doesn’t tell you why. So, the designer has to guess at why. And, in my experience, they often guess wrong.

It’s not just A/B tests that do this. Inspecting your web logs will produce the same results. You’ll see a bunch of folks are ending up on a certain page. But you can’t tell if they are there because they want to be or because they are lost. You have to guess as to why they end up there. And you’re likely to guess wrong.

Had Mr. DeHaven conducted usability tests or interviewed users, in addition to the A/B tests, he probably would’ve come to a different conclusion about why users prefer the page with the extra links. I’m guessing, based on our research, he would’ve found out the users found the pages to have more trigger words in the right places.

Whatever he would find out, it is likely he’d be more informed about the decisions he was making going forward. And, in the long run, he’d have more information to work with when making future decisions about his designs.

Several people asked about where trigger words come from. One asked, “So, using the appropriate trigger words on the home page very much depends on knowing the user’s intention?”

Absolutely.

Trigger words are the words or phrases on a web page that trigger the user into acting. If you don’t know your users’ trigger words, you might still make an easy-to-use page, but it will be by chance. Knowing your users’ trigger words is the only way to ensure a great page every time.

To discern the trigger words, you need to know why users are coming to the page in the first place. When talking about a site’s home page, that means you need to know why they are coming to the site and what they hope to accomplish.

We learned a long time ago the days of visiting a web site to “see what’s there” are long gone. Nobody says, “I’ve got 15 minutes to kill, I’m going to go check out the GE web site.” If they are going to a site, it’s because there’s something they intend to accomplish. You need to know what that something is.

The only way I know to determine why people visit a web site is to talk to them. Sometimes you can guess, like when Ralph Lauren advertised their new line of furniture in a 2-page spread in the New York Times Sunday Magazine with the only words in the entire ad being “Ralph Lauren Furniture – www.polo.com.” It wouldn’t be hard to guess that many users came to the site to see more about the new furniture. (Interestingly enough, they didn’t bother to include any information about the new line of furniture on the site, confusing users immensely.)

But, mostly, you need to talk with users. Why do people go to a university web site, like Olin College? Are they a high school student, thinking about attending? Are they a parent wondering about tuition and financial aid? Are they an existing student? A faculty member? An alumni? Probably all of the above.

But for each person, you could start to identify their intentions for visiting the site. What does a parent need to know? What words will they use to find that information? “Tuition,” “Costs,” “Scholarships,” and “Financial Aid” are probably some of the trigger words required.

The ADMISSION menu on the Olin College Home Page uses Costs and Financial Aid as trigger words.

Starting with a good set of Personas and the scenarios that bring those personas to the page in question will tell you a lot about trigger words you need. If you’ve done a good job of creating your personas, you will find it easy to generate a list of trigger words.

Sam divides the business rationale for good design into three pieces:

• Designing to Simplify
• Designing to Differentiate
• Designing to Innovate

I thought this was a nice, straightforward way to talk about why design pays off.

He also mentions HP’s use of deep ethnography and interviews to collect unmet customer needs. In addition, he talked about how they use internal newsletters from HP’s worldwide network of designers to keep everybody up to date on trends and design ideas.

Very interesting stuff.

One benefit, recently pointed out to me, was it is eye opening for clients to see how a user’s eyes bounce around the page. Clients believe that people read the content on pages in an orderly fashion. Seeing users’ gazes meander across the screen like a drunken sailor, hopping from one element to another, can convince the client their users don’t view the content the same way they do.

And Maria Stone, from the User Experience group over at Google, told me, of the two labs they use for usability testing, the developers prefer to use the lab equipped with the eyetracker because it’s more fun. She believes the developers pay more attention to the test when the little dot is bouncing around the screen.

I do agree that it’s always good when your clients and developers become aware of how users behave. And an eyetracker is a great demonstration of fine grain behavior. In just mere moments, you can easily see how users gaze at the screen. So, I agree eyetrackers have demonstrative value.

But do they have diagnostic value? Can we actually learn what to change in our designs from them?

Well, after watching hundreds of eyetracking tests, I can tell you it’s still really hard to know what you can learn from them.

First, they are expensive devices. It’s not cheap to outfit a lab with a decent eyetracker. (Even Google, which as far as I can tell has all the money in the world, has only outfitted one of their many labs.) In addition to the money spent on the equipment, you have to spend money training people how to use it. Not a cheap proposition.

Second, not every participant can work with an eyetracker. Depending on the hardware, people with a variety of attributes automatically are disqualified from eyetracking. Everything from contact lenses to long eye lashes can get in the way of the device working properly.

Third, they reduce the amount of time you actually collect data from your users. Getting a participant set up and calibrated with the device can take time away from learning about your design. The most valuable piece of any usability test is the time the participant is interacting with your design, not setting up the measurement equipment. What’s worse is many devices lose calibration quickly, forcing the test to stop and the participant to spend more time futzing with recalibrating. This tool time is distracting and not adding to the session’s value.

Fourth, the results are really hard to analyze. The colorful heatmaps are cool (or warm?) to look at, but what are they actually telling you? When someone is gazing at something, is it because they want to look there? Or because the page made them look there? Or because they are resting their eyes there?

When we first started conducting eye tracking, we noticed some interesting behaviors:

• Participants often would acquire the scroll bar without looking at it. They’d move their mouse over to the right edge of the screen and start scrolling, but their gaze wouldn’t leave the center of display. It seemed they were using their peripheral vision to acquire and use the scroll bar.
• Participants would orally tell us they couldn’t see something their gaze was focused on. (Women in my life have referred to this as “Male Refrigerator Blindness” — the inability to see something right in front of you.)
• Participants often would click on objects they barely gazed at. They’d focus their vision on some part of the screen, then move their mouse to some place else to actually click.

From this, we began to question what the eyetracker was actually trying to tell us. It seemed to us that what the user focused their gaze on was not necessarily what they were seeing. So, if the eyetracker doesn’t tell us what a user sees, what does it tell us? I’m not sure.

Eyetracker vendors go to great lengths to try to justify the value of their devices. For example, this vendor used eyetracking to show how changing a useful portion of the SFPD web site to a non-useful graphic encouraged users to spend more time focusing on the site’s navigation, claiming this somehow improved the site’s usability. But the vendor didn’t explain how focusing more on the navigation improved the usability of the site. In fact, it’s likely users spent more time gazing at the navigation because the site was now more unusable.

Eyetracking is fun to watch and produces cool output. It can serve as a good demonstration that users approach designs differently than we imagine. But can we find a useful place in our research process that is worth all the hassle and expense? I’m still not convinced.

Interestingly, the designers didn’t change (or haven’t yet changed) the interior pages of the site, driving consistency-minded designers nuts everywhere. Fortunately, however, this gives us the opportunity to look closely at the decisions they made, and to highlight several issues involved in making such a change.

Here’s what their homepage looked like this recently:

And here’s what an interior page looked like:

As you can see, the layout has changed dramatically. The content has been shifted left, where navigation used to be. The navigation is now in a thin blue bar across the top of the page. This has the positive effect of moving the visual weight away from the navigation and toward the actual story. When the navigation is on the left, our eyes are instinctly drawn to it, even when we already know what it is. This distraction, however small, is less noticable in the new horizontal design.

To get this change, the designers didn’t simply put the left-hand nav buttons end-to-end across the top of the page. No, they had to make several design choices. Most of these choices were influenced by the width of the page and the number of links they were working with.

1. Link length shrinks
   One of the first things to change when navigation goes horizontal is the length of links. In the CNN design, the designers shortened several links. “Business at CNN Money” became “Business”. “Sports as SI.com” became “Sports”. “Autos with Edmunds.com” became “Autos”. And “Special Reports” became simply “Specials”.

   Our research here at UIE has shown that the longer the link, the better it works. In this case, however, it doesn’t seem like much was lost when the links were cut down. Was “at CNN Money” really adding any relevant information to the “Business” link? Probably not much. Even so, reducing the “Special Reports” link to just “Specials” seems to have lost something.

   Navigation bars, however, are a special case of links. In general, we’ve seen short, one-word navigation links work at the top navigation level on sites when the subsections are relatively distinct. However, as you delve deeper into the site, links tend to work better when they are longer because they need to contain more information that differentiates them from their siblings. For instance, “Sports” and “Business” are two relatively distinct categories, so creating top-level navigation links with just those words might be appropriate. But when on the “Business” subsite, everything has to do with business, so the links need to clearly differentiate what those differences are. The more words there are in the links, the easier it is to do that.

2. # of navigation choices changes
   In the vertical nav configuration on the CNN subpages, there are 17 navigation choices. In the horizontal nav configuration on the homepage there are only 12. The designers have cut down on the choices in order to save space. There would not have been enough room to put all 17 on the horizontal nav bar at the same page width.

   In order to keep the same navigation choices as before, the designers chose to utilize a dropdown box to hold the remaining links. This is a common method that designers use when they don’t want to place all the links on the page at the same time. However, our research has shown again and again that dropdowns and flyouts don’t work very well. Our classic article Users Decide First, Move Second talks about this in more depth.

   As a result, I would bet dollars to doughnuts that those links in the dropdown have fewer people clicking on them than they did when they were in the vertical nav bar. The reason is simple: people have to work to see them now.

3. Prioritization of choices
   The designers at CNN didn’t simply take the last five links and put them into the dropdown. If they had done so, they would have chosen “Travel”, “Education”, “Special Reports”, “Videos”, and “Autos”. Instead, they chose “Law”, “Science & Space”, “Travel”, “Education”, and “Videos”. Obviously, the designers thought that “Autos” was more important than “Law” and “Special Reports” was more important than “Science & Space”. They made an explicit decision to prioritize some over others.

   This is a common problem issue we see with the move to horizontal navigation. At some point, things have to be cut. Instead of losing value, however, design teams can use this as an opportunity to make a site stronger focusing on what is most important on the site. We’ve seen many design teams try to include everything under the sun into their web site, when in almost all cases the vast majority of visitors come for a small sliver of content.

Those are three obvious design challenges that the team had to face when moving to a horizontal navigation scheme. Most teams deciding this very question will face these and more. Do you have any interesting challenges from a recent move to horizontal navigation? Are you considering such a move?

Usability testing is a powerful tool. It guides the design of our products. It informs us on the behaviors and expectations of our users. It gives teams a way to measure how close they are to achieving their goals.

Yet, there isn’t a single way to conduct a usability test. Every team we talk to has their own variations. They have their own tricks for creating tasks, recruiting users, facilitating the tests, and writing the reports.

In today’s UIEtips, we bring back an interview Christine Perfetti conducted with Rolf Molich back in 2003, where they discuss the amazing work that Rolf has done with his Comparative Usability Evaluation (CUE) studies.

The CUE studies pit usability experts against each other, allowing us to compare the practices and results. This type of research is amazingly informative — we learn so much when we compare our own methods against our peers.

In the interview, Christine asks Rolf about CUE-2 and some of his thoughts about the state of usability testing. I think you’ll agree that it’s a fascinating view on where we are today and where we’re going.

Have you been comparing your usability testing practices to others? If so, what have you learned? Share your thoughts in the comments below.

Rolf has now conducted five CUE studies. In fact, CUE-5 was conducted last year at our very own User Interface 10 Conference!

Speaking of the conference, the best place to get the latest on all these studies will be in Rolf’s seminar at User Interface 11 this October. I can tell you that Rolf’s session is going to fill up quickly — it always does — so, if you are interested, you’ll need to register right away. (Plus, we’re giving away iPod nanos to everyone who signs up by April 25th! )

I’m also thrilled to announce that Rolf will conduct CUE-6 at UI11. You’ll have the chance to participate in the CUE if you sign up for Rolf’s UI11 Master Workshop: Assessing Your Usability Skills. (You can find preliminary workshop details here. In June, we will provide more details with information on how to sign up for the Master Workshop.)

Read the article here.

I’m rolling out a search log tool and am wondering about capturing and reporting referrers to searches, including penultimate referrers.

A referrer to a search request would be the page the search is done from. The penultimate referrer is of course the page previous to the page the search is done from. Thus, for example, if a search is done from a given product page, then would it be interesting to know from whence they came to that page. For example, the user arrived there from Google, but then immediately did a site search.

I think this tool could be very useful and something I could see myself insisting our clients make use of.

Our research has shown that on most (not all) sites, only about 14% of search queries are issued from the home page. The vast majority of queries come after visiting lower level pages, such as galleries and content pages.

From this, we infer that many users are making a stab at trying to find what they are seeking by using the categories and links. Only when the scent trail dries up do they turn to Search.

When they turn to Search, our research has shown the queries they enter are the trigger words they were seeking. In essence, they are using Search to create their own links because the links on the page failed them.

For years, we’ve been telling our clients to look at their search logs and extract the trigger words. We’ve also suggested they make an effort to determine what pages the user was on when they made the search and to pay attention to the path they traversed to get there.

Our experience shows that you can see substantial improvements in user satisfaction and task completion by adding the links on the page that duplicate the queries users are making.

Having a tool that makes this simple would be extremely valuable.

(How people use on-site Search is something we talk a lot about at the UIE 2006 Roadshow. A few seats still available for Seattle, San Francisco, and Minneapolis.)

Often, when we meet with design teams, we’ll reserve a few minutes at the tail end of the meeting to do an unusual type of wrap-up. We ask each participant in the meeting to, on a sheet of paper, answer the following three questions with a total time limit of 60-seconds:

1. What was the big idea? (What was the most important thing you heard at the meeting?)
2. What was your big surprise? (What was the thing you saw or heard that surprised you the most?)
3. What’s your big question? (What’s the biggest unanswered question you have at this time?)

After everyone has had a chance to write down their answers, we’ll share them. If the meeting was friendly, we’ll have everyone just go around and read what they wrote. If the meeting was stressful or contentious and we want to take advantage of anonymity, we’ll collect up the papers and the meeting moderator will read them aloud.

This is a trick we picked up while doing a project at the Harvard Business School years ago. One of the professors used a modified version of these questions at the end of every class, collecting up the answers and seeing what class the students thought they just sat through.

We find that it’s not unusual to discover that different people in the room had just attended completely different meetings. People are surprised by things that other people take as a matter of course. People take away a different emphasis about what was discussed. People’s fears and concerns are reflected in their outstanding questions.

With a group of 5 people, you probably need 10 minutes for the exercise — 15 or 20 minutes if you have 10 or more folks in the room. We encourage discussion of the things people wrote, particularly asking why certain things were a surprise. (Often why we’re surprised by something is as important or more important than what we were actually surprised by — something snuck past our radar and we want to know why.)

We could just have everyone just go around the room and answer orally, but we’ve found that writing down produces different answers. When you write something down, you don’t have the benefit of being influenced by answers of others. Interestingly, while we don’t check up on folks, we’ve found people are honest and read what they actually wrote instead of changing their answer after they’ve heard others. (Occasionally, someone will say, “Well, I wrote [whatever] but now hearing what other people said, I would also say [something else].”) We find having people write down answers before sharing them in a group is a great way to get uninfluenced individual perspectives on an issue.

As an outside consultant, it’s a great way for us to find out if everyone is on the same page. It helps us understand what the concerns are, so we can be sensitive to the group’s need and where their priorities lie. A simple tool that yields great value. Gotta love that…

But one panelist, Laurie Vertelney, really changed the way I looked at things. Laurie, working for Apple Computer at the time, showed some prototypes she’d created for her work.

What made these prototypes different from any I’d seen before was they weren’t of the design of the device (in this case, a predecessor to what would become the Apple Newton), but instead were of the experience using the device. The prototype was a comic strip and each panel represented what the user’s life would be like as the interacted with the handheld unit.

I immediately saw how this was different. When you build a prototype design, you typically only see it from the design’s perspective: What are the buttons? How do they work? What displays when? Where is everything in relationship to everything else in the design? No doubt, these are important issues, but they are only one side of the story.

What’s often missing is the view from the user’s experience: What’s frustrating? What works well? What is their motivation? What do they do with the results of the interaction? How does it dovetail into other aspects of their life?

With this one short demonstration, Laurie showed me I’d completely missed this perspective. It completely changed how I looked at design.

What was even more remarkable about her comics was how simple they were. She’d used simple, almost stick-figure characters to draw her people. Yet, each person was expressive, showing the observer how they felt about the frustrations of the current problems and the delight that came from the solutions Laurie’s team was proposing. You couldn’t help but get excited about the designs they were looking at.

After that session, I spent the next few years focusing on capturing these experiences. For the Harvard Business School, I drew really crappy looking cartoons that got the faculty all excited about a new system they were installing in addition to giving us in-depth feedback on what details we’d missed in the initial design. For a major resort hotel chain, we used video interviews of imaginary guests to describe their hotel-stay experience, which we then used as a use-case like requirements set for an information system upgrade. For another client, we hired animators to create a short film that detailed a day in the life with their product.

In each case, the actual design never showed up — only the user’s experiences while interacting with the design. I was always amazed how much valuable information we yielded from that.

As time went on, we moved to more traditional approaches of prototyping and those early techniques fell by the wayside. It wasn’t until I was sitting in Kevin Cheng (one half of the OK/Cancel team) and Jane Jao’s excellent Communicating Concepts through Comics session at the recent IA Summit that I realized I’d forgotten how important this type of prototyping really is to the design process. (Luke wrote up a great description of the session here.)

Comic strips, even poorly drawn ones ( the only kind I can do), allow us to focus on the experience the user has with the design and get feedback during the early concept and discovery stages, where the broad brush strokes are being worked out. Teams can evaluate the strips with real users and collect rich information which will guide every subsequent of the design process.

It’s a powerful technique that I’m surprised we don’t see used more often.

This week, I was reminded of a practice we conduct at the conclusion of a usability test. After the participant(s) have finished with their tasks and filled out whatever paperwork we give them to subjectively rate the design, our test facilitator will ask two questions:

1. What are two things about the design that you really liked?

followed by

2. What are two things about the design that you didn’t like?

The first question helps turn the session into a positive. This is really important, especially if things didn’t quite go as smoothly as everyone would like. The participant, along with the team, needs to focus on the positive for a few moments.

The second question helps prioritize. There may have been lots of issues, but what really jumps out? It’s possible these will be the last two problems the participant encountered, but, in my experience, more often then not, they are two issues that really stand out in the participant’s mind. (We often let the participant “tour” the design while attempting to answer these questions, to refresh their memory of what they just experienced.)

In addition to listening to the words they use, we also pay attention to the speed of their answer. If they produce an answer quickly, that tells us one thing. If it takes them a long time to think of a complaint (or a compliment), we give it less weight — it could be they were just fulfilling our request to name two things.

That doesn’t mean, if they take long to arrive at an answer, we disqualify it outright. Often, we’ll just make a note of it and see if it shows up as an issue with anyone else. If we don’t hear it anywhere else, then it probably goes to the bottom of our “issues list” for the design.

We started doing this technique years ago and it’s just become ingrained in our practice. It actually comes from an conflict resolution technique called Stop, Start, Continue. When dealing with individuals who are in constant conflict (such as dysfunctional work relationships), the mediator asks each person to list 2 things they want the other person to stop doing, 2 things they’d like them to start doing, and 2 things to continue doing. In the conflict case, you end on the positive note (continuing good behaviors). When we adapted it to testing, we inverted the polarity so the positive was at the beginning.

This simple technique of asking two questions often provides us some nice insights into where the participant’s mind is at and the lasting impressions they had from the test experience.

Every time I give a course, I learn a tremendous amount about the problems that designers face. One of the big themes of the day was how difficult it is for organizations to manage the dozens (and sometimes hundreds) of people contributing content and creating design elements on their sites.

Some of the questions attendees asked:

• How can they get everyone contributing to the design on the same page?
• How can they guarantee that all of the designers produce successful results?
• Should they stick with their current templates or style guides?

Lately, one promising strategy we’ve shared with design teams is the creation of a Design Pattern Library. A design pattern describes a specific design problem that an organization has dealt with, such as presenting a login screen. The design pattern consists of a pattern name, a description of the design problem, the design solution, and the designer’s rationale behind that solution. (For more details about how to create a pattern, Jared recently wrote an article about the different components of a design pattern.)

We’ve really started to see design patterns take off within organizations. We’ve worked with several design teams that are now focused on building their own design pattern libraries. Even more exciting is that organizations have started to share their pattern libraries with the design community. For example, just this month, Yahoo published the Yahoo! Design Pattern Library.

We don’t advocate that design teams blindly follow the patterns outlined in Yahoo’s library. After all, your sites have their own unique goals and users. However, Yahoo’s design pattern library is a great model for organizations to review before getting started on creating their own internal libraries. While building a design pattern library takes a lot of time and energy, we’re definitely seeing organizations reap the benefits from their efforts.

One way to help alleviate our frustration at this moment is to look to other designers for the answer. If you’re working on a web site, for example, you might go look at your competitors and see what they’re up to. How did they solve it? Does their method seem better or worse than the one you’re considering? Would it be wise to simply assume that they figured out the best way to solve it and do something similar?

In research this is akin to a “product review”, and if done cautiously, can save a lot of time and frustration. For example, we’ve done in-depth competitive analyses on top e-commerce web sites where we look at how competing sites solved similar problems: like helping users find that perfect laptop, for example. When we present our findings, we make sure that our road to recommendations was based on how the sites actually performed, not on what site had what cool feature.

Unfortunately, it’s all too easy to take a different route at the moment of innovation. Many times, when the answer is not clear and doing research seems daunting, we see “feature grabbing”, a technique where you simply grab features that others have used because they seem like a good idea.

But it’s not always a good idea to feature grab. First of all, you can’t be sure that the feature even works. Just because a competitor has a feature doesn’t mean you should have it too. We’ve seen many cases where features that were copied directly failed to work, and it wasn’t because they weren’t copied faithfully, it was because they weren’t addressing the actual problem.

This idea was captured perfectly in an interview of Dean Kamen I read recently in Make magazine. (subscription required ;( ). Dean is one of the greatest inventors of our time and his company, Deka, is right up the street in Manchester, NH. You may have heard of their much-hyped Segway Human Transporter, their dialysis machine, or their current incarnation of the Stirling engine.

In the following excerpt, Dean talks about how, when they were designing their personal mobility device called the IBOT, they realized that if they did all their research on the existing solution (the wheelchair), then their solution would look and act too much like a wheelchair. This was undesirable because the actual problem was something that a wheelchair doesn’t readily solve. I love his simple and straightforward philosophy on design:

“I try to understand the basic laws of nature. Beyond this, I do very little research as to what the product should be. You would never get the iBOT by doing research on wheelchairs. If you do “product research,” the product that you end up with will be similar to what already exists. For example, if you went out to people who make wheelchairs and said, “I want to make the next great improvement,” they would typically conduct focus groups with people who use wheelchairs. And these wheelchair users, operating within the context of their existing wheelchairs, might ask for things like a new cup holder. They saw a great cup holder in a minivan and realized that their wheelchair didn’t have one. So they ask for a cup holder, or some other incremental improvement. You have to start with basic question: if this person is now missing this amount of functionality, is there some alternative to a wheelchar that is both dramatically better and not prohibited by the laws of physics and the current state of engineering and technology?

“Focusing on the problem in this fundamental way allowed us to understand that wheelchair users need to have the same small footprint on the ground as you and I so they can navigate around areas and obstacles as we do. They need to have their eyes and hands at the same level as a standing person, so they can see over counters and get things down from shelves. They need to be able to get water out of a faucet. And so on. In order to achieve any of these things, we looked a how fully functioning humans do it. They do it by being dynamically stable – by constantly adjusting themselves to maintain balance. Balance is a preprequisite condition to living in a world that is architected by people who walk around balancing themselves. So we decided to forget about wheelchairs and focus on the real problem. The real problem isn’t locomotion – wheels solve that problem fine. The real problem is that these people typically lost their ability to move around while also physically elevating themselves within a small footprint, which requires dynamic stability. Solving this problem would dramatically improve their lives.”

When I set out to design a website, I do it backwards. I start with the design of the smallest, deepest element: the story page or search results. Then I work backwards to design their containers: section pages, indexes. Then, lastly, I work on the home page.

Derek’s approach to site design is consistent with what we’ve seen work most effectively. All too often, clients tell us they spend the majority of their time focusing on the design of the home page when we’ve found that it’s actually the least important page on the site. As Jared mentioned in his post, Is Home Page Design Relevant Anymore?, the home page serves only two purposes for users: it delivers the content, or it provides strong scent to get users to the content page they want.

We’ve seen that the most successful design teams focus on designing the content pages first, ensuring they have all the information that users need on those pages. They understand that the content page is the most important page to users for a very simple reason: this is where users find the information they’ve been seeking.

Does your team spend a large amount of time and resources focused on the homepage? Which page gets the highest priority with your team?

Advertising is all about disruption. TV ads disrupt users from the content. Billboard signs can disrupt people from focusing on driving. Online ads function similarly — they disrupt users from the content they’re looking for.

That’s why it’s always a surprise when I encounter online advertisements that actually work effectively. This happened to me today on Rotten Tomatoes, a site where users can find a summary of the movie reviews from top film critics.

When users arrive, Rotten Tomatoes often disrupts them with ads even before they’ve had a chance to enter the site. When I visited today, I was exposed to an ad for the popular movie, Grizzly Man.

The ad disrupted me from looking for the content I came for — the reviews for George Clooney’s Good Night, and Good Luck (a movie I really want to see.) But I wasn’t annoyed or frustrated. The ad supplied me with information I am actually interested in: Grizzly Man is showing on the Discovery Channel tomorrow night. I now have my TiVo all set up to record the movie.

We often remind our clients that users don’t want to be disrupted from their tasks. But disruption can actually work sometimes. Have you seen any effective online ads recently? How did the site get you to pay attention to the ad? Was the disruption acceptable?

Though we’ve only been at it since July, we feel that blogging has been a very positive experience. So I want to put it out there and see if you have had similar experiences or not. Are you blogging, or thinking about it? If so, do you think that it has been worth it? What has happened as a result?

If you have recently added blogging to your repertoire, we would love to hear about it!

Since we first talked about them in 2003, Design Patterns have become a popular method for teams to tame the consistent-design-management tiger. We’re seeing many of our clients turn to an in-house built pattern library to help them coordinate a consistent look while allowing innovative design to appear as required.

Building a library takes a lot of work, especially at first. However, unlike standards and guidelines, organizations can distribute that work across all members of the design team. Our research has turned up informal techniques, where teams hold a lunch party where each written pattern buys you a slice of pizza, to formal review techniques, like those employed by the folks at Yahoo! (See the excellent Boxes and Arrows case study.)

We’ve been very interested in how various clients are going about building out their libraries. In this week’s UIEtips, I recount some of the more interesting pattern elements we’ve come across. If you’re thinking of building out your library, this could help you get started.

Are you building a design pattern library? What are the elements that you like the best? Share your thoughts in the comments below.

Read the article here.

Christine Perfetti and I will be discussing design patterns in the Effective Dissemination portion of our upcoming UIE Roadshow tour. If you haven’t registered for this, you may want to do so very soon — Seattle is on the verge of selling out and the other cities are filling up quickly. See the details at the UIE 2006 Roadshow site.

Link order is also very important. Well-designed portals put the most important links at the very top and order the remaining links by priority and need. In our studies, poorly-designed portals often resort to alphabetical order, which confuses employees as they expect related links to be group together with the most critical functions near the top.

Several folks have asked what techniques we’ve used to determine the ideal order of links.

We’ve had good success on intranets with a rating system. Using the use cases/tasks associated with each link, we ask users to rate them on two scales:

1. Is this an activity that is important to your work?
2. Is this an activity that you do frequently?

(There are things that are frequent, but not very important, such as checking the lunch menu. There are things that are critically important when needed, but needed only once per year.)

We’ve done this with both binary scales (Circle the “F” or “I” if the use case/task is Frequent or Important) and Likert scales (1 is very infrequent/very unimportant, 5 is very frequent/very important) and not seen any real difference.

Your link order becomes a descending list of the combined scores.

What MIT doesn’t change is the primary navigation. They have a set of top-level and 2nd-level links that stay the same no matter what day you visit. Only the background image and spotlight change. When I visited recently, the homepage featured a large image of what looks to be the handles and digital interface of a treadmill. This was used to spotlight a program called the Getfit Challenge, a three-month team-oriented fitness challenge for the MIT community. Ordinarily this content wouldn’t be featured on the homepage. But it happened to be the spotlight on the day I visited the site.

When I first saw it, I actually thought I may have typed in the URL incorrectly. After all, I had been to the MIT site before but I didn’t recall seeing anything close to this. My expectations were certainly shaken. However, upon a second look I realized that I was indeed at the correct URL, and one that was more memorable now that I knew what was going on there.

At UIE, we’ve seen a lot of homepages, but we haven’t seen many where designers take this approach. We often talk to design teams who make small changes to the homepage over time, usually promoting their latest offer or information to the most visible part of the page. Some even redesign it completely every once in a while. But changing it every day like this is unique. In most cases consistency is seen as a vital attribute for the look of the homepage. We often hear the phrase “consistent look and feel” to describe this belief.

Not so at MIT. They do what would seem preposterous to the “consistent look and feel” crowd. (I can almost hear marketing and design teams cringe.)

On the other hand, MIT does provide consistent change, showcasing content that folks might not have known about. They have obviously chosen the current knowledge approach instead of the consistent look and feel approach. They trust that new visitors will quickly understand what is going on, making the decision to highlight various programs and the creativity of the community at the expense of visual consistency. And they’ve kept at it for almost 3 years. That tells me that they’re confident it’s working.

So what do you think? Preposterous or ingenious?

“Affecting positive change is our end goal. Having solid recommendations is how we make that happen. Ensuring we’ve done our diligence to produce those recommendations is absolutely critical.

By the time we’re producing a recommendation, we want to have a bundle of recommendations to support it. We’ll have formed an army of inferences, only to pick and choose those that the evidence tells us are the strongest and most likely. If we’re not sure, well, we go back to the well and get more evidence.”

Read the article here.

Do you have a different process for producing recommendations? Have you seen design teams jump to the wrong conclusions? If so, we’d love to hear your thoughts. You can leave a comment below.

Yahoo’s frustration with focus groups isn’t that surprising. At UIE, we hardly ever use focus groups because they just don’t work very well at uncovering user needs. The biggest problem: what users say in a focus group rarely matches what they do in a real-life setting. Users’ opinions about a site or product are very rarely consistent with how they behave when they actually interact with it.

We still find that usability testing is the best way to gather input from users. Nothing replaces the power of observing users interact with a site.

Since my first posting, many readers have asked whether they can use the 5-second test to evaluate their home page designs. While the technique is great way to get a glimpse into what happens during the first moments a user sees a page, it hasn’t given us valuable results when we’ve looked at home pages.

We’ve found that a 5-second test works best when we use it on content pages designed with a single primary purpose. However, a site’s home page typically serves the needs of several audiences, each with their own set of tasks. As a result, each of these different users see different things on the page, depending on their context and immediate goals. We’ve found that other techniques, such as traditional usability tests and inherent value tests do a better job of measuring the effectiveness of the home page.

In case you missed it, a few months back, I wrote an article about the 5-second test methodology, outlining how we set up these types of tests and when they are most effective.