Usability Tests with 30 Observers

Jared Spool

September 13th, 2007

Often times, the clients we’re working with have never conducted a usability test before. Since it’s not part of the regular process, almost no one in the organization has been exposed to watching users.

We’ve found it’s critical to the adoption of testing in the organization for people to quickly see the benefits. We’ve also found there’s no better way to show the benefits of testing than to observe a test firsthand.

Because there’s a large number of design agents and stake holders involved and often a small number of available tests to observe, we can find ourselves in a situation where we need to have many observers in one session. It’s not unusual to have 30 or more observers sitting in a single usability test. Here’s how we pull it off.

First, get a large room

We love training rooms. They work really well. (We’re not a big fan of usability labs — they distract and hurt more than they help, in our opinion.)

You want a room where you can set up the seats for the observers, a projector, an audio system, and a small station for the participant and facilitator.

Often, we’ll set up the room to look like this: (Thanks to Dana Chisnell for supplying this image from the upcoming revision to the Handbook of Usability Testing.)

The room layout for a 30-observer usability test

The participant and session facilitator will sit up front. The user (and, if you can get a mixer, the facilitator) will have a mic, which plugs into the speakers, so everyone can hear what the participant says (or even mumbles).

We project the participant’s screen onto the large screen so all the observers can watch what the user is doing.

(If you want to get super fancy, you can put a video camera plugged into a monitor below the projected screen to see the user’s expressions. We don’t do this because we rarely find it adds anything for the extra effort and stress on the user’s part. If you do decide to use it, don’t do a picture-in-picture display on the big screen — in inevitably obscures an important part of the interface. )

Brief the observers

We’re hold a 15 minute meeting with all observers before each participant shows up. In this meeting, we go over the rules for observing. We’re quite strict about these, since our number one objective in a usability test is to lookout for the stress of the participant. We hammer in how a session like this can be stressful on the participant and perfect behavior is absolutely necessary to have a successful study.

We don’t allow observers to come or go during the session. They have to arrive before it starts and stay for the entire duration. (You’d be surprised how often other things can be rearranged to accommodate this when you’re strict about it.)

We also explain how noises affect the participant’s behavior, so we tell them they are not allowed to laugh, sigh, or make any other noises while the participant is focusing on their design.

We explain how questions will work (I’ll get to that in a bit). Finally, we go over the tasks (which we’ve provided a copy for each observer) and the test protocol, including the objectives.

Brief the participant

Outside the session room, we greet and brief the participant. In this session, we walk through the usual pre-test protocol of a test: explaining what the purpose is, what their role will be, how we want them to behave, and how they are in control (they can stop or take a break at any time).

At this point, we’ll explain how there will be 30 (or however many there actually are) members of the design team observing, what the observer’s role is, and how the participant can practically ignore their presence. We explain the observers will see (on the big screen) and hear (thru the speakers) everything the participant does and how this will help us create simpler and more delightful designs in the future.

Introduce the observers to the participant

We’ll then bring the participant in the room. At this point, the facilitator will say something like, “Everybody, say hi to Lisa” to introduce the participant. The facilitator will then review the pre-test paperwork to “introduce” the participant and familiarize the observers with the participant’s background and experience.

Conduct the Test

Once the facilitator has explained the protocol to the participant (we do this in the room so the observers can hear it again), we start on the test. In addition to ensuring the participant has a stress-free experience, the facilitator’s role becomes that of sportscaster — making sure the subtle things happening with the participant are observed by the spectators.

If the participant makes faces, non-verbal utterances (like sighs or mumbles), or other things the observers wouldn’t pick up on, the facilitator will make a comment, such as “I noticed you seem to be frowning. Is something not right?”

If the observers have questions for the participant, they can either wait until the Q&A period we hold after each task, or they can write their question on a note and pass it to the facilitator. The facilitator has the discretion to ask the question or hold it (if they think it will disturb the participant’s flow or gives away too much information).

If observers want to talk to each other, we recommend passing written notes. (After years of doing this, we’ve found participants rarely pay attention to notes being passed. They do pay attention to whispers and typing, so we ask observers to refrain from whispering, instant messenger, or taking notes online.) Observers have to be careful not to make a joke in the notes that might cause a colleague to laugh. (Participants will almost always assume the laughing has something to do with them, even though it rarely does.)

When the end-of-task Q&A session starts, any observer can ask a question. (During the pre-test briefing, we often explain not to ask “Design” questions — questions like, “How would you order the options in the menus?” since these questions rarely yield interesting answers and take up valuable time while the participant makes a stab at answering. Instead, we suggest they ask behavioral questions, “Can you tell me what is confusing about the menus?”)

Saying Goodbye

Once the participant has completed the test, we ask the observers to thank them for their time. We then have the facilitator escort the participant out of the building. This gives the participant a chance to give any feedback that they weren’t comfortable giving before the entire group. (This rarely happens, but it has happened enough we allow for it.)

Debriefing Observers

After the session, before the observers disperse, we ask them to share important things they learned. Often, this won’t take much prompting.

It’s in this point where we can help them form the best inferences. (Inexperienced observers often jump to the first inference that comes to mind, often overlooking serious alternative causes of the problems. If left unchecked, they may rush to make changes that won’t improve the experience and may break something else.)

Do participants get more stressed with 30 observers?

With proper preparation of both the observers and the participants, we find it’s just like any other study. The key takeaways from doing this are:

  • Make sure the room is set so each observer can clearly see and hear what’s going on. No matter how many you have, when observers can’t follow along, they get bored and fidgety, which leads to problems.
  • Make sure the participant is not surprised upon entering the room by the crowd. Talking to them before they walk in will help tremendously. If you can warn them when talking to them on the phone the day before, that’s even better.
  • The facilitator needs to have both crowd-control skills and comfort-giving skills. They’ll be tending to the observers and participants’ needs simultaneously. In our experience, this is something that gets better with practice.
  • The observers need to understand from the outset that a usability test is a structured process with a defined protocol and agenda. Once the study starts, things must progress as planned, so if they have specific concerns, they need to address them during the test planning stage, before the first participant arrives on site.

12 Responses to “Usability Tests with 30 Observers”

  1. Daniel Szuc Says:

    Would the result be the same if the participant sat in a different room from the 30 observers?

    Are there benefits by having them in the same room?

    On initial reads this seems scary for the participant (especially when walking in to see 30 people)

  2. Fresh: новости мира юзабилити » Blog Archive » Юзабилити-тестирование с 30 наблюдателями? Says:

    [...] Usability Tests with 30 Observers [...]

  3. Chris Blow Says:

    I would be terrified to have my back to a room of 30 stakeholders and experts.

  4. Dr. Pete Says:

    I’m curious, Jared: do you find that doing observation on an actual subject has more value than just taking the 30 observers through a simulated session? Do you throw that subject’s data out?

  5. More on Usability Tests with 30 Observers » UIE Brain Sparks Says:

    [...] had some great comments on my post, Usability Tests with 30 Observers. Here are my [...]

  6. Comment on Jared Spools post “Usability Tests with 30 Observers” « Information Design & Usability Says:

    [...] Comment on Jared Spools post “Usability Tests with 30 Observers” October 1st, 2007 — haakonha This is my response to Jared Spools post: “Usability Tests with 30 observers“: [...]

  7. Watching them work « Useology Says:

    [...] written a couple interesting posts recently regarding usability tests. Please check them both out: Usability Tests with 30 Observers and More on Usability Tests with 30 Observers. In More on Usability Tests with 30 Observers Jared [...]

  8. Why using 30 observers in the same room as the participants is 99% bad* « Information Design & Usability Says:

    [...] thats not afraid to go upstream with your opinions. In this case though I think that your article Usability tests with 30 observers points the usability testing community in the wrong [...]

  9. blogandtell » Brukertest med publikum? Hva med 30 observatører? Diskusjon med Jared Spool. Says:

    [...] 1. Jared Spools opprinnelige blogginnlegg [...]

  10. Bookmarks from January 21st to January 22nd | BlobFisk.com Says:

    [...] Usability Tests with 30 Observers » UIE Brain Sparks [...]

  11. Mickey Says:

    Hi Jared,

    Thanks for your post on “usability tests with 30 observers”. It’s an interesting approach. I just don’t see the utility in it. To me it would make sense if there were way less observers in the room, and they were participating in the session as well. What’s the value in it? Is it to make the observers feel more invested in what they’re building? More invested in UCD? I was looking for the specific purpose in this article, but didnt’ see it.

    I’ve been practicing usability testing for 12+ years. I tend to keep the observers separated from the participants, unless I’m using more of a collaborative approach…then I’ll include a very small number of observers.

    If observers just want to watch, I keep in them in the observation room. I let the participant know of the audience, and I even take him/her and make introductiosn to the observers.

    I see the value in having observers asking questions at the end of the session, and dialoging with the participant. This approach could extract some insights that the facililator may not. Still, 30 ppl is alot. :-)

    It’s been 2+ years since you posted this. Has your opinon changed on this approach?

  12. Jared Spool Says:

    Hi Mickey,

    Our approach is still the same.

    When you’re working with large organizations where there are many design agents (people who have direct & indirect influence over the design), there are godo reasons to have them all exposed to the testing.

    Most two-room set ups can’t accommodate 30 observers and keep them engaged in the test. And, in many organizations, important people otherwise wouldn’t get exposure to the experience of using their designs.

    So, it’s good to have a method in our pocket that works well when we need it.

    Jared

Add a Comment