More on Usability Tests with 30 Observers

Jared Spool

September 18th, 2007

We’ve had some great comments on my post, Usability Tests with 30 Observers. Here are my responses:

Daniel asked:

Would the result be the same if the participant sat in a different room from the 30 observers?

Are there benefits by having them in the same room?

On initial reads this seems scary for the participant (especially when walking in to see 30 people)

In my experience, the result is different for both the observers and the participant when they are in different rooms.

For the observers, when there’s a window or a video link separating them from the participant, that distance sends a message. I’ve found observers are less likely to pay attention and more likely to make jokes and derogatory comments when they are isolated.

Participants aren’t dumb and know someone is watching. They can see the mirrors and the cameras and know they are being observed. But, because they can’t see their observers, they can only imagine what those observers are doing. I’ve noticed participants tend to assume the worst and the mirror/video-link unnerves them more than having the observers right in the room.

Another benefit comes from direct interaction. Having the observers in the same room as the participants means they can interact. I’ve found developers and stakeholders come away from sessions they attend this way with a solid connection to the users they are trying to design for.

Chris said:

I would be terrified to have my back to a room of 30 stakeholders and experts.

Chris, I’m assuming your fear may be justified because of your organization’s culture. In my experience, most of the time, this fear is completely unfulfilled and everyone behaves nicely, getting a lot out of the study. However, if you don’t want to have the stakeholders behind you, there are two options:

  1. Rotate the moderator and participant station 90 degrees, so the stakeholders are not behind you, but on one side of you. (You’ll be positioned between the participant and the observers.) This way you can keep an eye on both. I’ve done this plenty of times and it works well.
  2. Get a buddy to help you with “crowd control”. Either you or your colleague sit with the observers and, through careful passing of notes, help them understand what they are seeing. Don’t underestimate the power of a well-timed hand-written note.

Dr. Pete asked:

Do you find that doing observation on an actual subject has more value than just taking the 30 observers through a simulated session? Do you throw that subject’s data out?

Dr. Pete, it sounds like you think I’m doing this only for demonstration purposes. In fact, these are real tests that the observers are participating in. No data is thrown out. The entire series of tests use the same setup.

The last time I did this was for a top-10 e-commerce site (who averages approximately $1.2 billion in sales each year). We observed 24 users this way, collecting the data from each one. Each session was filled with 30 folks (we had to put together an elaborate signup and waiting-list system to handle it), often with senior executives — directors, VPs, and even the CEO. Most observers came to more than one session and would cancel other meetings to attend.

As a result, changes were made that look like they’ll realize about $250,000,000 in additional revenue this year. I don’t think we could’ve had the impact if I didn’t have the 30 observers in each session.

6 Responses to “More on Usability Tests with 30 Observers”

  1. Dr. Pete Says:

    Dr. Pete, it sounds like you think I’m doing this only for demonstration purposes.

    Thanks for the clarification. You’re right; I was under the impression that only a couple of sessions had 30 observers and the rest were more standard (the subject and a couple of observers). It’s a really interesting approach, especially the idea of treating it more as a group discussion. I’m so used to more rigidly-controlled experimental settings that that kind of setup wouldn’t have occurred to me, but I can definitely see the value, both for the process and engaging the client’s team.

  2. Daniel Szuc Says:

    Thanks Jared.

    How quickly before the participant forgets that there are people watching the session?

    “Having the observers in the same room as the participants means they can interact.” – How much time are observers given to ask questions?

    Look forward to learning more about this when we chat at UI12 and looks like an approach that would need to be planned for and facilitated very carefully.

    rgds,
    Daniel

  3. Jared Spool Says:

    How quickly before the participant forgets that there are people watching the session?

    I don’t think participants ever truly forget people are watching the session, in any testing environment. They always seem aware of their observers, though, with the right facilitation, they can prevent that awareness by focusing on the task at hand.

    This is true whether there are 30 people in the room or they are alone in a room with a camera or big mirror.

    How much time are observers given to ask questions?

    This will depend on the length of the session, the number of tasks, and the specific protocols in use. Ideally, we’d give observers 5 minutes after each task and about 15 minutes at the end of the session.

  4. Haakon Halvorsen Says:

    I’ve been conducting usability testing for several years (5+), but I have never even considered mixing the observers with the participants.

    The reason for this is twofold:

    1. I can’t see the real benefit from doing this. You say that it seems to keep the observers alert and quiet while conducting the test and that they don’t get so easily bored when they are in the same room. In my experience its often good to let the observers express some emotions for whats happening. Sometimes they try to downplay or comment on whats happening and those comments can sometimes be just as enlightning as the participants actions. Other times you can easily explain why things happen so that the observers doesn’t jump to wrong / pre-determined conclusions about what he sees. A silent room of 30 observers? Why?

    2. Usability test ethics. You say that:

    “Make sure the participant is not surprised upon entering the room by the crowd. Talking to them before they walk in will help tremendously. If you can warn them when talking to them on the phone the day before, that’s even better.”

    This must be a cultural thing. I think that if you put an average Scandinavian test person in a room with 30 observers you will have one seriously nervous participant. Nervous participants are (as you well know) not of good use for the client you do the usability test for. I don’t know if this will be just as severe with American test participants.

    So I think the participant will be nervous and reserved throughout the whole session. And what if he/she really messes it up in the usabilitytest and does something really funny or stupid? No matter how well informed the 30 observers are – you will have a problem with laughter (or even anger?). This creates a very bad situation for the poor test participant and in worst case u get a person that is a bundle of nerves for the rest of the session.

    “Having the observers in the same room as the participants means they can interact.”

    This is another thing I wouldn’t like to see. I can imagine this would create a few situations where you get observers that goes on with “why did you do that?” and then constantly reminding the test participant that there are 30! people he have to explain the error to.

    I think both situations are unethical because it makes the situation akward for the participant.

    I’m not pretending I’ve got THE answer on how to do it but here is our setup:
    We use 2 cameras and live transmission of the screen/mousemovements projected on canvas in the observer room. A similar setup like the one you have with loudspeakers and also headphones to the main observer than takes the (main) notes. The cameras are connected to 2 TVs about 25″ big. So the real focus is not on the person but on the big canvas showing the actual website in action + sound. The expressions and facial tells is mainly for the trained observer and not for the bulk of the observers anyway.

    I can see the entertainment effect of 30 people watching in the same room, but I can’t see the real benefit here.

  5. Talley Says:

    Interesting article. I’ve been researching remote usability techniques, but I can definitely see the potential in this approach as well. For those interested, I was compelled to chime in on my own blog ( http://www.coherentmedia.eu/fuzzylogic/2007/09/29/usability-tests-with-30-observers/ ), but here’s the upshot.

    As an alternative, how about a hybrid wherein a small group of observers (3-5 key stake holders) is allowed to join the participant in the testing room and ask questions between tasks while the throng is kept in a two-way mirrored cage where they can’t cause trouble? With such an approach, questions could be forwarded silently and invisibly to both the moderator and key stake holder observers while diminishing the participant’s stage fright. Obviously, this would place requirements on the testing environment, but it’s not too unusual for usability labs (at least those of larger companies) to have adequate space for such an arrangement these days. This would seem to be a prudent trade-off that would enable the moderator to focus more on the participant and less on the crowd. Any CEO’s gracing the testing room with their presence could be provided Groucho Marx disguises to hedge the bet!

    ( additional thoughts at http://www.coherentmedia.eu/fuzzylogic/2007/09/29/usability-tests-with-30-observers/ )

  6. Why using 30 observers in the same room as the participants is 99% bad* « Information Design & Usability Says:

    [...] response to Daniel Szucs question about how quickly the participant forgets he is being observed Jared answer that “They always [...]

Add a Comment