As an alternative, how about a hybrid wherein a small group of observers (3-5 key stake holders) is allowed to join the participant in the testing room and ask questions between tasks while the throng is kept in a two-way mirrored cage where they can’t cause trouble? With such an approach, questions could be forwarded silently and invisibly to both the moderator and key stake holder observers while diminishing the participant’s stage fright. Obviously, this would place requirements on the testing environment, but it’s not too unusual for usability labs (at least those of larger companies) to have adequate space for such an arrangement these days. This would seem to be a prudent trade-off that would enable the moderator to focus more on the participant and less on the crowd. Any CEO’s gracing the testing room with their presence could be provided Groucho Marx disguises to hedge the bet!

( additional thoughts at http://www.coherentmedia.eu/fuzzylogic/2007/09/29/usability-tests-with-30-observers/ )

The reason for this is twofold:

1. I can’t see the real benefit from doing this. You say that it seems to keep the observers alert and quiet while conducting the test and that they don’t get so easily bored when they are in the same room. In my experience its often good to let the observers express some emotions for whats happening. Sometimes they try to downplay or comment on whats happening and those comments can sometimes be just as enlightning as the participants actions. Other times you can easily explain why things happen so that the observers doesn’t jump to wrong / pre-determined conclusions about what he sees. A silent room of 30 observers? Why?

2. Usability test ethics. You say that:

“Make sure the participant is not surprised upon entering the room by the crowd. Talking to them before they walk in will help tremendously. If you can warn them when talking to them on the phone the day before, that’s even better.”

This must be a cultural thing. I think that if you put an average Scandinavian test person in a room with 30 observers you will have one seriously nervous participant. Nervous participants are (as you well know) not of good use for the client you do the usability test for. I don’t know if this will be just as severe with American test participants.

So I think the participant will be nervous and reserved throughout the whole session. And what if he/she really messes it up in the usabilitytest and does something really funny or stupid? No matter how well informed the 30 observers are - you will have a problem with laughter (or even anger?). This creates a very bad situation for the poor test participant and in worst case u get a person that is a bundle of nerves for the rest of the session.

“Having the observers in the same room as the participants means they can interact.”

This is another thing I wouldn’t like to see. I can imagine this would create a few situations where you get observers that goes on with “why did you do that?” and then constantly reminding the test participant that there are 30! people he have to explain the error to.

I think both situations are unethical because it makes the situation akward for the participant.

I’m not pretending I’ve got THE answer on how to do it but here is our setup:
We use 2 cameras and live transmission of the screen/mousemovements projected on canvas in the observer room. A similar setup like the one you have with loudspeakers and also headphones to the main observer than takes the (main) notes. The cameras are connected to 2 TVs about 25″ big. So the real focus is not on the person but on the big canvas showing the actual website in action + sound. The expressions and facial tells is mainly for the trained observer and not for the bulk of the observers anyway.

I can see the entertainment effect of 30 people watching in the same room, but I can’t see the real benefit here.

I don’t think participants ever truly forget people are watching the session, in any testing environment. They always seem aware of their observers, though, with the right facilitation, they can prevent that awareness by focusing on the task at hand.

This is true whether there are 30 people in the room or they are alone in a room with a camera or big mirror.

How much time are observers given to ask questions?

This will depend on the length of the session, the number of tasks, and the specific protocols in use. Ideally, we’d give observers 5 minutes after each task and about 15 minutes at the end of the session.

How quickly before the participant forgets that there are people watching the session?

“Having the observers in the same room as the participants means they can interact.” - How much time are observers given to ask questions?

Look forward to learning more about this when we chat at UI12 and looks like an approach that would need to be planned for and facilitated very carefully.

rgds,
Daniel

Thanks for the clarification. You’re right; I was under the impression that only a couple of sessions had 30 observers and the rest were more standard (the subject and a couple of observers). It’s a really interesting approach, especially the idea of treating it more as a group discussion. I’m so used to more rigidly-controlled experimental settings that that kind of setup wouldn’t have occurred to me, but I can definitely see the value, both for the process and engaging the client’s team.