September 13th, 2007
Often times, the clients we’re working with have never conducted a usability test before. Since it’s not part of the regular process, almost no one in the organization has been exposed to watching users.
We’ve found it’s critical to the adoption of testing in the organization for people to quickly see the benefits. We’ve also found there’s no better way to show the benefits of testing than to observe a test firsthand.
Because there’s a large number of design agents and stake holders involved and often a small number of available tests to observe, we can find ourselves in a situation where we need to have many observers in one session. It’s not unusual to have 30 or more observers sitting in a single usability test. Here’s how we pull it off.
First, get a large room
We love training rooms. They work really well. (We’re not a big fan of usability labs — they distract and hurt more than they help, in our opinion.)
You want a room where you can set up the seats for the observers, a projector, an audio system, and a small station for the participant and facilitator.
Often, we’ll set up the room to look like this: (Thanks to Dana Chisnell for supplying this image from the upcoming revision to the Handbook of Usability Testing.)
The participant and session facilitator will sit up front. The user (and, if you can get a mixer, the facilitator) will have a mic, which plugs into the speakers, so everyone can hear what the participant says (or even mumbles).
We project the participant’s screen onto the large screen so all the observers can watch what the user is doing.
(If you want to get super fancy, you can put a video camera plugged into a monitor below the projected screen to see the user’s expressions. We don’t do this because we rarely find it adds anything for the extra effort and stress on the user’s part. If you do decide to use it, don’t do a picture-in-picture display on the big screen — in inevitably obscures an important part of the interface. )
Brief the observers
We’re hold a 15 minute meeting with all observers before each participant shows up. In this meeting, we go over the rules for observing. We’re quite strict about these, since our number one objective in a usability test is to lookout for the stress of the participant. We hammer in how a session like this can be stressful on the participant and perfect behavior is absolutely necessary to have a successful study.
We don’t allow observers to come or go during the session. They have to arrive before it starts and stay for the entire duration. (You’d be surprised how often other things can be rearranged to accommodate this when you’re strict about it.)
We also explain how noises affect the participant’s behavior, so we tell them they are not allowed to laugh, sigh, or make any other noises while the participant is focusing on their design.
We explain how questions will work (I’ll get to that in a bit). Finally, we go over the tasks (which we’ve provided a copy for each observer) and the test protocol, including the objectives.
Brief the participant
Outside the session room, we greet and brief the participant. In this session, we walk through the usual pre-test protocol of a test: explaining what the purpose is, what their role will be, how we want them to behave, and how they are in control (they can stop or take a break at any time).
At this point, we’ll explain how there will be 30 (or however many there actually are) members of the design team observing, what the observer’s role is, and how the participant can practically ignore their presence. We explain the observers will see (on the big screen) and hear (thru the speakers) everything the participant does and how this will help us create simpler and more delightful designs in the future.
Introduce the observers to the participant
We’ll then bring the participant in the room. At this point, the facilitator will say something like, “Everybody, say hi to Lisa” to introduce the participant. The facilitator will then review the pre-test paperwork to “introduce” the participant and familiarize the observers with the participant’s background and experience.
Conduct the Test
Once the facilitator has explained the protocol to the participant (we do this in the room so the observers can hear it again), we start on the test. In addition to ensuring the participant has a stress-free experience, the facilitator’s role becomes that of sportscaster — making sure the subtle things happening with the participant are observed by the spectators.
If the participant makes faces, non-verbal utterances (like sighs or mumbles), or other things the observers wouldn’t pick up on, the facilitator will make a comment, such as “I noticed you seem to be frowning. Is something not right?”
If the observers have questions for the participant, they can either wait until the Q&A period we hold after each task, or they can write their question on a note and pass it to the facilitator. The facilitator has the discretion to ask the question or hold it (if they think it will disturb the participant’s flow or gives away too much information).
If observers want to talk to each other, we recommend passing written notes. (After years of doing this, we’ve found participants rarely pay attention to notes being passed. They do pay attention to whispers and typing, so we ask observers to refrain from whispering, instant messenger, or taking notes online.) Observers have to be careful not to make a joke in the notes that might cause a colleague to laugh. (Participants will almost always assume the laughing has something to do with them, even though it rarely does.)
When the end-of-task Q&A session starts, any observer can ask a question. (During the pre-test briefing, we often explain not to ask “Design” questions — questions like, “How would you order the options in the menus?” since these questions rarely yield interesting answers and take up valuable time while the participant makes a stab at answering. Instead, we suggest they ask behavioral questions, “Can you tell me what is confusing about the menus?”)
Once the participant has completed the test, we ask the observers to thank them for their time. We then have the facilitator escort the participant out of the building. This gives the participant a chance to give any feedback that they weren’t comfortable giving before the entire group. (This rarely happens, but it has happened enough we allow for it.)
After the session, before the observers disperse, we ask them to share important things they learned. Often, this won’t take much prompting.
It’s in this point where we can help them form the best inferences. (Inexperienced observers often jump to the first inference that comes to mind, often overlooking serious alternative causes of the problems. If left unchecked, they may rush to make changes that won’t improve the experience and may break something else.)
Do participants get more stressed with 30 observers?
With proper preparation of both the observers and the participants, we find it’s just like any other study. The key takeaways from doing this are:
- Make sure the room is set so each observer can clearly see and hear what’s going on. No matter how many you have, when observers can’t follow along, they get bored and fidgety, which leads to problems.
- Make sure the participant is not surprised upon entering the room by the crowd. Talking to them before they walk in will help tremendously. If you can warn them when talking to them on the phone the day before, that’s even better.
- The facilitator needs to have both crowd-control skills and comfort-giving skills. They’ll be tending to the observers and participants’ needs simultaneously. In our experience, this is something that gets better with practice.
- The observers need to understand from the outset that a usability test is a structured process with a defined protocol and agenda. Once the study starts, things must progress as planned, so if they have specific concerns, they need to address them during the test planning stage, before the first participant arrives on site.