Moderating A Usability Test Part 2 - Show Notes - [Producer's note: This is the script we worked from when creating this episode. It is reasonably complete, but during the show we do ad lib, so there will be things mentioned there that are not here.] B: This is the second and final part of our first episode in a series of Usability Tools Podcasts on the basics of usability testing. The purpose of the series is to create a complete resource for anyone looking to plan and conduct usability tests to learn everything they need to know to be successful at it. In today's episode we're going to pickup where we left off in part one, discussing how to moderate your session. After today's episode, you should have an idea how to start your session, and understand the flow of tasks during the session. Jared, our first participant has just arrived. How do we get the actual session started? Starting the session Moderating the session starts when the participant walks in the room. A typical pre-test protocol might be to: 1. Introduce yourself to the participant 2. Collect up and verify any forms they might've received in preparation of the test 3. Ask them to fill out any questionnaires that happen before they use the design 4. Introduce them to the observers 5. Tell them the overall purpose of the test 6. Tell them their rights as a participant 7. If appropriate, tell them your recording the session and start the recorder 8. Give them the first task to work on When you introduce yourself, make sure you get the proper pronunciation of their name. Don't bring the observers in the room until all the paperwork is done. It's unnerving to have people watch you fill out forms. As you hand them the paperwork, you might also ask them if they want a beverage, like a soda or coffee. This will give you a reason to leave the room while they are doing their papers, so they can work uninterrupted. When you bring the observers in, you don't want extensive introductions. Really, just a first name is all that's necessary from each observer. You don't want job titles, even if the person is someone really important. (Would you want to know that the CEO of GE is watching you have trouble figuring out how to schedule a dishwasher repair appointment?) For the purpose of the test, say something like: "We're here today to see how people use some designs we're trying out. We know their not perfect and we think people might find them frustrating or confusing in places. Unfortunately, we don't quite know where, so we need your help. "We're going to ask you to do a few things with our design. Some will be straightforward and some may be tricky. As you work with the design, we'd like you to tell us what you're thinking. Anything that is easy or confusing we'd love to hear about. Does that sound ok?" For their rights, we tell them: "We're not testing you today. We're testing the design. We think you're the type of person we want to use this product, so if you find it hard to use, that's something we'd like to fix. To that end, you don't need to stay for the entire session. We can stop at any point. We can also take a break any time you'd like. If you just want to stretch or go for a walk, we can do that. Is that ok?" Now, about the recorder. Hopefully, if you're recording, you've asked the participant if it was ok when you first recruited them. There's nothing worse than being surprised at the session that someone doesn't like being recorded. So, at this point, we're just trying to verify the participant realizes that, from this point on, we're recording. Of course, if they object, we'll have to go forward without recording, but if you've done your homework well, they won't object. Sometimes, once we've turned on the recorder, we'll go over the paperwork they've filled out. (This would be something you'd decide when putting your protocol together.) The purpose is to get, on the recording, some of the important things about this participant, such as any previous experience they might have with the design or other facts. This might sound like: "So, it says here you've been using our product since version 3.0 and that you mostly use it to check your mail and book travel. Is that correct? Also, it says here that you regularly use Orbitz as your travel site. How many years have you used that for? 5? Excellent." The flow of tasks For today, I'm going to assume your doing what we call a "scavenger-hunt" test. These are a standard type of usability test where we give participants pre-defined tasks that we'd like them to complete. A typical task might be to read your email and if any trips show up, using the travel booking functionality to make your flight and hotel arrangements. (In a later podcast, we'll share how to create these tasks and other types of tasks, like interview-based tasks.) There's three ways to start the participant on a task: 1. You can read the task to them 2. You can have them read it out loud 3. You can have them read it to themselves, then have them summarize what they think they need to do (Of course, you'll want to have your tasks printed on individual sheets of paper, so that every participant is working from the same instructions. That's part of the scientist role.) I prefer the third option -- having them read it to themselves then summarize out loud. For this type of testing, that's fine. (In benchmark testing, I'll often use a protocol that has them read it out loud. We start the timer from the moment they stop reading.) By having them summarize, we make sure they understand what we're asking of them. It also helps the observers follow along. (They should have copies of the tasks and you've told them, in advance, what order you'll ask the participant to do them.) Once they start, the moderator role now shifts to a heavy sportscaster. I'd be watching the participant closely, looking for anything that I think the observers should see or hear. For example, if the participant mumbles something under their breadth, I might repeat it (assuming I caught it). If they pressed an unusual button on the screen, I might say, "I see you pressed the 'Options' button." B: Do you explain to them why they've just stumbled into the Truman Show? At first, this might seem unnerving to the participant, but they get used it pretty fast. So much so that they often start telling you about things you might've missed. The idea is to "narrate" the session for the observers. Of course, if your room is set up (maybe with a projector and good sound system), you won't need as much narration. But, they may be looking at their notes when something important happens, so you still want to punctuate those moments. Once the participant has finished their task, you need to do some follow up. Your protocol might have some standard questions you always ask at the end of a task, such as "How do you think that went?" (Some protocols have paperwork after each tasks, like a task load index or a engagement metric. This is when you'd have them fill out these questionnaires.) Once that done, it's Q&A time for the observers. In a future podcast, we'll talk about how to brief observers before the session, so they know how to ask questions and which questions are appropriate. As part of the scientist, you'll need to watch the clock. You'll want to know how long the task took and how long the Q&A is taking. You'll have to decide if it's more important to let the questions continue or if you need to move on to the next task. Then you start the cycle again with the next task. B: OK, so how does one wind down a session? Ending the session Once you've completed all the tasks or run out of time, it's time to wrap up. This is important: even if the participant is late for the session or you started late for any other reason (like a problem with the equipment), you shouldn't go over the originally scheduled time. Your schedule should have some flexibility built in, but you've promised to the participant and the observers that it would end at a certain time. When it goes over, it just makes everyone anxious and stressed. This means you'll need to be keeping a strict eye on the clock. You'll need to know how long the end-of-session routine will take and plan accordingly. If you've got the participant for an hour and it takes 10 minutes to wrap up the session, then you need to stop the last task when there is only 10 minutes left in the scheduled period. It's ok to interrupt a participant on the last task. You can say, "I'm sorry, but that's all the time we have for today. It's ok if you don't finish -- we need to wrap things up here. This has been really helpful." Once you've ended the tasks, you'll have things to do, which will vary depending on your protocol. First, if you have any post-test paperwork, you'll want to ask the participant to complete that. Then, you'll want to open the discussion up for any final Q&A. It's at this point that the observers can ask those questions you've told them to save to the end. I always like to end with two questions, which I ask orally: 1. What are two things that you saw today that you really liked? 2. What are two things that you saw today that you didn't like? The answers to these questions are often insightful. Sometimes, things you thought were really complex turn out to be the things that the participant really liked. Sometimes, you'll be surprised by what they didn't like. Once you're done, you thank your participant and give the observers a chance to say thank you. Then you escort the participant out of the building (or have one of the observers assist you with that). In a future podcast, we'll talk about the short meeting you should have with your observers right after each session. Ok, that's it. Summary To summarize: 1. Good moderation is critically important 2. You can learn how to do it, but it takes practice 3. You'll want to learn the roles of flight attendant, sportscaster, and scientist 4. You need to know that sportscaster is more important than scientist and flight attendant is more important than both 5. You want to start with the right steps, run the tasks using the protocol, and then end on time with a clear wrap-up B: Thanks Jared. That brings to an end this week's episode of the Usability Tools Podcast. As we mentioned previously, this is only the first topic of many we'll be publishing in a series introducing the basics of Usability Testing. Thanks for listening and come again soon for more on the basics of usability testing. Thanks Jared. Jared: Thank you B: Farewell for now.