Moderating A Usability Test - Show Notes - [Producer's note: This is the script we worked from when creating this episode. It is reasonably complete, but during the show we do ad lib, so there will be things mentioned there that are not here.] B: This is the first recording we're making in a series of Usability Tools Podcasts on the basics of usability testing. The purpose of the series is to create a complete resource for anyone looking to plan and conduct usability tests to learn everything they need to know to be successful at it. By the end of today's installment, you should have a good foundation for moderating your first test. Feel free to take notes and ask us your questions in the comments of this post on our blog at UIE.com/Brainsparks. Hi Jared. J: Hello... Jared, there's lots of discussion among people who have run or want to run usability tests on what the role of the host is. How does moderating work, and why are we starting with that topic? J: While it's not the beginning of the process—planning your test is—we're starting with moderating the test session as our initial topic in the series. Good planning and preparation for a test is important. We'll cover those in future episodes. However, if you don't moderate the test well, you'll lose the chance to learn valuable insights from your participants, which was the entire point of conducting usability tests. The moderator is the person in charge of the usability test session. It may or may not be the person who planned the tests. Nor do you need need to have the same moderator for all the sessions in a test series, though doing so will increase the consistency of the session results. Anyone can be a moderator. It doesn't take any special talents. However, it is a skill that practice makes better. So, the more you do it, the better you'll become. Today we're going to talk about the basics of moderating your session. We'll cover these points: 1. Why Good Moderation is important 2. The Three Basic Roles: Flight Attendant, Sportscaster, Scientist 3. The job of the flight attendant 4. The job of the sportscaster 5. The job of the scientist 6. Starting the session 7. The flow of tasks 8. Ending the session For this cast, we're assuming we're talking about what's called a formative study. This is the most common type of usability test you would use in a development process. We're going to assume you've decided to conduct a set of tests to see if a design is working well. Maybe it's your current design and you want to learn what works and what doesn't. Or you could be testing out a new design, to see if it's improving things. There are other types of tests, such as benchmark tests, where you're trying to get some measurements about your design you'll use to detect improvement in future designs; or a competitive analysis, where you're looking to see how you rate compared to your competitors. These are different tests and, therefore, require different types of moderation. We'll do special podcasts later to talk about these later on. Why Good Moderation is important You only get one shot with this participant. I wish that we could just open up the participant's head and look right in, but we can't. Instead, we need to coax the information out. If we do it the wrong way, the information we get will be flawed in ways we may not realize. If we then make decisions from that information, we may end up with a design that is worse than what we started with. Chances are you have a limited number of participants in the study. Need to make each one count. You're balancing the needs of the project against the needs of the participant. It's a tricky operation. Part of the purpose of a usability test is to have the team members get first hand experience with what it's like to use your design -- to see the design through the user's eyes. However, a risk of a poorly moderated test is that some team members don't believe that what their seeing is true user behavior. They think the user is too stupid or is acting in an abnormal way. If they are protective of their design or wrapped up in being right, it may be more difficult to get them to acknowledge change is necessary after observing these tests sessions. A good moderator (and good upfront planning, including getting team buy-in on the test protocol), is necessary. As we like to say here at UIE, it's not rocket science. However, it is a skill that practice will help you master. Understanding the basics will get you so far. Once you've had the chance to do it a few times (or a few hundred times like us), it will feel more natural. B: So, to get us into the mind set for moderating a session, can we envision any other roles that might be a little like the roles we'll assume as the moderator in the test? J: Why as a matter of fact… The Three Basic Roles: Flight Attendant, Sportscaster, Scientist Back in the '90s, Carolyn Snyder who used to work at UIE came up with a great way to explain the differences. She divided all the things the moderator had to do into three parts: * The flight attendant takes care of the participant to make sure they are comfortable and not getting stressed out. * The sportscaster takes care of the observers, to make sure they are getting the most out of the session * The scientist takes care of the study itself, to make sure that things are done according to the plan That's the priority: Flight attendant comes first, sportscaster comes second, scientist comes third The job of the flight attendant As the flight attendant, your job is to keep the participant comfortable. It's stressful being in a usability test. Unfortunately, we call it a "test". You need to remind the user they are not being tested -- we're testing the design. Things we find from them and others will help us improve the design going forward. In this role, you monitor the participant's mood. If they look like they are getting stressed, particularly when things in the design seem difficult, you need to reassure them and remind them that this is very helpful. B: Speaking of "tests," can our nervous testers "fail?" The job of the sportscaster As the sportscaster, your job is to make sure the observers have all the information they need. Observers are in the room to to see the users interact with the design. When we record the podcast we're planning on setting up the "testing lab", we'll talk about where the observers need to be. For now, I'll just say that you need to make sure they can see what's going on. If they can't see what the participant is doing or hear what the participant is saying, they won't get what they need, so it's your job to ensure they can see and hear. In some cases, the participant might do something the observers miss. In this case, you can repeat what the participant said or announce what just happened. (This is also a handy trick to make a note on a videotape, if you're recording the session.) B: So the Sportscaster is going to be doing a little "play by play" to vocally describe some of the important elements, to ensure the observers get what they're there for? In your testing protocol -- the rules that you moderate from -- you should have points designated when the observers will ask questions. As sportscaster, you'll want to facilitate these Q&A periods. (Since the observers are not trained at good questions, you may want to rephrase some of them, just to make sure you don't lead the participant in a way that could hurt your later results or bias their answer.) Some protocols we use allow for the observers to pass notes or send instant messages to the moderator during the working portion of the session. As moderator, your job is to decide if and when you'll answer those questions. Sometimes, observers are in a separate room. In this case, you might want to have a separate moderator to sit with them. The job of the scientist As scientist, your role is to make sure the study happens as you've planned it. Part of this job involves understanding how you'll ask participants to interact with the design. It's about making sure you've gathered all the pre-test information you may want, including their background information and pre-use perceptions of the product or brand. It's also about collecting all the post-test information, such as overall satisfaction and opinions. The scientist collects the data for later analysis. Thanks Jared. In the interest of keeping each of installment short, we'll pause here and end this week's show. During intermission feel free to help yourself to the popcorn and the rest rooms. Look for the second half part of this podcast in the next Usability Tools Podcast. We'll see you then! Episode 1 ends.