September 29th, 2008
On one of the member-only lists I hang out on, there’s been a discussion about asking participants in studies to role play in a usability test’s scenario. Instead of saying,
“Find information about the costs for summer camps in Vermont”
the list member asked if there was a problem with using a scenario like this one:
“You’ve got a nine-year-old boy and an eleven-year-old girl. Pretend you need to find a sleep-away camp for both of them and explore what your options are.”
Asking participants to pretend is always a red flag to me. It typically signals that the overall test plan or the recruitment process isn’t doing what it should.
Years ago (before we started number our web versions with things like 2.0), a client asked us to help them with their tests. The agency they’d hired had been recruiting Wall St. execs and asking them to “pretend you’re interested in Leonardo DiCaprio and find out something you don’t know about him.” Of course, these folks weren’t interested in Leo and didn’t work very hard to discover something to satisfy the test moderator.
This prompted us to study the use of scenarios like this in testing. We found that when a participant is pretending, it’s common for their behavior to be very different than when they are actually doing the task for real.
One of the places we kept noticing this was when we watched people shop online. Asking a shopper to pretend to purchase (“Could you find a pair of shoes you might like to buy and put it in your cart?”) produced extremely different behaviors than when we recruited people who needed the product and gave them the cash to make a real purchase. In the former case, they went through motions and skipped steps that we didn’t see when they were considering and purchasing the product for their own true use.
And here’s the kicker: Had the team changed the site’s design based on the data from the pretend purchasers, they would’ve created a design that would’ve prevented sales from the real shoppers. The behaviors were that different. In other words, listening to people pretending could’ve made the site worse and reduced sales substantially. It could’ve been a huge mistake for the team.
The list member mentioned that her clients were pushing this idea, say that they’d “had success with this technique in the past.” If the client thinks “success” means “we watched people and saw things we didn’t think of before”, that might be a good thing. After all, when clients see the design through the user’s eyes, it helps them inform the decisions they’ll make going forward. However, it could also be a bad thing if it leads them in a direction that could make the site worse for people in the real situations.
To help our clients with this, we developed a technique we call interview-based tasks. Instead of asking the participant to pretend, we recruit participants that would likely interact with the design and we interview them to create their tasks in real time.
Instead of asking the participant to pretend “You just got married in the spring and you’re already thinking about a baby,” you would recruit participants that just got married and are considering a new family. (Any good recruiter can find someone like this pretty quickly. If you don’t know any good recruiters, contact me. I do.)
Then you interview each participant about their situation. During the discussion, you and the participant would collaborate to make a task for the test that would be the same scenario as what you had planned, only it will be for real within the context of their life.
Asking participants to pretend could work just fine, as long as they behave the same when pretending as when they are really in those situations. But it could turn out to be a very bad thing. And you won’t know until you compare against real behaviors.Tweet