Asking Participants to “Pretend” in User Studies

Jared Spool

September 29th, 2008

On one of the member-only lists I hang out on, there’s been a discussion about asking participants in studies to role play in a usability test’s scenario. Instead of saying,

“Find information about the costs for summer camps in Vermont”

the list member asked if there was a problem with using a scenario like this one:

“You’ve got a nine-year-old boy and an eleven-year-old girl. Pretend you need to find a sleep-away camp for both of them and explore what your options are.”

Asking participants to pretend is always a red flag to me. It typically signals that the overall test plan or the recruitment process isn’t doing what it should.

Years ago (before we started number our web versions with things like 2.0), a client asked us to help them with their tests. The agency they’d hired had been recruiting Wall St. execs and asking them to “pretend you’re interested in Leonardo DiCaprio and find out something you don’t know about him.” Of course, these folks weren’t interested in Leo and didn’t work very hard to discover something to satisfy the test moderator.

This prompted us to study the use of scenarios like this in testing. We found that when a participant is pretending, it’s common for their behavior to be very different than when they are actually doing the task for real.

One of the places we kept noticing this was when we watched people shop online. Asking a shopper to pretend to purchase (“Could you find a pair of shoes you might like to buy and put it in your cart?”) produced extremely different behaviors than when we recruited people who needed the product and gave them the cash to make a real purchase. In the former case, they went through motions and skipped steps that we didn’t see when they were considering and purchasing the product for their own true use.

And here’s the kicker: Had the team changed the site’s design based on the data from the pretend purchasers, they would’ve created a design that would’ve prevented sales from the real shoppers. The behaviors were that different. In other words, listening to people pretending could’ve made the site worse and reduced sales substantially. It could’ve been a huge mistake for the team.

The list member mentioned that her clients were pushing this idea, say that they’d “had success with this technique in the past.” If the client thinks “success” means “we watched people and saw things we didn’t think of before”, that might be a good thing. After all, when clients see the design through the user’s eyes, it helps them inform the decisions they’ll make going forward. However, it could also be a bad thing if it leads them in a direction that could make the site worse for people in the real situations.

To help our clients with this, we developed a technique we call interview-based tasks. Instead of asking the participant to pretend, we recruit participants that would likely interact with the design and we interview them to create their tasks in real time.

Instead of asking the participant to pretend “You just got married in the spring and you’re already thinking about a baby,” you would recruit participants that just got married and are considering a new family. (Any good recruiter can find someone like this pretty quickly. If you don’t know any good recruiters, contact me. I do.)

Then you interview each participant about their situation. During the discussion, you and the participant would collaborate to make a task for the test that would be the same scenario as what you had planned, only it will be for real within the context of their life.

[You can find out about interview-based tasks in this article and in this podcast]

Asking participants to pretend could work just fine, as long as they behave the same when pretending as when they are really in those situations. But it could turn out to be a very bad thing. And you won’t know until you compare against real behaviors.

6 Responses to “Asking Participants to “Pretend” in User Studies”

  1. 2008 September 30 - Links for today « My (almost) Daily Links Says:

    [...] Spool on Asking Participants to “Pretend” in User Studies and a great data visualization from [...]

  2. Martin Says:

    I assume this is different for paper prototyping? i.e I’m testing an application at the moment where the results are quite specific, and generating these results on the fly (on paper) is almost impossible!

  3. Benjamin Ho Says:

    Great entry, Jared! Great advice on the road to refining testing principles.

  4. Carolyn Snyder Says:

    I’m often in the same boat as Martin. With a paper prototype, sometimes you’ve prepared just one set of data. I think it’s fine to tell a user, “Ok, so in real life you’d have picked A but we’ll ask you to pretend you chose B and proceed from there.” But asking them to pretend to have a certain perspective and/or motivation is what gets you in trouble. For instance, next week I’m testing a concept (not even a real site yet) aimed at people who are buying a new car. We are recruiting people who intend to buy a new car soon – that’s the important part. None of our research questions pertain to the model of car they’re buying, so the designer is just mocking up one and we’ll ask people to pretend that’s the car they want. This will work well enough for our purposes. But if we recruited anybody and asked them to pretend they’re buying a new car, that would be bad.

  5. When to Use Which User Experience Research Methods - USiT Says:

    [...] it. This can have a big impact on the method you would employ, and highly scripted approaches risks Asking Participants to “Pretend” in User Studies (which Jared Spool recently wrote [...]

  6. Flog: The Official Fluid Blog » Blog Archive » Recruiting Interested Participants for Usability Testing Says:

    [...] Spool recently wrote that pretending and roleplay in usability testing is not as effective as recruiting people with an actual need or real interest in a product or [...]

Add a Comment