Comments on: Two Simple Post-Test Questions

By: Jeff Bridgforth » Some Best Practices from Jared Spool and UIE

Jeff Bridgforth » Some Best Practices from Jared Spool and UIE — Tue, 11 Apr 2006 13:02:43 +0000

[...] Two simple post-test questions – these are two questions they ask users after a usability test. I could see using these questions with others to evaluate a current site or in the process of a redesign. The questions are: “what are two things about the design that you liked?” and “what are two things about the design that you did not like?”. [...]

By: UIE Brain Sparks

UIE Brain Sparks — Thu, 30 Mar 2006 11:59:54 +0000

The One-Minute Test

A simple test we use to tell if everyone just sat through the same meeting.

…

By: E. Ears

E. Ears — Mon, 27 Mar 2006 10:28:07 +0000

Jay said:
It’s always interesting when they list something they obviously struggled with as “easy”.

This can also be because they don’t want to look stupid, even though no-one else would think they were, some may think a task is supposed to be easy or would be considered so by others and although they didn’t find it so say it was.

Fear of looking stupid affects a lot of people’s behaviour!

By: Jared Spool

Jared Spool — Mon, 27 Mar 2006 07:11:04 +0000

Mike wrote:

direction in usability testing training always seems to focus more on the “pay attention to what they do, not what they say” concept.

How does actually asking them what they like affect that?

We ask the questions after we’ve seen them work with the design. We compare the answers to the observations we made. Often, they match, thereby giving us perspective into what the user perceived as well as what they did.

In the cases where they don’t, we pay more attention to what they did than what they said. However, it tells us that what we saw doesn’t match what the user perceived, which can help us down the road when we’re trying to evaluate the severity of the issues.

By: Eric Smith

Eric Smith — Fri, 24 Mar 2006 19:26:54 +0000

These summary questions are also something I regularly ask every participant towards the end of a test session. Aggregating the “issue” findings with all participants and ordering it according to frequency provides potent backup data for design recommendations. Invariably, there are often one or two criticisms suggesting improvements that may not have been previously considered by the design team.

In my practice, I adminstered the questions silightly differently by asking for things participants “liked” and “liked least.” This phrasing of the negative issues gets around any objection a participant might have to inventing something they didn’t like when in actuality they liked everything about the product/website. In my experience, most participants have little difficulty describing those less positive aspects. For evaluating a more complex website/product, I further modify the questions and ask participants for 3 things liked and liked least to increase amount of collected data and possibly expose more repetition in the feedback.

By: Artie Pajak

Artie Pajak — Thu, 23 Mar 2006 19:22:18 +0000

I’ve been questioning the value of the data I get from post-test questions because, like Jay said, people will say it was easy even though they’ve obviously struggled. Is it just that they don’t want to hurt someone’s feelings or what? I’m always careful to state (sometimes several times at different points) “Don’t worry about hurting anyone’s feelings”, “I didn’t design this”, “Your responses will be anonymous”, etc. but they are still reluctant to share “bad” news verbally or in writing. Maybe it just depends on the user base.

I like Jared’s questions, because it gets at data that isn’t obvious from observation. Asking people whether something is easy to use after you’ve watched whether or not it was easy for them to use is almost like asking a politician if they support children. What’s the point? You already know what the answer is.

By asking them what they like, it seems like one of three things can happen. 1. It gets at the same data you observed (because people often lump their likes with ease-of-use); 2. It gets you nothing but useless subjective design ideas (user’s are not designers); or 3. It could take you on a path you weren’t expecting. Both of the first two are pretty easy to discount. The third can give you some nuggets that you can use to prop up the team, make them realize what’s most important, or, if a pattern starts emerging, it could give you some different insight into what to keep or get rid of.

By: Jay Zipursky

Jay Zipursky — Thu, 23 Mar 2006 16:41:18 +0000

I’ve been asking for “easiest to use” and “hardest to use” things on my written post-test questionnaire. I think I got these questions from Jeffry Rubin’s usabiliy testing book and I think they are more in line with studying usability (as opposed to say, usefulness, which I imagine Jared’s questions may uncover).

It’s always interesting when they list something they obviously struggled with as “easy”. Sometimes it’s because it’s simply better than what they have today and other times it seems to be because the learning curve was difficult but they think it will be easier in the future.

I do like noting the speed of their responses. I may switch to getting a verbal response.

By: mike madaio

mike madaio — Thu, 23 Mar 2006 15:06:37 +0000

I find it interesting how you word those questions — I always try to stay away from questions about “like” or “want”, because I find that people generally have a difficult time understanding and articulating what they truly like or want.

Furthermore, direction in usability testing training always seems to focus more on the “pay attention to what they do, not what they say” concept.

How does actually asking them what they like affect that?