Would You Bet Your Life Savings On It?

Jared Spool

May 11th, 2007

The point of a usability test is to determine areas in the design we’d like to improve. Often, when the testing is complete, the team expects us to put together a list of recommendations for them to act on. Each recommendation should point to an improvement which will benefit the design and therefore our organization.

Improvements happen when the design truly eliminates frustration. Organizations benefit from those improvements when the eliminated frustration increases sales (due to a better user experience) or reduces expenses (due to decreased support or development costs).

One problem we often see, however, is when the user researchers get overzealous in their improvement suggestions. They start to make recommendations in cases where there may not be any clear evidence the change will eliminate frustration or improve the design in a measurable way.

The worst case scenario is when someone issues a poorly thought out recommendation, which turns out to make the design more frustrating. When this happens, it hurts the reputation of the user research effort and puts into question other recommendations.

At the recent CHI conference, our good friend Meghan Ede, who runs the user research effort at Symantec, told us she’s instructed her team of researchers to ask the following of every recommendation they write: “Would you bet your life savings on this recommendation improving the design?” They remove any recommendation which doesn’t meet this criteria from the final presentation.

Meghan reports, since she’s instituted the policy of asking this question for every recommendation, the number of recommendations has dramatically decreased, but the quality of their results have substantially improved. The researchers are now more confident when they report their results and the teams are less argumentative.

What do you think of Meghan’s approach?

3 Responses to “Would You Bet Your Life Savings On It?”

  1. Mike Says:

    Love the idea because many of the improvement ideas literally cost that much do. Planning, developement , testing, and installation have a lot of costs associate with it. You need a quick way to chose which improvements you need to do over others.

  2. Zephyr Says:

    I understand the goal behind the question, but, primarily, usability tests identify obstacles, not solutions. When you know the problem, you can start thinking of a solution, but that solution hasn’t actually been tested at that point. Nevertheless, it seems like a good question to make you look critically at your recommendations.

    Unfortunately, many companies don’t seem very critical when judging the validity of recommendations. I’ve seen reports by reputable consultancy companies where some recommendations could not be connected to the data measured.

  3. Benjamin Ho Says:

    What I do is “let the data/evidence speak for itself”. (Taken from CSI.) If I don’t have it on video or on a survey from user feedback, then there’s no proof. Every problem needs to be substantiated by some sort of proof. If there isn’t any of that, then the problem doesn’t really exist – only in our heads. Sometimes even when a user complains about something, it’s not necessarily a big issue – only a preference issue.

    I think a usability practitioner must be able to use discernment, and practice it with every point of data that may affect the overall design of the user experience or interface. Without discernment, there’s no prioritizing. Without prioritizing, there’s no focus.

Add a Comment