The point of least astonishment: how do you anticipate how many people will be needed to reach the ‘point of least astonishment’? Knowing this would enable better planning for how many people to recruit!

Great question. A few years back, Will Schroeder came up with an algorithmic approach to predicting how many users you need for a study to reach the point of least astonishment–that point where you no longer are learning anything new from a study.

Basically, you track how much you’re learning from each test by quantifying the new things you learn. As you continue to run tests, you should start to see a increase in the amount of redundant things you see and a decrease in the new stuff you learn. If you plot these on a curve, you can extrapolate where the curve will approach the zero point.

The formula is helpful, but has a flaw: you have to run tests to determine how many tests you need to run. What we’ve found in practice is experienced usability practitioners get a sense as to how many tests will be useful. If you’re not sure, you err on the side of too many. If you finish the study thinking you saw new stuff in every session, without any real decrease in the rate the new stuff was coming to you, then you run more tests in the next study. Like anything useful, experience helps you work it out.

For the interview-based tasks in the ZipCar example, it sounds like you basically have them do the same task but you present it differently according to how they expressed themselves in the initial interview. Or, is it totally open and could you have all 6 people in the second group do totally different things?

As with all interview-based task techniques, you’ll base the range of tasks for your participants on the range of activities that emerge from the interviews. If all your users do essentially the same things, just with their own terms, you won’t see a variety in the tasks. If their needs are more diverse, your tasks will be more diverse.

One of the attributes of interview-based tasks is there may be tasks in your study only performed by a single participant. That makes it hard to produce statistics like, “7 out of 10 users completed the registration task.” You don’t want to use interview-based tasks if that type of quantification is important to your research.

However, for the purposes of Inherent-Value Testing, quantification is not the primary objective. Instead, we’re trying to learn the design’s value, which is a qualitative analysis. So, it’s ok if each user performs different tasks.