Six Slick Tests for Docs and Help

By Tara Scanlon

Originally published: Sep 01, 1998

Usability testing isn’t just for software and web sites. Testing documentation can ensure that it includes — and accurately conveys — all the information users expect and need.

Testing gives you accurate information on how well your documentation and Help work. It can even uncover problems that are better solved by changing the interface.

As with most parts of the development, it’s easier and less expensive to find and correct doc problems early in the process, so we try to test as soon as possible.

Choosing a method depends on what you want to learn, how much time you have, and where your greatest risks are.

1. Incredibly Intelligent Help

This technique lets you discover what information users actually need, before you’ve written anything.

We bring in test participants as soon as we have a version of the product that will let users do some of the tasks they’d do in the real world. This may be well before we’ve written the documentation, but this isn’t a problem because a member of the development team simulates the documentation.

The incredibly intelligent part comes in when users get stuck or seem confused. At this point, the person simulating the docs asks, "What is your question right now?" When the user replies, the Help person provides a verbal clue, but gives as small a piece of information as possible. If this isn’t enough, the user can ask another question.

By giving information only a bit at a time, we can find out exactly which piece is most important to the user. Knowing this helps teams provide appropriate documentation — not too much, not too little — or, better yet, to fix the interface itself.

2. Index Tests

You can determine whether users can get to the right topic in your index by having them work with the draft index. You can even test the index before you’ve written the documentation.

To test the index, we show users screen shots from the product and ask them to write down the first three terms they’d look up to find more information. This test lets you look inside users’ heads and learn some of the terms they use — and expect to see. We write down all the terms they use; they could be words we’ve omitted or not referenced. The more of these we include, the better the index will work.

3. Summary Tests

By testing how well users’ comprehend conceptual material, you’ll know how successfully you’re communicating the information.

We ask users to read a section or examine a picture in the docs, then ask them to describe the three main points this element is trying to convey. (You should already know what you expect them to be!) If the users’ answers match yours, you’re probably getting your information across; if not, you may have some work to do.

4. Procedural Tests

This is a good way to test how accurately you’ve written the procedural details of your docs.

We give users the documentation or Help, asking them to read a specific page or topic and then work with the software to complete the procedure this section describes.

We note where they have problems, where they get confused, or where steps are missing. Then we make the necessary changes in the next draft.

5. Lookup Tests

Lookup tests demonstrate if the docs or Help include the necessary information — and if users can get to the right topic and use it. They also let you find the users’ biggest unanswered question.

To do a lookup test, we give users questions to answer or tasks to complete and see if the draft documentation helps them succeed. For example, when testing the documentation for a video product, we told users, "The picture you’re getting isn’t as clear as you’d like. Use the documentation and online help to find out what’s wrong and how to fix it."

After conducting lookup tests, some teams have changed from printed to online help (or vice versa), added index entries, or reorganized the information.

6. Rose in the Thorns Tests

It can be enlightening — and fascinating — to discover what experienced users don’t know about a product. To do this, we’ve recently started experimenting with the Rose in the Thorns test.

We ask experienced users to look through the documentation or Help until they find information about features they didn’t know before.

Then we ask them if this new information is valuable to them. If there are important features users don’t know about, we work with the interface designers to communicate them better.

Tips for Success

Three things help ensure successful tests of docs and help:

These tests work best with a couple of users at a time. This increases the likelihood of brainstorming and lets us collect lots of information or keywords at once.
We try to avoid product-specific terms in the task instructions.
After the tests, we discuss with the team what we’ve seen and decide on next steps — including changes to the docs or more testing.