Usability Tools Podcast: Statistical Significance

Jared Spool

October 22nd, 2007

UIE Usability Tools Podcast: Statistical Significance
Recorded: October 19th, 2007 from the studios of UIE
Brian Christiansen, UIE Podcast Producer
Duration: 26 min | File size: 15.5 MB
[ Subscribe to our podcast via iTunes. This link will launch the iTunes application.]
[ Subscribe with other podcast applications. ]
[ Text Transcript ]


Each week in our Usability Tools Podcast, I will be sitting down with UIE’s Managing Director, Christine Perfetti to discuss tips and tools for improving your site’s user experience. The goal of our weekly podcast is to share some of the most important findings from UIE’s research on web design and usability.

This week, Christine was tied up getting ready for our upcoming UI12 conference, so Brian Christiansen joins me for a show whose topic came from our reader, Colin. He wrote in to ask about the statistical significance of usability or field tests.

Statistical significance revolves around having enough participants to make your findings valid. However, the number of participants necessary can vary widely, depending on what you’re studying and how.

During the podcast, Brian and I answered these common questions:

» Many people talk about statistical significance, but what does it really mean?
» How many people does it take to produce reliable results?
» Why do usability tests require many fewer participants than surveys?
» How do we know we’ve gathered enough data from our research?

As always, we’re very interested in hearing from you. Do you have questions or comments about this episode? We’d love to create a show based on your questions, just like we did with this episode. Please leave a comment below or email us directly at

UIE’s Latest Research: If you’re interested in the topics Christine and I discuss in the podcasts, I highly suggest you sign up for our free newsletter, UIEtips, to read our latest usability and design research as soon as we publish it. We’ll also notify you in UIEtips when we publish new podcasts.

New: Survey and listener drawing!
We would like to give you and your co-workers free admission to our next Virtual Seminar program, with full, lifetime access to the archived program as well! All you need to do to win is give us your feedback on your podcast listening experience. Fill out the following survey and each week we’ll randomly send one survey participant a free admission to the next UIE Virtual Seminar and Archive, a $169.00 value! We appreciate your input!

Participate in our survey to win!

5 Responses to “Usability Tools Podcast: Statistical Significance”

  1. Daniel Szuc Says:

    We welcome an opportunity to test with more users — but often its a case of can the client do it given “time” and “budget”.

    In research where we have tested with 8+ users, its nice to be able to have the flexibility to adjust the test plan, investigation questions, iterate the designs accordingly (using wire frames) and test for new findings during the test.

    This works well but requires the client to understand an iterative and more agile approach 🙂

  2. Colin Says:

    Thanks, guys! It’s good to be armed with some other ideas, since this question will keep coming back from potshot-takers both informed and un-.

  3. Significância estatística em testes de usabilidade « Blog de Pirauá Says:

    […] Se você entende inglês falado, ouça o podcast. […]

  4. Cathy Says:

    How many users do you need to show that podcasts stink? Not this one in particular mind you – podcasts in general. You can’t browse them, you can’t search them, you can’t make them go faster – how is this good?

    I’d love to see a study that took a group of podcasts and compared offering just the podcast versus just a transcript versus offering both. Anyone?

  5. Laurie McCanna Says:

    I was disappointed to follow a link to what I thought would be an article that I could read immediately…and found a podcast without a transcript. In order to get the information, I would need to remember to add the link to my podcast subscriptions (on my home computer, since we can’t use iTunes at work), download the file, and listen to it to find the information….at least 6 hours later, if I bother at all. This is an improved experience over a text page how?

    Like flash sites that offer non-flash versions, I really think that informational podcasts should offer a text transcript.

Add a Comment