Eyetracking: Worth The Expense?

Jared Spool

June 13th, 2006

Is eyetracking a valuable usability tool? I’m not sure.

One benefit, recently pointed out to me, was it is eye opening for clients to see how a user’s eyes bounce around the page. Clients believe that people read the content on pages in an orderly fashion. Seeing users’ gazes meander across the screen like a drunken sailor, hopping from one element to another, can convince the client their users don’t view the content the same way they do.

And Maria Stone, from the User Experience group over at Google, told me, of the two labs they use for usability testing, the developers prefer to use the lab equipped with the eyetracker because it’s more fun. She believes the developers pay more attention to the test when the little dot is bouncing around the screen.

I do agree that it’s always good when your clients and developers become aware of how users behave. And an eyetracker is a great demonstration of fine grain behavior. In just mere moments, you can easily see how users gaze at the screen. So, I agree eyetrackers have demonstrative value.

But do they have diagnostic value? Can we actually learn what to change in our designs from them?

Well, after watching hundreds of eyetracking tests, I can tell you it’s still really hard to know what you can learn from them.

First, they are expensive devices. It’s not cheap to outfit a lab with a decent eyetracker. (Even Google, which as far as I can tell has all the money in the world, has only outfitted one of their many labs.) In addition to the money spent on the equipment, you have to spend money training people how to use it. Not a cheap proposition.

Second, not every participant can work with an eyetracker. Depending on the hardware, people with a variety of attributes automatically are disqualified from eyetracking. Everything from contact lenses to long eye lashes can get in the way of the device working properly.

Third, they reduce the amount of time you actually collect data from your users. Getting a participant set up and calibrated with the device can take time away from learning about your design. The most valuable piece of any usability test is the time the participant is interacting with your design, not setting up the measurement equipment. What’s worse is many devices lose calibration quickly, forcing the test to stop and the participant to spend more time futzing with recalibrating. This tool time is distracting and not adding to the session’s value.

Fourth, the results are really hard to analyze. The colorful heatmaps are cool (or warm?) to look at, but what are they actually telling you? When someone is gazing at something, is it because they want to look there? Or because the page made them look there? Or because they are resting their eyes there?

When we first started conducting eye tracking, we noticed some interesting behaviors:

  • Participants often would acquire the scroll bar without looking at it. They’d move their mouse over to the right edge of the screen and start scrolling, but their gaze wouldn’t leave the center of display. It seemed they were using their peripheral vision to acquire and use the scroll bar.
  • Participants would orally tell us they couldn’t see something their gaze was focused on. (Women in my life have referred to this as “Male Refrigerator Blindness” — the inability to see something right in front of you.)
  • Participants often would click on objects they barely gazed at. They’d focus their vision on some part of the screen, then move their mouse to some place else to actually click.

From this, we began to question what the eyetracker was actually trying to tell us. It seemed to us that what the user focused their gaze on was not necessarily what they were seeing. So, if the eyetracker doesn’t tell us what a user sees, what does it tell us? I’m not sure.

Eyetracker vendors go to great lengths to try to justify the value of their devices. For example, this vendor used eyetracking to show how changing a useful portion of the SFPD web site to a non-useful graphic encouraged users to spend more time focusing on the site’s navigation, claiming this somehow improved the site’s usability. But the vendor didn’t explain how focusing more on the navigation improved the usability of the site. In fact, it’s likely users spent more time gazing at the navigation because the site was now more unusable.

Eyetracking is fun to watch and produces cool output. It can serve as a good demonstration that users approach designs differently than we imagine. But can we find a useful place in our research process that is worth all the hassle and expense? I’m still not convinced.

45 Responses to “Eyetracking: Worth The Expense?”

  1. Sensorydrive - Stuart Church’s blog » More on the usefulness of eyetracking Says:

    [...] After Michael Hatscher’s critique of eyetracking a couple of weeks ago, it’s very interesting to see that Jared Spool also has reservations. Spool’s conclusions, based on observing hundreds of eyetracking tests, are very similar to the theoretical objections raised by Hatscher: [...]

  2. Daniel Szuc Says:

    What if eye tracking studies could help tell us where users were looking for certain components on a repeatable basis e.g. Help, Search, Shopping Cart etc, could we overlay this with what we are learning about Design Patterns to inform the foundations of design?

    That is, we have confidence to place the component on this section of a design because we know people look there regularly for that component type.

  3. Jared Spool Says:

    Daniel,

    The folks at Wichita State came up with a much cheaper solution to collect that data.

    And it’s not clear knowing expectations are that useful anyways.

  4. Eddie Says:

    It seems like you’re throwing the baby out with the bathwater a bit. I agree you’re likely not going to answer every question, and it may not be enough value added to justify the expense… but these examples seem lacking…
    Points one through three are spot on though. Those are the same things I would have said if anyone asked “what’s the downside to eyetracking?”
    But the others I have some questions on…

    When someone is gazing at something, is it because they want to look there? Or because the page made them look there? Or because they are resting their eyes there?

    If they are just resting their eyes, or focusing on something- does it *always* matter? I would use eyetracking for the “hunt” that goes on -it seems most useful in during the part where the user is actively moving around looking for the next step to accomplish the task. Where they are resting or focusing is less important than where they decide to go next. I will admit though- I can see how this would skew the heatmap generated at the end.

    Participants often would acquire the scroll bar without looking at it.

    ok- but scroll bars are a bad example to use here. We all know it’s “somewhere to the right.” Assuming the window is maximized, we could probably all get there without looking right?

    Participants would orally tell us they couldn’t see something their gaze was focused on.

    But isn’t that useful to know as well? Sure you could space out and just not *see* but I have to imagine that’s a “less than often” occurance no? The other aspect could be that your using a image, word, etc. that is not universally, or culturally known. So say- you have a graphic of a mailbox for your email link (assuming this is 1995 of course :) ) but your testing it with a user that has never seen a “mailbox” before but knows what email is. So in that case, the user could be looking right at it, but not see it. Isn’t that a useful piece of information to know?

    Participants often would click on objects they barely gazed at.

    Like the scroll bar? ..the back button of a browser? …the “home” link they’ve already clicked on six times this session? In other words, are they barely gazing at things they are already well familiar with?

    I’ve only seen a handful of eye tracking studies, so I’ll certainly take your experiences over mine. But your points from 4 on (I don’t think) paint a strong enough picture against eye tracking.

  5. Michael Zuschlag Says:

    It is correct that there’s often ambiguity associated with eye tracking data that can frustrate interpretation if you approach it in a disorganized manner. A long gaze time on piece of content might mean it is easy to read and understand so the user depends on it for detailed information, or it might mean it is hard to read and understand so the users must stare at it for a long time to extract even minimal meaning. For this reason, it’s hard to know what to make out of raw “heat maps” in isolation.

    On the other hand, eye-tracking combined with other data can be very helpful. For example, if users are not using a specific menu option that they should for their task, you can look at eye tracking data to see if they are even looking at it. If not, then one solution is to move the menu item to where users *are* looking. If users are already looking at it, then the more likely improvement is to give it a more meaningful label.

    Eye-tracking is just another dependent measure. Whether an eye-tracker is worth the expense and hassle depends on what questions you’re asking and whether you are getting satisfactory answers using your current techniques.

  6. Jared Spool Says:

    So, Michael, since eye tracking can raise the cost per user by as much as a factor of 3, when is it worth spending the extra money on?

    There are many other ways to discover if the reason people don’t click on something is because they aren’t “seeing” it (such as oral interviews or retrospective analysis). These are much cheaper than the added expense of eye tracking.

    I know these devices are all the rage. I just think our client’s money should be spent on better things.

  7. Michael Zuschlag Says:

    When is eye-tracking worth 3 times the cost per user? Oh, I don’t know, maybe if it provides 1.5 times the information or accuracy over the alternatives? If it cuts the number of alternative design solutions by 2/3rds on average, then it reduces your design iterations by 2/3rds. That’s 2/3rds the number of users to test, 2/3 the number of usability tests to set up, 2/3rds the amount of re-design work, faster delivery of the product to market… yeah, I’d guess that’ll probably add up to offset the higher cost per user. YMMV.

    I’d need to compare eye-tracking to the alternatives to see what level of relative performance it provides when. It’s hard to imagine a making a decent cost-benefit analysis without at least an estimate of that. But, yeah, make some attempt to look at the costs and benefits. Don’t buy a gadget just because it has stirred a buzz.

  8. Jared Spool Says:

    Michael, you’re right, if it did all those things. But the way we’re seeing it used, it doesn’t reduce the number of tests or cut the number of iterations. I’d love to know if anyone who has an eye tracking lab actually is seeing faster design evolution because of it. I just haven’t seen that from any of our clients.

    With eye tracking, you have to schedule more users (because not every user works and you can’t tell until you sit them down in front of the device), you have to shorten your test times (because calibrating with the machine frequently eats up 15 to 20 minutes), you have to lengthen the data analysis (because the current tools for analyzing eye tracking data are crufty and open to wide interpretation), and when you’re all done, you don’t know a whole lot more than you did before.

    That’s my take.

    If you want to buy a nifty new toy, I recommend the wide selection at Think Geek. You can get something wonderful there for under $50 that won’t affect your testing costs. :)

  9. Rob Tannen Says:

    Getting back to the main point, this is ultimately an issue of cost/benefit. If it were free, many more people would use it. The jury may still be out on its effectiveness on research/design quality for the Web, but it has been used for many years as useful tool for complex user interface design – yes there is more to user interface design than Web sites.

    Like any tools, its use is limited by appropriate application. I believe Eye Tracking adds value IF used appropriately – that is, with the context of other quantiative and qualitative data.

    Keep in mind that:

    -Calibration with a new user takes less than five minutes

    -Eye Tracking can be done simultaneously with other usability methods (e.g. think alound protocol); they are not mutually exclusive

    Only in the fuzzy world of Web usability would a tool that actually provides valid (accurate) and reliable (repeatable) measurement be considered a potential impediment.

  10. Rob Tannen Says:

    Also, in the case of “refridgerator blindness” it sounds like ther is an assumption that what the user self-reports is more veridical than what the eye tracker shows. In fact the eye tracker data may be indicating that user’s actions may be driven by unconcscious behaviors. Now would what that do to self-reporting as a mainstay of usability testing?

  11. Yuri Says:

    Isn’t it just amazing how complicated things are? Now you need to get someone to track someones’ eyes, wandering on your website, along with track the path they visit to understand what they want. Why not just think hard of best ways of presenting the information, without relying on the technology.

    Of course, you’ll need to do user testing to see what works and what doesn’t, but it is not in tracking the visitor paths. The matter is in visitor and site interaction.

  12. teehan+lax Says:

    [...] Jared Spool recently weighed in on the subject on his company’s blog, articulating what we think is a more balanced and down-to-earth view of the situation. [...]

  13. Martin Dowson Says:

    Jared,

    this view sounds like the same view the industry has been espousing of eye-tracking for the last five years. Things have definitely moved on, but often this industry’s views dont! We’re now looking at methods of interviewing with eye-tracking that combine the best of what the eye tracking data has to offer with the best of protocols like talk-aloud have to offer. Most reputable companies selling eye-tracking technology will suggest combinations not JUST eye-tracking so let’s not evaluate it as a standalone.

    Eye-tracking can be useful in ways that extend beyond the company web-site. It can be useful to also evaluate communication material… In a large corporate do users really read the instructions on their system, if they are struggling with a screen… and it has instructions on it… how did they fare with the e-learning training that was deliverd when the software was rolled out… tracking eye movements across screens can help you understand if users are really reading these elements. that is one specific example…

    the equipment can be used to test materials that are web-based, paper based (advertisements, magazines), any kind of software at all. The thing I like about this is that it takes your client’s business as a whole and, through one tool, you are able to gather very interesting data. Think about evaluating someone’s use of a consumer goods web-site, magazine advertising AND marrying that data up with their movement through a grocery store as they scan the shelves for their purchases…

    Is it a GREAT tool for web-site design… it depends how good your designers are… the more user-centred you are in your design and methodology the less you’ll need to do testing full stop… but if, like I do, you’re dealing with the whole business, not just a web-site, tools like eye-tracking can really expand the range within which your user-experience expertise can be used…

    What’s more… the technology is fast to set-up (5 minutes), calibrate to a user (2 minutes) and learn to use (me… from scratch… 20 minutes)… As an outright purchase in a lab I think can show it earning it’s money back in about a year through the expansion of applicability, meaning that it’s use in the standard website tests is covered.

    all the best

    Martin Dowson

  14. Louise Hewitt Says:

    I too have problems with eye-tracking. Working with small scale agencies and clients, I hear too often that they have no budget for IA consultants, no time for user-testing, no need for either. Stick them infront of an eye-tracking demo though and their cheque books verily fly out of their back pockets.
    Like many things in life, people like it ‘cos it’s cool. I have yet to come across a convincing and scientifically based interpretation of the results that would hold as a rule for every user in every circumstance. There are just so many variables to consider that it must cancel out the value of the results. Where is the control test!

    Bah! I’m ranting. But I’ll stop now and say that if IA could come up with something that looked cool to clients and actually delivered useful results then we’d have cracked the toughest nut. Maybe we should register brain waves.

  15. UIE Brain Sparks » Blog Archive » Do Links Need Underlines? Says:

    [...] Many people who use the web for a long time start to become conditioned to look for underlines. If you watch them with an eye tracker, you can see their focus dart from underlined-text to underlined-text when they first see a page. [...]

  16. Formed Function » Blog Archive » Eyetracking: nuttige usabilitymethode of niet? Says:

    [...] http://www.uie.com/brainsparks/2006/06/13/eyetracking-worth-the-expense/ http://www.user-experience-design.com/?p=27 [...]

  17. UIE Brain Sparks » Blog Archive » The Death March for Advertising Says:

    [...] If you watch users interact with web sites containing advertising, you quickly notice how the users develop techniques to avoid looking at the ads. We’re not the only ones seeing this. Anybody who watches users with an eye tracker (this may be the one good use of them) on pages with advertising can see how users avoid looking at the ads. Others have seen how users even avoid looking at the innocuous Ad-sense ads that populate many sites. [...]

  18. user-experience-design.com Says:

    [...] Following up on my critique of eye tracking as a technique for gathering usabilty input, I just discovered a post from Jared Spool from mid-June titled “Eyetracking: Worth the Expense?” Great to see that Jared seconds my thoughts from his wealth of experience: [...]

  19. Brandon Says:

    It’s now more than a year later. The technology has improved quite a bit. Calibration and “orientation” takes no more than 5 minutes and is almost never lost in the middle of a session. Now that it’s less intrusive time-wise and, I imagine, possibly a bit cheaper, is it any more useful to the usability profession?

    It’s certainly still new technology, and even now I think we’re not sure what to truly make of what it can do. Perhaps it’s up to universities to do some research with it, particularly in concert with psychologists, to tell us what we are and aren’t likely to be able to take away from results.

  20. Jared Spool Says:

    Brandon asks,

    Now that it’s less intrusive time-wise and, I imagine, possibly a bit cheaper, is it any more useful to the usability profession?

    I don’t think so. We still don’t know how to read the data. It depends on (what I believe to be) sketchy interpretations to bring meaning and you still don’t learn anything you couldn’t learn without the device.

  21. Stephen Oliver Says:

    I do not know of any eye tracker that takes more than 15-20 seconds to calibrate and having undertaken thousands of data collections over the last 25 years I can state that fewer than 5% of subjects will be unsuccessful, if the operator knows what they are doing. The key to obtaining data that is useful to determine whether alterations to a page design are appropriate is to show the respondent what they looked at and ask questions about their actions and observations. Without fail you get responses like;
    ” I didn’t understand what that (text) meant” ” I wasn’t sure what to do next or if that was a link” A decent analysis program like Gazetracker will also provide mouse click and key stroke data. Every usability study we have undertaken has resulted in changes and the simple fact is that eye tracking provides information about a users behaviour that you would not otherwise discover.

  22. Peter Irons Says:

    I have been using eyetrackers for several years, looking at the ease with people read text on screen.
    It is clear that the spatial/temporaldata which a retina collects mduring na fixation is not necessarily processed in terms of ‘attention’/awareness. The two eyes are often looking at quite different parts of a screen. It can appear if you are doing monocular eyetracking that one eye is collecting data from one part of the screen and the other from elsewhere. A sort of parallel processing.
    e.g a pianist is collectin g data from separate staves, one for the right hand , one for the left.other wise they would be unable to sight read.

  23. Markus Joos Says:

    @Peter
    Your notion of the two eyes processing visual data in parallel is wrong. Both eyes do converge on a single point. If they don’t, as for e.g. in case of squinting, the information from one eye is blocked resulting in amblyopia.
    @Jared
    I totally agree with your view that eye tracking in itself is not necessarily the one and for all method in usability research. And I agree that the value of a simple heatmap is limited to answer questions about usability. But let me give you an example from one of our customers where eye tracking actually added to the interpretation of a usability study: Subjects task was to use two versions of a prototype search website. During the task their eye movements were collected and after the task they were asked which version they would prefer. 70 % preferred one version, 30 % the other. Well, one could they mission accomplished, the 70% version is the one to go with. But the much more interesting question is, why did the 30 % prefer the other version ? In that case our client did a comparative attentional landscape analysis contrasting attention between the 30% and the 70 % group. And just with a few clicks it turned out that the 30 % missed some navigational elements the 70 % group has actively paid attention to. So what was the conclusion? Obviously these navigational elements where important to prefer one version. But since they were missed by 30% it was good advice to increase visibility of these elements.

    To summarize: I think that eye tracking is not a simple tool to allow for a direct measure of usability. But properly used and in combination with other usability testing techniques it can give insight into usability issues which would otherwise be hard to find out.

  24. Reflections: User Research Smoke & Mirrors « jenjenjen… Design Blog Says:

    [...] another blog linked from the article, it mentioned some interesting finding and I thought I should just share it [...]

  25. 33 Website Success Metrics Instead of Rankings, Google PageRank and Traffic | SEOptimise Says:

    [...] maps of actual eye movements. You need more than a web analytics package to check that you need real people to take part in a study but if you are large company depending on your website you should check this for sure. Do people [...]

  26. Online Marketing Blog » Blog Archive » 33 Website Success Metrics Instead of Rankings, Google PageRank and Traffic Says:

    [...] maps of actual eye movements. You need more than a web analytics package to check that you need real people to take part in a study but if you are large company depending on your website you should check this for sure. Do people [...]

  27. 33 Website Success Metrics Instead of Rankings, Google PageRank and Traffic · SEO Guide - Tips, Tricks and Secrets Says:

    [...] eye movements. You requirement more than a web analytics package to analyse that you requirement real grouping to take conception in a study but if you are large consort depending on your website you should analyse this for sure. Do [...]

  28. Alt attribute Vs. title attribute in links and images · SEO Marketing Tips - Tricks, Techniques & Secrets Says:

    [...] movements. You requirement more than a scheme analytics package to analyse that you requirement real grouping to verify conception in a study but if you are super consort depending on your website you should analyse this for sure. Do [...]

  29. Embracing the Dark Side of Social Media — and How To Make Money From It · SEO Marketing Tips - Tricks, Techniques & Secrets Says:

    [...] movements. You requirement more than a scheme analytics package to analyse that you requirement real grouping to avow conception in a study but if you are super consort depending on your website you should analyse this for sure. Do [...]

  30. Net Neutrality article · SEO Guide - Tips, Tricks and Secrets Says:

    [...] eye movements. You requirement more than a web analytics package to analyse that you requirement real grouping to take conception in a study but if you are large consort depending on your website you should analyse this for sure. Do [...]

  31. The value of click-stream data derived from toolbars « Nowhere North Says:

    [...] User interviews! Good qualitative research, but not always easy. But remember what Jared Spool said about eye tracking in: “Eyetracking: worth the expense?“. [...]

  32. McCool Says:

    My thoughts on this is that the tests themselves would be more relevant if we could measure brain wave patterns to go along with the eye-movement.

  33. Müssen Links nun unterstrichen sein? | //SEIBERT/MEDIA Weblog Says:

    [...] nach und nach auf die Suche nach Unterstreichungen konditioniert. Beobachtet man sie mit einem Eyetracker, sieht man sehr schön, wie ihre Aufmerksamkeit von Unterstreichung zu Unterstreichung rast, wenn [...]

  34. Top Ten Eyetracking Blogs | The Acagamic Says:

    [...] Interface Engineering. Again, not really a blog, but a company site that has a few articles related to eye [...]

  35. What is Usability metrics? | CSS Resources Says:

    [...] maps of actual eye movements. You need more than a web analytics package to check that you need real people to take part in a study but if you are large company depending on your website you should check this for sure. Do people [...]

  36. Usability metrics | CSS Experiments Says:

    [...] maps of actual eye movements. You need more than a web analytics package to check that you need real people to take part in a study but if you are large company depending on your website you should check this for sure. Do people [...]

  37. De Web Times - Share Your Resources » Blog Archive » Eye tracking: Eye candy vs. I can do Says:

    [...] also been a bit of a backlash against eye tracking recently. Jarred Spool, for example, has dismissed it as an expensive gadget that doesn’t tell a usability analyst [...]

  38. Robert Stevens Says:

    There is much poor practise in the eyetracking industry, some of this is unfortunately being perpetrated by Tobii: http://bit.ly/6wyFa and http://bit.ly/em93W . This is symptomatic of the lack of understanding of how, why and when to use eyetracking within the industry.

    Fortunately Kara Prenice and I will be debating eyetracking at the UK UPA meeting on the 20th May 2009. My position is ‘Why you NEED eyetracking for web usability studies.’ Kara will be opposing this view.

    As part of my argument I’ll be sharing how, why and when Bunnyfoot deploys eyetracking, I hope this will bring some clarity to the industry. In the mean time you can see one reason why we use eyetracking at Bunnyfoot in a short interview here: http://www.youtube.com/watch?v=NyNImF67nUk

  39. Mesurer le succès de votre site sans le trafic, les classements et Google PageRank - 2ème Partie | Le Blog Kinoa Says:

    [...] Mieux encore que les “heat maps” qui retrace uniquement le comportement sur les clics, “l’eyetracking” permettent de suivre le parcours de l’oeil d’un utilisateur sur un site web. Il faudra alors en plus des outils de web analytique, disposer d’un panel de personnes qui se prêteront aux tests. [...]

  40. Is eyetracking worth it? | Erin Lynn Young Says:

    [...] to run a couple of rounds of user research. In a June 2006 article by Jared Spool entitled “Eyetracking: Worth the Expense?“, he notes: Well, after watching hundreds of eyetracking tests, I can tell you it’s still [...]

  41. Eyetracking Metrics for Usability Studies | inspireUX - User Experience quotes and articles to inspire and connect the UX community Says:

    [...] Eyetracking: Worth the Expense? [...]

  42. Eyestrack « abcendigital Says:

    [...] J.: “Eyetracking: Worth The Expense?” (2006), UIE Brain Sparks Jared (Junio [...]

  43. Greg Zobel Says:

    Expense is a critical issue, especially if an eye tracking system costs over seven or ten thousand dollars. It is exactly this reason, expense, that Grinbath developed EyeGuide: an eye tracking hardware/software solution under $1,500. Only by working to lower the price and by making eye tracking ubiquitous and available to virtually everyone will we reap the full benefits that eye tracking can offer. Hopefully cost will no longer be an issue for people who want or need to conduct eye tracking research.

  44. carly Says:

    thank for post.
    http://www.lovebargaining.com/volkswagen-vw-bora-wheel-badge-wheel-centre-cap-hub-caps-car-emblem-badges-for-vw-bora-55mm-p-566.html

  45. Eye Tracking: What Is It For And When To Use It - Usability Geek Says:

    […] every impressive demonstration is hours of effort and interpretation. Some of the industry’s foremost experts remain conflicted as to its value, and it’s still only undertaken in a minority of web design projects. Despite plummeting costs […]

Add a Comment