Comments on: Determining Usability from Analytics http://www.uie.com/brainsparks/2006/03/08/determining-usability-from-analytics/ UIE\'s latest insights on the world of design Sat, 11 Feb 2012 08:11:47 +0000 hourly 1 http://wordpress.org/?v=3.2 By: Ace your user assistance - by Becky Lash, Epic Trends » Blog Archive » Web logs have limits http://www.uie.com/brainsparks/2006/03/08/determining-usability-from-analytics/comment-page-1/#comment-11969 Ace your user assistance - by Becky Lash, Epic Trends » Blog Archive » Web logs have limits Wed, 07 Jun 2006 11:53:00 +0000 http://www.uie.com/brainsparks/?p=198#comment-11969 [...] Determining Usability from Analytics [...] [...] Determining Usability from Analytics [...]

]]> By: Los textos de qweos.net» Blog Archive » El accidente, la crónica, los libros y MapSurface http://www.uie.com/brainsparks/2006/03/08/determining-usability-from-analytics/comment-page-1/#comment-1993 Los textos de qweos.net» Blog Archive » El accidente, la crónica, los libros y MapSurface Fri, 10 Mar 2006 17:24:06 +0000 http://www.uie.com/brainsparks/?p=198#comment-1993 [...] Jared Spool, de UIE Brain Sparks [...] [...] Jared Spool, de UIE Brain Sparks [...]

]]> By: Jared Spool http://www.uie.com/brainsparks/2006/03/08/determining-usability-from-analytics/comment-page-1/#comment-1938 Jared Spool Thu, 09 Mar 2006 13:16:45 +0000 http://www.uie.com/brainsparks/?p=198#comment-1938 Jeremy wrote: <blockquote>In the example you cite, usability observation tests could have uncovered the problem with site confusion, and you could have tested software improvements with analytics: less confused users would result in higher booking rates in the hotel flow, and lower dropout rates in both the hotel and theme park flows. You also could have measured rate of booking for users who jumped from the hotel flow to the theme park and watched that change with rollouts of software changes.</blockquote> If you could isolate out those who <em>intend</em> to book a room at a WDW hotel from everyone else, then the analytics would be very helpful. However, millions of people visit Disney.com every day with only a segment of them ready to book a hotel at the theme park. Plus, Disney spends millions on promoting their resorts. How do you know that the increased booking rates are due to the changes to the site and not to something else? Maybe, on that particular day, the number of people who aren't interested in booking a hotel just dropped off and artificially raised the percentages. We have to be very careful when drawing inferences from analytic data that we're not just resorting to hopeful thinking and seeing what we want to see. That just hurts us in the long run. Until the analytics package can isolate streams by the intent of the user, I think they will continue to be very limited in their usefulness. Jeremy wrote:

In the example you cite, usability observation tests could have uncovered the problem with site confusion, and you could have tested software improvements with analytics: less confused users would result in higher booking rates in the hotel flow, and lower dropout rates in both the hotel and theme park flows. You also could have measured rate of booking for users who jumped from the hotel flow to the theme park and watched that change with rollouts of software changes.

If you could isolate out those who intend to book a room at a WDW hotel from everyone else, then the analytics would be very helpful. However, millions of people visit Disney.com every day with only a segment of them ready to book a hotel at the theme park. Plus, Disney spends millions on promoting their resorts. How do you know that the increased booking rates are due to the changes to the site and not to something else? Maybe, on that particular day, the number of people who aren’t interested in booking a hotel just dropped off and artificially raised the percentages.

We have to be very careful when drawing inferences from analytic data that we’re not just resorting to hopeful thinking and seeing what we want to see. That just hurts us in the long run.

Until the analytics package can isolate streams by the intent of the user, I think they will continue to be very limited in their usefulness.

]]> By: Jeremy Kraybill http://www.uie.com/brainsparks/2006/03/08/determining-usability-from-analytics/comment-page-1/#comment-1934 Jeremy Kraybill Thu, 09 Mar 2006 04:21:25 +0000 http://www.uie.com/brainsparks/?p=198#comment-1934 While I agree that analytics tools can't read the user's mind and tell you what their desires were, they are pretty useful as a tool for validating hypotheses after releasing software. You do some in-person usability studies, you realize that users are getting confused in one of your site flows and are therefore dropping off / not buying / not completing a form, so you create a hypothesis: adding help text here, larger green button here, less text here etc are going to confuse less users and improve flow output. You validate it with post-development/pre-release usability tests, plust post-release analytics comparisons. I find that analytics tools + in-person usability tests give you a pretty good picture of what is happening. In the example you cite, usability observation tests could have uncovered the problem with site confusion, and you could have tested software improvements with analytics: less confused users would result in higher booking rates in the hotel flow, and lower dropout rates in both the hotel and theme park flows. You also could have measured rate of booking for users who jumped from the hotel flow to the theme park and watched that change with rollouts of software changes. Saying it's impossible to use analytics to assess usability is far-fetched; analytics are a piece of the picture, and should be used as such. While I agree that analytics tools can’t read the user’s mind and tell you what their desires were, they are pretty useful as a tool for validating hypotheses after releasing software. You do some in-person usability studies, you realize that users are getting confused in one of your site flows and are therefore dropping off / not buying / not completing a form, so you create a hypothesis: adding help text here, larger green button here, less text here etc are going to confuse less users and improve flow output. You validate it with post-development/pre-release usability tests, plust post-release analytics comparisons. I find that analytics tools + in-person usability tests give you a pretty good picture of what is happening.

In the example you cite, usability observation tests could have uncovered the problem with site confusion, and you could have tested software improvements with analytics: less confused users would result in higher booking rates in the hotel flow, and lower dropout rates in both the hotel and theme park flows. You also could have measured rate of booking for users who jumped from the hotel flow to the theme park and watched that change with rollouts of software changes.

Saying it’s impossible to use analytics to assess usability is far-fetched; analytics are a piece of the picture, and should be used as such.

]]> By: Hiten Shah http://www.uie.com/brainsparks/2006/03/08/determining-usability-from-analytics/comment-page-1/#comment-1922 Hiten Shah Wed, 08 Mar 2006 18:51:32 +0000 http://www.uie.com/brainsparks/?p=198#comment-1922 Jared, We at Crazy Egg are trying to make an effort to bridge the gap between analytics and usability, at least as much as we can without being right in front of users. Email me at info at crazyegg.com, I'd love to provide you with a further look into what we have going on. Hiten Jared,

We at Crazy Egg are trying to make an effort to bridge the gap between analytics and usability, at least as much as we can without being right in front of users. Email me at info at crazyegg.com, I’d love to provide you with a further look into what we have going on.

Hiten

]]>