The Back Story for the $300 Million Button

Jared Spool

October 17th, 2011

By far, The $300 Million Button is the most popular article on UIE.com. Here’s the back story for how we discovered the problem and the role that analytics played:

We had been working on a client project, helping their team redesign their checkout process with some new user research and design techniques.

As we were watching seasoned shoppers buy products in the lab, we noticed that people were getting stuck at a screen right before the checkout process, where they had to authenticate their account. Repeat customers couldn’t remember their user ID and password combination. The site used an email address as the user ID, but many of our repeat customers had set up their accounts years before and couldn’t remember which of their email addresses was the ID.

In the lab, the customers who couldn’t authenticate would either give up or request a password reset. However, the password reset required they remember which email address was their ID, which many couldn’t do. We witnessed a remarkable number of abandonments on the password reset screen.

When the password reset was successful, the customer had to go to their email client, find the reset message (often lost in a spam folder), and click on a link in the email. In the lab, we observed that this was a complex process.

All of this led us to ask if this was only happening in the artificial environment of the lab, because we were watching users in our space, not using their own machine. We set out to look at the site’s analytics to see if there were clues to this behavior happening in the real world.

The first thing we asked the analytics team was what percentage of visitors to the authentication page ended up on the reset password request screen. Turns out, they had never instrumented either page. We had to wait three weeks while they instrumented it and we collected a reasonable sample size.

We learned a substantial percentage of customers were requesting password reset, approximately 40%. Two out of every five users was getting stuck and needing their password to be reset.

We then wondered what percentage of those people actually came back to finish the transaction after the reset. Again, we discovered the analytics team hadn’t instrumented the return from the reset. That was another three week delay.

We learned that fewer than 25% of the resets were executed — the user clicked on the reset link and returned to the site. Of those who did execute it, fewer than 20% finished their purchases.

A little math and we could calculate out the amount of revenue being abandoned in the carts by all the people who couldn’t authenticate. That’s where the $300,000,000/year number came from.

Once the team implemented a guest purchase capability (which didn’t require authentication to start the checkout sequence), they saw an immediate jump in sales increase of about $6,000,000 in the first week, which remained constant. Password reset requests dropped by about 80% in that first week and remained constant too.

Authentication pages are usually owned by a different group in the company. In this case, they were owned by IT. IT didn’t have the foresight to instrument these pages.

Until we did this research and asked these questions, nobody knew how many people were dropping off at authentication. Because authentication was between the shopping-cart and the Enter-your-shipping-info pages, everyone thought they were getting a much lower percentage of users clicking the Checkout button than they really were. The site had a huge abandonment on authentication that heretofore had gone undetected.

Analytics only work when we know that they are measuring everything correctly. Working with clients, we regularly discover they aren’t capturing the entire picture, leaving out critical information.

13 Responses to “The Back Story for the $300 Million Button”

  1. Ken Styles Says:

    Nice read Jared! Makes good sense, just a matter of getting the statistics to the team.

  2. Constantine Kipnis Says:

    Nice article. But where is the outrage at the notion that “authentication pages are owned by another group in the company”?

  3. Gary Perlman Says:

    It’s a wonderful success story, which reminded me of one of my favorite success stories: “Accessing Large Data Bases: The Relationship between Data Entry Time and Output Evaluation Time” by Carla Springer and James Sorce (1984) INTERACT’84, 263-267.

    This was a Bell System study for 411 service. The contract for the database system was awarded based on how long it took for results to start to appear on the operator’s screen. The “winner” gave out keychains with their slogan “Type Less, See More”. Their method, which failed to take into account how long it would take the operator to wade through the results, was to type 4 characters and view the results. Springer and Sorce showed that by typing 6 characters (about 200 msec up front), the operators were much faster at providing the information. How much faster? First let me tell you that at that time, 411 service was free and cost the Bell System $11 million per second per year, so a savings of 1 second would save $11 million. The average time savings was over 5 seconds per call, or $55 million per year.

  4. Alex Says:

    What’s the place called? I would love to shop there, just to save the trouble of making an account if I want to buy something they happen to carry.

  5. Josh Says:

    I’d love to share the methodology with a skeptical exec in my company!

    How many subjects were used in the test? Did you have a single test blending some return users and some first-timers or did you do separate tests for the 2 segments? Did you test hundreds? Tens?

  6. CameronB Says:

    @Josh:
    Josh, the retailer made an extra 300 million dollars a year. If they only tested 10 people, then sure, the sample was at high risk for being fundamentally flawed, yes. However, the results speak for themselves. 300 million is a lot of “extra” money.

    And honestly, don’t we all freaking hate websites that say “register for an account”? When you shop online, are you excited and thrilled at the chance to collect yet another account login, like some sort of bizarre internet equivalent to getting the the complete 1994 TOPS baseball card collection?

    Theoatmeal even made a joke about it. http://theoatmeal.com/comics/shopping_cart

  7. Alex Says:

    Unfortunately, this does not work for any e-retailer.

  8. Lars Says:

    Jared,
    I’m impressed at the depth of testing that enables you to see what happens when customers come back *years* after setting up an account. How did you do that in the lab? Obviously your testing cycle doesn’t last for years.
    Do you have a stable group of testers that you drawn on year after year?
    Or do you solicit testers from among the customer base from past years?
    Or what?

    Thanks,
    Lars

  9. Jared Spool Says:

    Hi Lars,

    In this instance we were recruiting people who shopped online. During the recruitment process, we would interview them in detail to learn where they shopped regularly. This helped us learn how long they’d had their account.

    We don’t keep a pool of participants, per se.

    For our studies these days, we use Usability Works for our recruiting. They do a fabulous job. — Jared

  10. Matt Morgan Says:

    Hey Jared. So here I was thinking this result was so well-documented and understood that I’d never see another registration-required checkout page, and just the other day I bought my first Zappos purchase, and what do you know–registration required. This may be old news to everyone else, but I was surprised.

    I asked them about it and it turned out interesting … basically they so stridently pursue their business model of great post-sales service (no-questions-asked, simple returns, for example, but it goes beyond that) that requiring registration is ultimately beneficial. I.e., they favor the prospect of a repeat customer over the near-term benefit of a one-time purchase. I have no doubt that they have numbers to back this up, given who they are and who their parent is.

    Of course, that could change over time … for example when we’ve all already decided that Zappos is the place to buy shoes. Just goes to show that different places get lots of different results, I guess! More good reasons to keep up the research.

    Thanks,
    Matt

  11. Visualizing an E-Commerce Customer Experience Map | Product Pad Says:

    [...] that can help in taking more customers to the next stage in the ordering process . The famous case study of USD 300 million button was in this stage of customer experience. Abandonment Tracking as well as Upsell and Cross sell if [...]

  12. Hannah Says:

    Hello Jared,

    I love that you guys asked such specific questions like “What % of visitors on authentication page ended up on password reset page?” and “What % of those actually came back to finish the transaction after reset”. Thank you for including the wait time for the analytics.

    Did you guys perform any AB testing of the solution? Or did you just do it?

    I’m working for a company that is somewhat new to using UX and completely new to using analytics. I’m trying to get a process in place for getting data to the UX department and setting up the environment for testing.

    You’re an inspiration :)

  13. la posh big brooklyn style shop online Says:

    la posh big brooklyn style shop online

    The Back Story for the $300 Million Button » UIE Brain Sparks

Add a Comment