Measuring Money Left On The Table – Moving Beyond Conversion Rates, Part 5

Jared Spool

July 10th, 2012

Moving Beyond Conversion Rates:

Part 1: Avoid Ratios for Metrics
Part 2: Not All Visitors Make Great Customers
Part 3: Visitors Are Not All The Same
Part 4: Campaigns Are Where Conversion Rates Shine
Part 5: Measuring Money Left On The Table (this)


Thanks to Marco Dini for translating this post to Italian.

As you can tell, I’m not a big fan of using conversion rates as a critical key measure of how well a design works. In my opinion, the metric is unreliable and doesn’t really give you an actionable picture on what you can do differently. That, of course, leaves the obvious question of what you can use.

One technique we’ve had great success with is something we call a Compelled Shopping Analysis. It’s designed specifically for e-commerce sites (however, there are easy adaptations to use for other types of conversion, like lead acquisition).

The Compelled Shopping Analysis comes from a thought experiment we had created a few years back, which we call The 7-Eleven Milk Experiment. The thought experiment works like this:

Imagine you have a magic app that goes off whenever someone within a 10-minute radius of where you are right now has run out of milk. You hop in your car, drive to their place, and find them sitting there with an dry bowl of cereal and an empty container of milk. You put them in your car and drive them to the nearest 7-Eleven convenience store. And just to make sure, you give them enough cash to purchase the milk.

What do you think the odds are that the customer will buy milk at that moment? Pretty good, right? The 7-Eleven would really have to screw up to not sell milk to that customer at that moment.

That’s exactly how a Compelled Shopping Analysis works. Through careful recruiting, we find people who need to buy products that are on the sites we’re studying. We bring them to those sites and we give them the cash to buy those products. In theory, we should see them make the purchase.

If they don’t make the purchase, we call that money left on the table. It turns out that most sites leave money on the table. The customer can’t find the product (even though we’ve already ensured it was on the site), or they can’t determine if they product they find is in fact what they want (often because the description sucks). Or they can’t figure out how to purchase the product, or they can’t get through the checkout process without a show-stopping issue.

In each case, we know how much money the customer wanted to spend and what they actually purchased. (We also look at whether they spend more money on the site, which tells us a lot about how well the site itself is designed to increase engagement.)

The beauty is this gets us past the objections we have with the conversion rate metric. Because we’ve are controlling the number of participants in the study, we’re not looking at a ratio with an uncontrolled denominator. Since we’re selecting customers who we know need the products and are ready to purchase, we know these are perfect customers for us and they are ready to make a purchase.


The money left on the table metric gives us a solid improvement tracking. Even better, the method points out exactly where in the shopping process (and on which pages) the money was left. We can see how we’re doing as we fix each problem we identify.

The Compelled Shopping Analysis does come with its downsides. First, it takes more time, because we have to conduct a series of usability studies to get the data. The analysis, particularly the diagnostic part, is also a bit time consuming. For a typical e-commerce site, we can get this analysis down to 4-5 weeks per round, but that’s still a long time.

Second, it’s not automated. We have to sit and watch real people do real stuff. It’s not something we get every day out of an analytics report.

Third, it’s significantly more expensive. Buying CDs or books is cheap, but giving folks the money to buy laptops or vacations (we’ve done both) can be very costly. You have to really want the data to make this worth the while.

Conversion rates are appealing because they are fast and cheap. But the quality of the results aren’t very good. Measuring money left on the table is high quality data, but you pay more.

A compelled shopping analysis isn’t the only alternative to conversion rates. But all the other alternatives we know are just as time consuming and expensive. They yield substantially better data and are more actionable than measuring conversion rate, but for a price. That means you have to really WANT to make improvement, instead of just having a data point that really doesn’t tell you much, but makes you look like you’re making improvement (even though you aren’t).


Moving Beyond Conversion Rates:

Part 1: Avoid Ratios for Metrics
Part 2: Not All Visitors Make Great Customers
Part 3: Visitors Are Not All The Same
Part 4: Campaigns Are Where Conversion Rates Shine
Part 5: Measuring Money Left On The Table (this)

One Response to “Measuring Money Left On The Table – Moving Beyond Conversion Rates, Part 5”

  1. Luca Says:

    Hi Jared, I am the web marketing manager on an italian bank. We offer an online savings account: people simply leave their money in our account and they have an high return from their savings.
    I was wondering this is a kind of situation in which your Compelled Shopping framework is very near to the conversion rates model: we give money for free (it’s a dramatic oversemplification) and *every* problem in the registration process for the account should be addressed only to usability issues.

Add a Comment