Beware of Poorly Formed Inferences
August 30th, 2006
MarketingSherpa, a great resource for online marketing case studies, recently published a nifty article showing the thinking behind the recent redesign of CareerBuilder.com’s Job Posting home page. (Note: The MarketingSherpa article is only available for free viewing until September 4. After that, you’ll have to pay to see it.)
According to the article, Michael DeHaven, who is in charge of the site’s ecommerce marketing efforts, used A/B testing to try out two different versions of the new design.
Version “B” looked like this:
Version “C” looked like this:
What it came down to was this (from Version “B”):
versus this (from Version “C”):
The article reports how everybody on Mr. DeHaven’s staff who saw version “C” said it was too crowded and people would feel overwhelmed with links, so they expected version “B” to win the A/B Test. As we would’ve predicted based on our work with link-rich home pages, version “C” should do better.
And we were right. Version “C” won more customer accounts than “the cleaner, more graphical design”.
But what really interested me about the article was the reason that Mr. DeHaven gave for the surprise success of version “C”:
It seems that business executives prefer to look at fairly plain textual content online rather than cheerful graphical interfaces. Plus, they prefer vertical to horizontal groupings of options and longer, wordier textual click links.
“After I thought about it awhile, it made total sense,” explains DeHaven. “Users are trained to allow their eye to scan down something that looks like search results. And what we have here now is something that looks an awful lot like search results. It’s a good quick scan. Graphics and images are not what the eye’s trained for online.
“It was a HUGE finding. Really exciting.”
Wow! What an inference! Mr. DeHaven got the observation right — people bought more when presented with version “C” — but I don’t agree with his inference at all.
Jumping to poorly formed inferences can get us into trouble. Now that he believes that (a) people prefer things that look like search results and (b) users’ eyes’ are not trained for graphics or images, he’s likely to make design decisions based on those beliefs. And, while many of the resulting designs are likely to work, many probably won’t.
Part of the problem comes from exclusively using tools like A/B tests. A/B tests tell you that one version performs better than another, but doesn’t tell you why. So, the designer has to guess at why. And, in my experience, they often guess wrong.
It’s not just A/B tests that do this. Inspecting your web logs will produce the same results. You’ll see a bunch of folks are ending up on a certain page. But you can’t tell if they are there because they want to be or because they are lost. You have to guess as to why they end up there. And you’re likely to guess wrong.
Had Mr. DeHaven conducted usability tests or interviewed users, in addition to the A/B tests, he probably would’ve come to a different conclusion about why users prefer the page with the extra links. I’m guessing, based on our research, he would’ve found out the users found the pages to have more trigger words in the right places.
Whatever he would find out, it is likely he’d be more informed about the decisions he was making going forward. And, in the long run, he’d have more information to work with when making future decisions about his designs.
August 31st, 2006 at 1:32 am
I agree with your comment that, if he’d done usability tests he’d have had more information to work with but I’m confused about how you picked ‘C’ as the winner based on the number of links. As far as I can see from the two images, there are exactly the same number of links in each case. The only differences are the wordiness of the links and the horizontal vs vertical layouts. There doesn’t seem to be any reason to conclude – on a count of links alone – that one would be better than another.
Or did I miss something?
September 1st, 2006 at 10:03 am
When you stack links on each other like that your eye can look in one spot and see parts of all three links without moving their eye. When you put them on a single line then you have to read the whole line with your eye before you get an idea of what’s there.
Stacking text works much better visually because text lines are so thin. That’s the obvious reason why “C” prevails for me — you can read it faster and they links look more associated with each other.
September 1st, 2006 at 11:33 am
I agree with Graham, in that there are not extra links either. So not sure what Jared is referring to in this statement, “he probably would’ve come to a different conclusion about why users prefer the page with the extra links.”
Perhaps the wordiness of the links gives the illusion that there are more?
Regardless, I do agree that A/B tests are not as effective as when combined with user testing because you are already forcing the user to choose the best of 2 scenarios, where as maybe either aren’t great at all. Nothing beats talking to the users first, creating a layout based on user needs, then watching the user perform a function/task with the presented, and getting their direct comments/feedbacks, and revising based on this feedback.
September 3rd, 2006 at 8:43 pm
The visual design has little to do with it IMHO. Sure version ‘C’ is denser which may aid in scanning, but I’d wager it is actually the link text that makes it more successful. As an added bonus ‘C’ is more accessible due because it less ambiguous than version ‘B’.
September 20th, 2006 at 5:04 pm
Jared — Glad you liked the Case Study. However, your blog is a tad confused. The Case Study actually talked about two sets of design tests. The first was a graphical horizontal design (you do not show in your blog) versus “version B” above. In that test version B (the textual vertical column) won. And that’s why the CareerBuilder folks were talking about in the quote you mention in results.
However, next they conducted a SECOND set of A/B tests — in this case between the two versions you show in blog. In that case the longer verbiage won for text links. And in that case it won because people’s eyes are skipping around so you need to be ultra clear in your click link…
So, the CareerBuilder guys weren’t quite as clueless as you make out — the quotre you quoted was their conclusions from a test you did not picture.
September 21st, 2006 at 7:45 am
Hi Anne,
Sorry I reported it wrong. I thought I’d read the article correctly. Didn’t realize there were two studies.
However, it still doesn’t explain the wacky inference about users being trained by search results to skip images. That’s more what I was commenting about.
There’s no evidence in any study we’ve run that suggests this inference is remotely true.