Comments on: Do A/B Tests Focus Us On The Wrong Problems?

By: Craig Sullivan

Craig Sullivan — Sun, 27 May 2012 12:05:13 +0000

The answer is to do both – A/B tests can be driven hugely by traditional Usability Research techniques. The customers provide excellent insight and help drive hypothesis design for split testing.

Sadly some companies do just tweak with testing but those that have a mixture of UX, agile and testing approaches (in my view) do far better than those reliant on ego or opinion *driving* the testing (oh, the irony).

So A/B tests and Multi variate tests are an excellent tool to get quant data on qual problems – and if carefully managed, yields excellent results. Our approach at Belron is to use a wide range of techniques but not to let them be the master – more use them as our servant to strive towards solutions to experience problems.

By: Roger Belveal

Roger Belveal — Sat, 19 May 2012 20:06:52 +0000

As much as I am a proponent of usability testing in general, including A/B, I’d like to challenge the notion that any kind of testing alone will produce a great product. Somewhere in the mix, somebody has to have a great idea. And it probably involves a leaping hypothesis. Oroville and Wilber didn’t test their way from bicycles to airplanes.

- roger
http://www.belveal.net

By: Todd Shelton

Todd Shelton — Wed, 16 May 2012 00:44:12 +0000

Hi Jared,

Thoughtful as always, thanks. A/B testing by definition can’t see anything beyond itself, right? Are you suggesting that there is some set of methods, which if applied in the right order, will create better results? What might that look like if one could do whatever they thought was best?

Thanks,
Todd

By: Nathanael Boehm

Nathanael Boehm — Tue, 15 May 2012 22:38:55 +0000

Great article, thanks Jared. Recently getting back into private sector work, the debate over “Conversion is everything” has been a bit of a battle.

By: Alex Genov

Alex Genov — Tue, 15 May 2012 16:38:52 +0000

You are making very good points, Jared!

On the other hand, A/B (quantitative) testing and usability (qualitative) testing are two very different and, in my opinion, complementary tools. It is not about which one to use, but which one to use at what specific time of the dev lifecycle to answer what research questions.

I agree that mindless A/B testing can be misused, but then again the results of a small sample usability test can be misused as well. If the two methods are used wisely and used together, then the major of well-done research happens.

One last point – 1% lift for a high-traffic business can translate in tens of millions of dollars and should not be dismissed lightly.

By: Andrew Anderson

Andrew Anderson — Mon, 14 May 2012 21:45:50 +0000

You are assuming that there is a correlation between happiness and revenue. Or more importantly, what people complain about and what actually influences their behavior. Psychology shows that there is none.

I can’t speak for the tests you saw, and would argue that .9% lift is noise, but any time you assume what matters, or believe that qualitative measures direct correlate to quantitative outcomes, you are showing a massive bias. Being able to take opinion out, from both the tester and the user, allows us to get a much better unbiased view of influence and of truly changing performance.

By: Petar Subotic

Petar Subotic — Mon, 14 May 2012 19:46:46 +0000

Interviewer: What did you do at your previous job?
Interviewee: Conversion rate optimization, we had an entire department.
Interviewer: Thank you, we’ll call! Cough cough.