Recently, I posted a tweet about conversion rates.

Conversion rate’s big crime is it focuses purely on pressing the purchase button, independent of the quality of the experience.

— Jared M. Spool (@jmspool) May 13, 2012

Immediately, I received several responses, all of them showing the common misunderstandings that people have about conversion rates.

The conversion rate has become a standard metric for many online businesses. I’ve seen clients watch it every day, looking at the variations they see and asking questions about why it changes (or doesn’t).

Unfortunately, conversion rate is probably one of the most dangerous and mis-used metrics available to site designers to measure how good their site is. They apply it as if it is telling them something and, worse, when they optimize for it, they can create a site that looks like crap and produces undesirable customers.

This is the first post in a series on using conversion rates, hopefully to clear up the common misunderstandings and help designers find better ways to measure their site’s success. In this episode, we’ll look at the basic nature of what a conversion rate is and why it’s such a crappy metric.

Avoid Ratios For Metrics

Conversion rate is a ratio. It’s the number of people who purchase (or sign up or whatever your “call to action” button does) divided by the number of visitors to the site or page. If you have a 15,000 people buy products out of the 1,000,000 people who visited your site, you’ll have a conversion rate of 1.5%.

Ratios are a problem because either the numerator (people who buy) or the denominator (people who visit) can change. Let’s say you still have 15,000 purchasers, but because of a “successful” marketing push, you end up bringing 2,000,000 visitors to the site. Now, you made the same amount of money from those 15,000 purchasers. But your conversion rate dropped to 0.75%.

Because it’s a ratio, you don’t have control over which side changes. The theory is that it should be uniform, but as we’ll see in other posts in this series, that rarely happens.

This means conversion rates often fluctuate for no discernible reason. It can be 1.5% one day, 0.75% the next, and 3.0% the next. This would just be a normal variation based on what’s happening with the visitors being attracted to the site. And this normal variation makes it hard to tell when something is broken or out of whack.

For any e-commerce site, I have the perfect advice on how to raise their conversion rate significantly. All they have to do is stop marketing. Once they stop marketing, the number of visitors will drop to only those who are already loyal customers.

Because those visitors are loyal, they are probably only coming to buy something. The ratio of purchasers to visitors will skyrocket. Sales will likely drop, but conversion will go sky-high.

Sounds great, right? That’s the other problem with the conversion rate ratio: it’s not at all related to the other business operations.

I have a little game I like to play with executives. I’ll put to doors on the screen. I’ll tell the execs that they can choose one door. Behind the first door is an increased conversion rate, but no increase in sales (in fact, a likely decrease). Behind the second door is a decreased conversion rate, but an increase in sales. Which would they like?

Every executive I’ve ever talked to has asked for the increased sales. My next question is then: Why focus on conversion rate then? That’s when I get that deer-in-headlights look, followed by a reduced focus on the conversion rate metric.

In the next installment, we’ll look at why not every visitor is someone you want as a customer.

I’ve been fascinated by critique lately. It’s a fabulous tool to help the entire team – designers and non-designers alike – learn more about what makes great design great.

I’ve learned that you can tell that a team is taking advantage of well-done critiques by the new, personalized language they are now sporting. They have a set of terms and phrases that only have meaning to them, but the meaning is deep and thoughtful.

I saw an example of this from Kate Brigham, as she talked about the work her team at PatientsLikeMe was doing. They had a term they used to describe the buttons on their pages: lickable. Lickable buttons are buttons that look like bright colored candies and are so vividly yummy, you just want to lick them.

They started employing these lickable buttons to get past the dull, mechanical look of the operating-system-supplied checkboxes and radio buttons. Lickable buttons are fun. They encourage pressing. They don’t look formidable, which is important when PatientsLikeMe’s designs are asking their users about serious things, like how much pain or fatigue they’re feeling due to their chronic illness.

What’s neat about this button nomenclature is that it’s 100% PatientsLikeMe. I’m not 100% sure of the origin story, but it sounded to me like it emerged during a critique session when the brightly colored buttons first made their appearance in a design. Someone made the comment about how they looked so delicious and the name just stuck.

This is often how these things work. Someone just says it and it catches on immediately. That’s the first stage.

That’s cool, but what happens next really excites me. The PatientsLikeMe team decided that being lickable is a good goal for future designs. Whatever that means.

Now, the team has the duty of exploring the true boundaries of the term. What does it really mean to be ideally lickable? When are buttons not quite lickable enough? When have they gone past the ideal lickability into something too undesirable?

It’s in these conversations where the team gets a chance to explore what design is really about. Well-run critique sessions let a designer bring proposed designs to the table, and the team discusses how close to an ideal notion of their design the can get.

This testing of the boundaries cements the language and gives everyone a way to talk about design in a positive, ever-learning fashion. The team grows as a result and the designs get better.

Critique is hard. Well-run critique is really hard. But the payoff is wonderful, because the team moves beyond the specifics of the designs into learning what makes a great design. Once that happens, you can see the output of the team jump to a new level.

That’s really exciting.

Storytelling is a natural form of expression. We’ve all been telling stories from a very young age. In the design process, personas become the tool we use to tell our users’ stories. And with good personas in place, usage scenarios can become the micro-stories that drive your design decisions.

Kim Goodwin tells us that scenarios put the design into the context of how and why the user will interact with it. In 2011 Kim presented a UIE Virtual Seminar, Designing with Scenarios: Putting Your Personas to Work.

Today’s UIEtips article is based on a discussion UIE’s Adam Churchill had with Kim. It’s based on two questions from the seminar: Do you need data to effectively do scenarios, and what’s the difference between scenarios and storyboarding?

Read the article Designing with Scenarios: Putting Personas to Work.

Since personas are a tool and not your goal, there are lots of useful things you can do with them. In our May 22 Next Step Virtual Seminar, The Characteristics of Effective Personas, Whitney Quesenbery will show you more clever ways to make your personas actionable and weave them into your work.

What’s your experience with scenarios? Are you using data when developing them? Share your thoughts with us below.

Last week, I attended a conference presentation where a team presented findings from their A/B Testing efforts. It was a cute presentation where they posted the control and test variants, then asked the audience to pick which one “won” the A/B test. They compared the audience answer to the variant that demonstrated the best increase in the conversion rate (sometimes as little as 0.9%, which the presenters declared as a “huge increase”). For dramatic effect, the variant that won often broke many commonly accepted design principles, supporting their case that A/B testing trumps our traditional rules of good design.

Maybe so. Yet, that wasn’t the part that interested me the most. What interested me was something completely different.

A few months ago, I had conducted usability tests on the online service produced by the presenters’ company’s — the very one they showed in their presentation. Our study looked at several sites, asking shoppers to buy the products and services offered, looking for what each site does well and where their weaknesses were. It was really insightful on many different levels.

In our study, we watched more than a dozen of the presenters’ company’s own customers attempt to buy products. While many were successful, a surprising number weren’t, even though this company is the biggest in its industry (and hailed by many as the most successful). Their site looks slick, but when folks sat down to use it for its primary goal, it’s design put up a ton of frustrating obstacles.

In many cases, the users thought they ordered the product they wanted, only to discover upon receipt that it wasn’t at all what they wanted. As we watched those shoppers make their orders, we could see that they would not get what they wanted. Yet, the design of the site was so convoluted and confusing that the shopper never detected the problem until it was too late.

That’s why, when I was sitting in the conference presentation, I was quite surprised by what the presenters showed. All the examples were not things that made the shopping experience difficult. They were not things that were giving their customers problems. They were little things that, if dramatically improved, wouldn’t, in my opinion, affect the overall experience of the site or the satisfaction of customers buying their products.

I get that the presenters probably didn’t want to reveal their secret sauce to an audience of a hundred or so user experience professionals who might (like me) be working for their competitors. Maybe they were also working on the problems I saw in our study, but didn’t want to talk about that publicly.

However, the A/B tests they presented showed they were applying a ton of effort to optimize things that weren’t close to the things we saw preventing sales on their site. If the message was that A/B testing helps, I didn’t get that because I saw them futzing around with tweaking insignificant button text when there were huge deficiencies in the design that they still haven’t resolved.

Maybe the presenter’s team is working on these things that are wrecking their shopping experience? Maybe they aren’t using A/B tests for these hard problems? I don’t know.

Yet I’ve seen this before: A/B tests are fun and they easily become part of the dog-and-pony show. They give numbers and a clear, easy-to-see winner. They get execs and stakeholders excited, because “improvement” is easy to spot. But are we doing damage to our mission when we give so much attention to these tests of trivial design changes?

Part of this might be because the presenters were determining their winners by using the wrong measure. Conversion rate has lots of problems as a measure of success, but its big crime is that it focuses purely on the pressing of the purchase button. It doesn’t measure whether the users are happy with that purchase or whether they are delighted with the product they finally received and the way they received it. It’s easy to optimize for conversion while sacrificing a great experience. Conversion ≠ Delight.

We should be careful about how we show off tools like A/B tests. Someone might think we don’t actually care about the users’ experiences, as we optimize for overly simplistic measurements.

Podcast: Play in new window | Download (Duration: 25:58 — 0.0KB)

[ Transcript Coming Soon! ]

A data visualization, when done well, can be an incredibly powerful way to communicate information. It ultimately boils down to the choices you make in how to design and present the data. If you make the wrong choice you can run the risk of not accurately displaying the data or struggling to effectively tell its story.

Brian Suda, author of A Practical Guide to Designing with Data, believes experimentation is a big part of arriving at the right choices. As ideas end up on the cutting room floor, not only do you arrive at a great visualization, but you’re building your toolbox along the way. This practice and experimentation leaves you with a template to apply to future projects.

Essentially, arriving at the right choices now allows you to make better choices later. If you learn the best ways to represent different types of data, you can then apply that knowledge to any data sets you may have to visualize.

Brian will be sharing his insights on data visualizations in his virtual seminar, The Design Choices You Make for Information: How to Create Great Data Visualizations, on Thursday, May 17. You won’t want to miss out on Brian’s pragmatic tips and techniques. Save your spot in Brian’s seminar.

As always, we love to hear what you’re thinking. Share your thoughts with us in our comments section.

Recorded: May, 2012
[ Subscribe to our podcast via ←This link will launch the iTunes application.]
[ Subscribe with other podcast applications.]

Read the rest of this entry »

Personas are one of the most controversial tools in the professional UX toolbox. People either swear by them or swear at them. When they work, they are awesome, but when they fail, well, they fail gloriously.

For the past few years, we’ve been researching why so many persona projects have such dismal results. We discovered there are basic factors that are critical for a project’s success, yet most teams ignored them.

In today’s UIEtips, I look back at an article that discusses five of these factors. We look at the role of research, who should be involved in the personas, and other essentials that differentiates between a successful persona project and a failed one. I’m sure you’ll find it helpful for planning your projects.

Read the article: Five Factors for Successful Persona Projects

If you’re looking to get more out of personas, you’ll want to tune in to Whitney Quesenbery’s May 22 Next Step virtual seminar, 7 things You Can Do with Your Personas. Whitney will tell you the 7 characteristics of effective personas and describe how to use them for improving user experiences. Learn more about this virtual seminar.

Have you tried to use personas in your projects? What have you found to be the keys to your project’s success (or the reason for your project’s demise)? We’d love to hear your thoughts below.

For many projects, self design works great. By designing for our own use, we can optimize the user’s experience to be smooth and seamless.

A while back, I wrote about the advantages of self design and the alternatives to self design. Of course, to be successful at self design, you have to use your design practically every day and you have to have a base of people just like you that’s big enough to support whatever business model pays for your work. That way, when something in annoying in the design, you’ll discover it quickly. And when you fix it for yourself, you’re also fixing it for a large enough user base.

A big problem with self design is that it doesn’t deal with things that you won’t do frequently with the design. One of those things is using the design for the first time, because, well, you can only do that once.

You see this in a lot of self-designed products: the initial user experience is rough, never explaining why you’d use any of the key functions, and hardly ever giving the user a great way to get started. Because self-designers rarely are working from a blank slate, the initial process of populating the application with data is not as smooth as the usage patterns once you get going. The entire getting-started process is really difficult.

While self design can help work out the rough edges of daily use effectively, design teams need to step away from the self-design approach to something more activity-focused for that out-of-box experience. Creating a process that usability tests with people new to the design will help keep things balanced.

With frequent and regular usability testing, you can answer questions like “Do people understand what this is for?” “Do they see how it can help them?” “Do they know how to get started?” “Are they overwhelmed by the functionality or choices?”

It’s ok to use self design to move a project along. It’s best to combine it with other types of decision styles, such as activity-focused design. This will keep the project balanced.

Small is good. We love small products. Why not small processes?

Mobile phones used to be big and bulky. Then we found ways to make them smaller and pack more stuff into them. Now we walk around with multi-purpose computers in our pockets. And guess what? We use them more than ever for things we never imagined we could do.

What if we did that same thing for our UX processes? What if we could “smallify” important artifacts to make them easier to carry around? Would we use them more for things we can’t imagine?

In this week’s UIEtips, I talk about the Short-Form Creative Brief – a compact design artifact we found in use by a team we’ve been working with. This tiny little document, which is small enough to carry around and pull out when needed, has changed how the team creates designs. It can change your life too.

Read the article, The Magical Short-Form Creative Brief

What do you do to keep your team on the same page during your design meetings? What design artifacts have you made smaller and easier to use? We’d love to learn from your experiences. Leave a comment below.

The other day, I wrote about how choosing the right words in your tasks makes a critical difference to the outcome of your user research. Mike Pauley wrote a comment, asking how to make sure you’ve got the right words:

Great timing on this, as I am dealing with the same issue with a test I’m currently running. Do you have any guidelines for how to approach question wording? Other than running the same test multiple times with different wording, how do you know when task failure is the fault of your IA or the way the question is worded?

A great technique is to adopt an interview-based task design approach. You start by interviewing your participants to ask them about their previous experiences and current needs.

For example, if we were testing the IKEA site for it’s navigation, we’d recruit people who either have purchased IKEA furniture or are likely to do so in the near future. Then, during the first 15 to 25 minutes of the session, we’d interview them about their furniture buying process and desires.

In that interview, we’d get them to use all their own terms and define their own tasks. If it’s something they’ve done in the past, we ask them to re-enact it for us. If it’s something they’re planning in the near future, we look for them to show us how they think they’ll do it.

The trick is that we let them tell us the words they use. After you’ve done this with a few users, it’s likely you’ll hone in on some generic ways to formulate the tasks that don’t influence the tasks’ outcome as directly.

Back in 2006, I wrote about interview-based tasks and recorded a podcast about how to do them. Interview-based tasks have become a very important part of our usability toolbox, specifically to deal with this problem.

Years ago, we were watching people try to find products on IKEA.com. (Not for IKEA, but for our other nefarious purposes.) We took several stabs at our study because, well, the first ones seemed fishy to us.

Our initial stab had simple tasks. One of them was “Find a bookcase for your living room that can hold 200 books.” Seems straightforward and nearly everyone completed the task easily. In fact, it seemed too easy.

It turned out that after reading the task instructions (we often write them on paper instead of delivering them orally, so the participant can refer back), every participant did exactly the same thing. They clicked in the search box on the IKEA home page and typed Bookcases. From here, it was pretty easy to get to the right product.

Yet, it was because everyone did the exact same behavior that made the UX hairs on the back of our neck tingle. That didn’t seem right.

In a second round of studies, we changed the task description to something more context-rich, avoiding any clues as to what to type in to the search engine. We ended up with:

“You’ve just moved into a new place with a living room that you love because of its expansive walls. Now, you just need someplace to put those 200 prized books that you’ve had in boxes forever. How would you do that?”

As we suspected, our participants’ behaviors changed. A couple folks typed bookcase into the search box. Others typed shelves, book shelves, book shelf, and one shelves for books. And others didn’t type anything into the search box, but clicked on the navigation links on the page, including storage systems. Most (but not all) found success using their own strategies.

What was happening was that we had guided the users with the wording of our task. By changing the wording, we saw different usability results. The design was the same, yet the results changed because of what we asked the users to do.

The wording we choose for our tasks is a critical component of the research we do, yet we hardly ever see any discussion of how to do it right or what happens when we get it wrong. If we want to make sure we’re guiding our teams in the right direction, we definitely need to nail the right research at the start.