Jared Spool: Welcome, everyone, to another episode of the SpoolCast. I have with me on the line today, Ken Kellogg. I’m very excited about this. He is the Director of User Research at Marriott International, which, if you’ve never been to Marriott.com, and he has been doing a tremendous amount of research on their most recent redesign. You may not know this about Marriott.com, but they are the seventh largest e-commerce website in the world. They do $6.5 billion every year in transactions, at least in 2009; 12 million visitors a month. Did I get all that right, Ken?
Ken Kellogg: I believe so.
Jared: Yes, it’s just really awesome. You just launched the redesign which I am betting, you, guys, said on, you know, it was like a Monday afternoon, you said, “Hey, why don’t we redesign the site?” Then, on Wednesday, you had the new thing up. It was that easy, wasn’t it?
Ken: No. Actually, we did it that Tuesday, like two days. [laughs] Yes, Jared, it was a very long term project, about a year or so. It was one of these things where we had a very good website, and doing all that transactions that it needs to.
People that used it, our rewards members and folks like yourself, they spend an awful lot of nights in our hotel. They really liked it because they knew it. They knew where to go, they knew when to go, they knew how to do things. They could get in and out of the website really, really quickly. They knew how to find their points. They knew how to reserve the room at the Boston Waterfront Renaissance really, really quickly.
But, one of the things that we were finding was, not only was our website starting to look dated, and if you’ll look at the old one, you could see that it’s pretty much somewhere around a 2000 design. It was looking a bit traditional, shall we say. Our research kept telling us that we might be missing out on a key segment, and those are folks that are a personal traveler, a leisure traveler. Of course, business traveler can be a leisure traveler anytime. But, we’re really thinking about folks that don’t normally travel for business, are a spouse of a frequent business traveler who wants to plan a vacation.
What we were seeing over and over, Jared, was these folks would come to Marriott.com and they get lost. One of the old rules about usability is you never get two chances to make the first impression. If someone gets lost, they will go some place else, that is still the truth. Everything kind of started off of those two facts.
Jared: Now, how were you discovering that these folks were getting lost?
Ken: Lots of research. Really from two different perspectives at the time, the team was doing a lot of qualitative research. So, we had just, literally, years of data from our lab. We also had our measurements and analysis group that were able to tell the behavioral story with real numbers. They were able to tell where people went and who these people were when they went to different places.
Jared: So, you were watching a lot of users go through the reservation process.
Ken: Oh, yes, yes. Absolutely.
Jared: When you were selecting people to go through the process, were you focusing on these different types of travelers and explicitly saying, “OK, now we’re going to look at spouses of people who are frequent travelers”?
Ken: I would not say that we got quite to that level of segmentation as far as the e-commerce team is or was concerned. It was just more a bunch of data that just kept boiling up. Now, given the economy, even though the economy is improving, the work that we did for those leisure travelers is kind of paving the way with where the company needs to go.
Jared: Now, Marriott is not a regional business, they’re a global business, in fact.
Ken: That’s correct.
Jared: The new home page that you guys put up, came up with a few months ago, that’s worldwide, right?
Ken: It is. It is. We have a Marriott.com in German. We have one in China. We have one in Spanish, and we’re soon to have one in France. Part of the research plan, actually, called for us doing global research. It was a lot of fun and it was good to—on a lot of different levels for the organization—to get those real facts and real data sets from folks in China, from folks in Germany. Soon, we’re going to do some work in Latin America, too.
Jared: That’s very cool. Now, in the process of doing the research and coming up with the design, were there any sort of really interesting things that came out that you didn’t really expect? You had to say, “Oh, we’ve to step back and rethink this for a moment.”
Ken: There were a couple of things that ended up being a little bit outside of the norm. I guess, both of them for the way Marriott’s product development does things. I guess, the first one, if you step back, and I’ll be able to touch on this when we do the talks.
When we developed our research life cycle, we wanted to do a good job of blending qualitative research and quantitative research. I firmly believe and the team believes that just doing all qualitative research now, that’s table stakes, and you’ve got to be getting beyond that.
So, the infrastructure of the Research Team, and the User Experience Team for that matter, is built to consume both types of data. What we would really want to do in a perfect world is to use qualitative research to hone down on what a concept might look like, what the concept might contain—you’ll have several concepts, of course—and narrow everything down to two or three, and then do some quant. research to see what’s moving the needle with some projectable numbers.
But, we couldn’t do that, we’ve got a very, very competitive environment. Something that shows up on our website one day, typically shows up at our competitors’ three or four weeks down the road.
So, we wanted to do that but we were really kind of unable to because we didn’t want the cat out of the bag until right before it was time to let the cat out. So, we ended up doing our quant. research very, very, very late in the game. While we expected the feedback and the numbers from the quant. research to be good, we were surprised at how good the numbers were. At that point in time, we all started kind of sleeping a little bit better at night.
Jared: When you say the numbers were good, what sort of things do you guys look for, coming back from these things? Which numbers can you share with us, generally? I know a lot of folks get a lot of data, they get a lot of analytics data, they got a lot of other stuff, but they’re not really sure how to pull out the numbers that are important to them. How did you guys figure out what numbers were important to you?
Ken: You always look at preference. You’d look at likelihood to recommend. You look at head-to-head comparison on feature sets for the concepts. You look at your variables, and then you look at your control, which is your current environment. What we were seeing was statistically significant movement of the needle across all of these axes that we were looking for.
We also had come up with terms, with descriptors, with attributes that we wanted people to say about the designs, about the concepts.
Jared: While you were doing that, I was going to ask you, were all these different questions that you were asking post-experience survey questions, or were you collecting behavioral questions too?
Ken: No, it was post-experience survey. As part of this particular piece of research, we had people go to an externally hosted…[audio problem] almost as a quick usability session. Very, very scripted. We got some overall ratings from them, and then we kind of deconstructed what they did, their actions, as well as elements on the page.
Ken: And then, early on, as we started coming up with our new designs, we came up with a list of attributes that we wanted to see or have associated with our new designs, and we started hearing those words qualitatively very early on in the research cycle.
Then, when we started getting those same words quantitatively—first in an unaided manner, and then in an aided manner, when we asked them to find the words that describe your experience, and so on, and so forth—at that point in time, it became longitudinal, right? Those words that we heard, the very first concept we evaluated in the lab, when it was one of eight, to the final concept, when it’s being looked at against the old Marriott.com website—when those terms are coming through eight or nine different rounds of research that you’ve done all over the world, that’s a good thing. That helps you sleep at night.
Jared: Now, one of the thoughts I had about this was that Marriott.com—they had a presence from the very early days on the Internet. They’ve always been forward-thinking in terms of being there, but your design, as you said, had the sort of circa 2001 look. It didn’t really change. As a frequent traveler myself, last year, I spent more than 80 nights in Marriott hotels, and I can book a reservation for a Marriott stay in about 45 seconds.
That comfort of knowing exactly where to click and what’s going to be asked of me and how to get through it—as a frequent business traveler, that is really quite important. But, as you were mentioning, the people who were planning a vacation—planning a vacation is a very different process than just “I’m going to Moline, Illinois for the third time this year, and I need to stay at the Courtyard, and I just want to get my room and then get out of there, because I have a million other things to do.”
Ken: Completely different mental approach to the situation.
Jared: Exactly, so how do you balance those really dramatic differences in approaching it? Did you consider one for the leisure traveler, one for the frequent one, or did you say, “No, we’ve got to find something that works for both”?
Ken: You know, Jared, that’s an interesting question. Early on—and it was just more of a thinking exercise; in no way, shape, or form would we do it, but we just thought about—do you remember the old AOL welcome screen?
Jared: Oh, yeah.
Ken: Circa maybe 1997, where you simply had the content channels? News, sports, and they were just blocks, right? We thought about something like that, just to stimulate our thought process—business travel, leisure travel, and having two different approaches and two different paths. But, for a lot of reasons, we didn’t go that direction.
That being said, one of the first things that you’re indoctrinated with when you start working in e-commerce at Marriott is that you do no harm. It’s the Hippocratic Oath. You do not hurt the frequent business traveler. So, everything that we have done, we have painstakingly done our best to make sure that we don’t put any stumbling blocks or any delays in the paths of our frequent business traveler.
Our thinking is this, when you come to the new Marriott.com, you should be able to reserve that room without extraneous clicks; much the way you are able to do with the old one.
Now, there will be some other options as we go forward, when we launch our reservations in February. We don’t want those options, in case you are a leisure traveler, to get in the way of booking a room because that is what the frequent business traveler wants to do.
And we have—I’m sorry, go ahead, Jared.
Jared: Well, I was going to ask—so you have this “do no harm” to your frequent business traveler, but you have this big problem, or at least it was clear that you needed to work on your leisure travelers.
So, what did you provide? How did you find this middle ground? Were there lots of different design options that you were trying out? Did you pretty much use the research to hone into a concept and make it work? How did you get to what you’ve just launched?
Ken: First up, it’s important to remember that the home page is just one piece of really a three-part plan of the new Marriott.com. The next one, as we talked about, will be the reservation process. That really takes off from the search results page. That’s really where most people will see a big difference in what we’ve done. The third part of the new Marriott.com is the actual property pages; the hotel websites.
These just simply become more and more important, and have, to a certain degree, displaced the home page in importance. Now people don’t go top-down like they did—like our users did in the early 2000s. Now any page within your website is a destination page; is an entry page.
You can Google anything and go straight to that hotel website and miss the home page, miss the search results completely.
When the project first started, we gathered all of our existing research. From that existing research, with the aid of a vendor, at that point in time, we emerged with six initial concepts. We did research against those concepts and we had two distinct segments in the research, themselves, as far as the participants were concerned. We had frequent business travelers and we had leisure travelers.
We actually used our participants, each one being different segments, to evaluate the concepts. At that point in time, a couple of different concepts started to emerge from the cloud. Then, at that point in time, we used the research to start honing down on those concepts.
Earlier on, Jared, we were talking about a couple of things that were challenges or we might have done differently. At that point in time when we had these two concepts, we did something that we had never really done here at Marriott, anyway. I know other companies do it, but for us, it was a big step.
What we did is, we brought in our standard demographic set of participants, business travelers, platinum folks, gold folks—like yourself—and then a few leisure travelers. We also did something else. We brought in 22, 23, and 24 year old recent college graduates that were just starting to travel.
Our thinking there is “if we are re-inventing the Marriott online presence, we better have that young set of travelers in mind when we do it.” That’s really—that test right there helped us get to what you see now when you go to Marriott.com.
Jared: That’s interesting because those younger folks don’t come into the travel space with the same sort of expectations and understanding as to how it works as your more seasoned travelers—even leisure travelers, who have planned their family vacation every year. They have all this experience, at least with what they want and to what works and what doesn’t.
They come with all these expectations, but those younger folks, whose previous travel experience might be hopping from couch to couch, in some cases. That’s a very different group to be aiming after.
Ken: We have to be relevant with those emerging generations. We must be. We’ve got a portfolio of over 3000 hotels; all different brands. Some of our brands are certainly going to be more suited toward those younger folks, some of the brands are going to be more suited for the more established traveler.
We’ve got to let that emerging set of business travelers—that 22, 23 year old IBM or Accenture consultant—we’ve got to let them know that we’ve got something for them; somewhere, somehow, in our portfolio of brands.
Jared: Now, would you say research is this going off and doing quantitative survey stuff and doing qualitative usability stuff that this is, that all of that is very much in tune with how Marriott does business? Or did you have to fight for some of this to be budgeted and paid attention to and to become part of this project?
Ken: It was—particularly the quantitative piece—was not something that eCommerce was doing.
Jared: Really? Often, I see it the other way around, right? That people are doing quantitative because they are used to at least having easy access to tools like BizRate or ForeSee. They’ve got that coming in, but qualitative is something they always have to fight for. So, you found it the other way around?
Ken: It was the other way around. My executives, when I’ve asked for something, and back it up with a good business case, they’ve given us the tools that we need to get our stuff done.
Now, while we didn’t do it, the executives certainly had the foresight to see that it would really help us to go where we needed to go. It’s one of those things, you’ve got to have complete infrastructure, that complete toolset because you can’t just “ready, fire, aim,” anymore, particularly here at Marriot’s e-commerce team. You must know what you’re doing before you do it and the only way to get to that point is to have the research infrastructure in place.
Jared: Now, that’s interesting, too, to me because the “ready, fire, aim” approach is a very entrepreneurial approach.
Ken: Absolutely, and there’s nothing wrong with it given the correct business scenario.
Jared: Right. So, a lot of folks who come in to design, who come in, being it start-ups or another entrepreneurial environment, they get frustrated by having to sit down and move away from that approach.
But, in larger enterprises and multinational companies and places where the website is bringing in billions of dollars of revenue, things have to be better understood than just “Hey, let’s turn everything purple today and see if it does anything.”
Ken: You’re absolutely right. For us, if we would make that type of mistake, a lot of people lose their livelihoods. We’re a big driver of the Marriot bottom line. And I’m not just talking about e-commerce people that made the mistake, but people in hotels—the housekeeping staff, the doormen, and things like that. If we make a mistake, other people pay for it.
Jared: You’ve got to have your research in line and to play with it. You mentioned that they had a good qualitative program in place. Was that true from when you got there, or did you bring that in?
Ken: Oh, no. The program was here. A lot of people on the research team had left for various reasons, to go explore other opportunities and so on and so forth. But, the value, by my executives, on research was always there.
So, I stepped into a bit of a void and the team had that void to fill. But, the point being, Jared, the need and the hunger for research was there when I got here.
Jared: That’s interesting. So, if you were talking to folks who are in organizations where the research isn’t as understood by their executives, what advice might you give them to raise awareness, both qualitative and quantitative, to get that into their design process?
Ken: Sure, my advice is going to be start one place at one time. Figure out what you do best, what you do well, and figure out if that can make an impact in the way you’re building products today. Then you put your heart and soul into that.
When you are successful—and you will be—when you are successful, then it’s much easier to evangelize the merits of research in the design process. And evangelizing is part of it, particularly if you are establishing a research discipline.
Jared: Do you find now that you still have to evangelize at Marriot?
Ken: Sometimes, sure. Nothing is ever a given. Economic times are what they are. Thank goodness it looks like we’re starting to come out of some really, really bad times, so development money is always at a premium and research is always one of the things that can slide by.
You got to stand your ground and if you think that a given project can benefit from research, you got to push for it. You got to push for it.
Jared: The travel industry was particularly hit hard in this downturn, right?
Ken: Wickedly hard.
Jared: People stopped traveling—businesses—and it didn’t help that companies were coming under scrutiny for executives taking their private jets and the big parties.
I know that companies that had real business meetings were canceling them just because they didn’t want to come under that microscope of “Oh, this is just some sort of boondoggle for the executives to use monies that should be going to keeping themselves in business.”
A lot of the travel industry was hit really hard, and yet in the middle of all that, that’s when you were doing this redesign. That’s when you were doing this very extensive research project. So, you had to be out there evangelizing, I would guess.
Ken: Yes, absolutely. To be completely truthful, we had started on the redesign before October and September of last year, so we were committed at that point in time. At the same time though, you’re absolutely right. The travel industry was hurt by the stuff that you read in the newspapers.
One of the real things though—and again it points to our leadership here—E [e-commerce] is the lowest cost channel for Marriot to transact with our guests. It’s lower cost to us than a phone reservation, than using an online travel agency.
Jared: This has been excellent, Ken, and you are going to be speaking at our UIE Web App Master’s Tour, which is going to be in San Diego, Minneapolis, Philadelphia, and Seattle.
You’re going to be joining us in each of those cities and talking in a lot more detail about your experiences of doing the research and what the design team went through to build what is one of the most critical business web-based applications on the planet, your whole reservations system, which is going to be coming out with a new version very soon.
We’re excited to hear you talk about that. Do you think that people who are designing all different types of web apps, do you think that really good lessons came out of your experience that they could benefit from?
Ken: I would hope so. If you learn by things that go wrong, then I’m living proof that…
Jared: [laughs] Excellent. I’m very excited about your presentation there and I look forward to that. I wanted to thank you very much for taking the time to talk to us today.
I want to thank our audience for listening and for once again encouraging our behavior. That wraps up this episode of SpoolCast and we’ll talk to you next time. Thank you very much.