Interview-Based Tasks Brian Usability Tools Podcast Episode Seven: Interview Based Tasks. Christiansen: Recorded from the studios of User Interface Engineering, September 26, 2007. [music] Brian: Welcome, I'm Brian Christiansen, producer of UIE Podcast. Each week we'll be sharing tools for improving your site's user experience based on our research at User Interface Engineering. If you're interested in the topics Christine and Jared discuss in the podcast, sign up for our popular free newsletter, "UIEtips." I'll have more details on how, at the end of the podcast. And now, here are Christine and Jared. [music ends] Christine Hello everyone. Welcome back again to the Usability Tools podcast. Perfetti: I'm Christine Perfetti, User Interface Engineering's managing director. Each week I'll be chatting with UIE's founding principal, Jared Spool to talk with him about techniques for improving your site's user experience. Hello Jared. Jared Spool: Hi Christine. Christine: Today, just to start out, I'd like if you could take us way back to 1996 and talk about what it was like, back in the day, to conduct usability tests of websites. Jared: OK, we'll get back into the time machine. [laughter] So 1996, let's see. Clinton was still President. Oh yes! I remember now. [laughter] Basically, the way things worked back then was that - the way we all did usability testing - everybody on the planet who was doing usability testing did it basically the same way. And the way that we did it was by using tasks where we defined very specifically. We'd give people these little scenarios, we'd give them these scripts and we'd have them do things on the site. The scenarios in the scripts were hopefully, created by doing research on what people did with things. But most of the time, in fact, they weren't done that way. In fact, just a handful of us would get together on a team and we would sit down and we would think about, 'Well gee, what do we want the users to do? We have this feature. We have that feature. Let's create tasks that go test those features.' Everybody thought this was the right way to do it. Everybody thought this was OK. So, that's how we did it. That was for all types of software, all types of hardware, everything we were testing at the time. For 20 years of having done testing, that's how we did it. Christine: So over time, at User Interface Engineering, the way that researchers at UIE and you designed tasks as evolved. Jared: Mm-hmm. Christine: Can you describe a little bit why the tasks we created at User Interface Engineering have evolved? What changed your perspective about task design? Jared: Well, it was a client meeting. We were sitting in this meeting. I remember this like it was yesterday. We were sitting in this meeting and the creative director, who had brought us in, was very frustrated because of some usability test results that he had gotten from another company. Basically, this other company was an advertising firm. So back in 1996, the web was becoming big and all these firms were trying to help companies do everything they could with their website. Their advertisers said, "Well you know, we know about customers, so we must know how to do usability testing. So, we're going to do a usability test for you." So they did. The company was testing a search engine. So, this was one of the many search engines that existed in 1996. They went out and found a whole bunch of people in the Wall Street district of New York and they brought them up to their Madison Avenue offices. They sat them down in front of machines and then they gave them a task. The task that they asked them to do on the search engine was pretty much worded exactly like this, "Pretend you're interested in Leonardo DiCaprio. Go find something out about him that you don't already know." Christine: And who were these users that they recruited? Were these just random people? Jared: Well they were sort of random people. They were all random people from the Wall Street district. So, they were probably really smart bond traders, financial assistants and all sorts of people who used computers a lot because in finances they do. But beyond that, they were pretty much random folks. I can tell you they were folks who probably paid very little attention to Leonardo DiCaprio. Christine: Right, right. So, when tasks say, "pretend to be interested in Leo DiCaprio," these users most likely were really pretending. Jared: Yes, they were definitely pretending to be interested in him. This was not something that they cared about. That, in itself, sort of jumped out at us. At the time we thought, "Oh my gosh. This advertising firm, they've done such a crappy job. Of course, you wouldn't create tasks that were like that. You'd want to ask them tasks about looking up stuff related to finances." But, the fact is that their rationale for doing that wasn't any better or worse than anybody else's rationale for doing that. The methods we were using at the time weren't any more scientific or rigorous or anything that you could consider to be better. That was just the way we all did it. After we were done giggling about this particular task and asking people to pretend that they liked Leonardo DiCaprio, we began to realize that there was a serious flaw in this idea. And the flaw basically comes from this. If you ask someone who doesn't care about Leonardo DiCaprio to look something up that they don't know about him, they're going to type in "Leonardo DiCaprio" into the search engine. Everybody is going to do the same thing. And then they're going to click around until they find the first fact that they think you, as a test facilitator, will be interested in. And they're going to say, "OK, I'm done." Christine: Which is completely different than someone who is actually a fan of Leonardo DiCaprio would behave. Jared: If you had someone who was really, really passionate about Leonardo DiCaprio, let's say his stalker, and you sat that person down, they might not type in Leonardo DiCaprio. They might know that everybody calls him Leo. So, they might type in "Leo DiCaprio". And they might probably skip past the first 10 or 15 things that they saw because they already knew that because they knew this stuff. And they would probably really get into what they were doing because they were so passionate about this. You might actually, as a test facilitator, have to bring them back to the fact that you have a limited amount of time because they're going to get so into it. Christine: Right. And our researchers at UIE have noticed that engaged users will read details more carefully, that they'll be attending to very different information from people that just don't care about the task. Jared: Absolutely! We've seen this time and time again now that we've started paying attention to it. When you ask people to do stuff that they're interested in, they pay attention to completely different things. Someone who is really into classical music buys classical CDs, really very differently than someone who has never thought about classical music before. It is a completely different process. Once we said that out loud, it made perfect sense. Christine: So, given that we started seeing that there was this flaw with what we refer to as Leonardo DiCaprio tasks, what did UIE's researchers come up with as a technique to solve this problem? Jared: We put together an experiment in conjunction with this search engine client. The experiment was called the Brimfield Project. It is called the Brimfield project because there is a huge antiques sale that happens in Brimfield, Massachusetts, three times a summer. Twenty thousand people descend on this tiny little town, which normally has a population of about six hundred. Twenty thousand people descend on the town for four days to buy and sell antiques. What we did was we decided to set up a tent in the middle of the fair. In 1996, this was really hard to do. We had to get power and phone lines, all of this stuff. We set up a tent in the middle of the fair and we would recruit people who were coming to the fair because they wanted to buy antiques, or in some cases sell antiques, or whatever they were there for. We would interview them for a while and then we would ask them to do whatever they wanted to do online. What we found was this method worked. That it told us you can create tasks in the moment as the users are sitting in front of you. You can learn really, really useful things from these hand created tasks instead of having all of the wording for all of the tasks put together up front before the first user walks in the door. That turned out to be really the sort of starting point of us understanding how to do these interview-based tasks. Christine: You mention that at Brimfield we really started gather really useful information from these users; information that we could not get from previous tasks. Can you tell our listeners a little bit about what types of information that we gathered? Jared: Well, for one thing, we went in with the assumption that everybody there was a buyer, when, in fact, not everybody there was a buyer. A good number of people we talked to were sellers. Sellers look up information on other people's sites just as much as buyers do, but they are looking for different things. We met one guy who was not only a seller, but he had created a book that was like the field guide to Oriental rugs. He was putting together this pricing guide as part of this book. What he was doing was constantly checking out other people's sites to see what prices they were selling certain things at, so he could adjust his pricing guide. So, he was using it for that type of research. So, we discovered that if we asked him tasks about buying rugs, which he wasn't doing, that wasn't going to help him. But, we could get very valuable information by finding out what he was doing on the sites and seeing how he used the search engine to find the sites that he wanted to get pricing on. Christine: You are also getting a really accurate assessment of how he would use the information. Jared: Right. But, we would not have known up front to ask him the questions differently. We are still going after the same thing. We want to see him use the website, we want to see him use the search engine and how he use the search engine. It was the same basic task underneath, but the words we used to describe the objective were different. He was far more passionate about completing the results than if we had asked him to pretend he was interested in buying a rug. He is not interested in buying a rug. He has 700 of them in his truck. He doesn't need anymore. He doesn't pay list prices for them; so he is not the least bit interested in it from that perspective. But, he is interested in what other people are charging for the rugs. So, once we worded it that way, we got the exact same information from him, but in his context. We learned something about our users that didn't know before. Christine: What would you recommend? If design teams are starting to think about creating interview-based tasks with their users, how do they go about doing that? It is a very different process from creating traditional tasks before users come into the lab. Can you talk a little bit about how design teams would go about doing this? Jared: Absolutely. The traditional way that you do it what we call a scavenger hunt. So, it is basically that putting together is a scavenger hunt. You think, OK, 'what are all the things you want people to find on the site. We need people to go see if they can find this particular product we are selling. We need to have them check the about us page to see if they can find the address. You sit down up front and think of all the things you have put into the site. You basically go through feature after feature after feature. Then you create tasks to test those features. If you could see my hands you would see I have little quote symbols going there. Christine: [laughter] Jared: So, that is what we do. That is the traditional way of doing it. In interview based tasks, you do things differently. You start by saying... if we are looking at a particular part of the site or we are looking at a site with a particular type of content, how do we find people who would be passionate about that particular site? For example, we want people to test REI. REI sells camping and hiking equipment; so we want to find people that are passionate about camping and hiking equipment. How do we do that? We go and start contacting people who we know are campers and hikers. We ask them for the friends that they know. We build up this list of people who just naturally are passionate about the subject. And naturally start with recording. That is where we start with recruiting. And then, once we bring them in, we have to allocate time at the beginning of each test talk to them, to basically develop the task. What you do is co- develop the task. During the test itself, while the users are interacting with the design, we are going to actually insert their terms, their phrases, and their tasks into the test. So we change the way the test is run in that part. Then we have to analyze the data. One of the things we have to realize when we are done analyzing the data is that no two users are likely to do the same thing. Yet we are going to see patterns anyway. We have to understand what the patterns mean and how we interpret those patterns since no two people will do that same thing. So, that is basically how it works. Those are the basic steps. Christine: So, let's delve a little deeper into talking about user recruitment. You mentioned earlier, when you are recruiting users and you want to create interview-based tasks with them, that you want to find users that are really engaged in the subject matter. Are there specific questions that you recommend to clients that they ask in the recruitment stage to guarantee they are bringing in the right people? Jared: There are not specific questions because it will very much have to do with their passion. The first thing you need is a recruiter that to some extent understands what they could be passionate about. If you are recruiting people for instance, let's say retirement planning, and you have recruiter that knows nothing about retirement planning, they are not going to be able to tell if this is a person that is really into doing this or not. The first think you need to do is make sure you have a recruiter that understands the domain area; one that understands what someone could get excited about so they can tell when they have someone that is really passionate and excited about this sort of thing. You want to look at that. Then, you want to develop a database of people to call. We find that using referrals is a great way to do this. Thinking of topics for future podcasts, we should definitely do one on how to set up a recruitment database because that is a podcast in itself. Christine: Also for our listeners we have a report on user recruitment. Jared: We do. Jared: We do, we do! Christine: ...that carries a lot of this information. Jared: We have a great report on user recruitment. If this is something you're really interested in learning how to do better, that report's excellent. As an author, I say so myself. We want to look at what the user does. Bring me someone in who's passionate about retirement, chances are they've done something about it in the past. They've probably researched it. They may have invested already. They may have a whole bunch of problems that they're trying to solve. They may have some goals already established that they're trying to meet. These are all things you want to talk about in the recruitment process. This is very different than a scavenger based approach to recruiting users where you basically just establish that they are females in the age of 18-34 and that they use the Internet for three hours a week and that they like to play games online, and then you're done - you're getting living, breathing bodies into chairs and that's good enough. But, this is really going really to the extreme of making sure these people are not just people who might use the site, but are really excited about the area and really can bring something to the test with that excitement and that passion. Christine: Right. Once design teams actually are bringing in the right users, they've recruited the right users, how is facilitating a usability test, once you have those users in the lab, how is it different from just using traditional scavenger hunt tasks? Jared: The big difference, you'll notice, is at the very beginning. You're going to sit down [indecipherable] after you've done the introductory stuff and double checked the paperwork that they've filled out. You're going to sit down and you're going to have a conversation. You're going to say... so, if you were my test participant, Christine, I'd say to you, "OK, Christine, what we're doing today is we are very interested in people who like to travel to Italy. I've been told by our recruiter that Italy is one of your favorite places in the world. Is that true?" Christine: Yes, it is. Jared: OK, so, why don't you tell me about your last trip to Italy. Christine: My last trip to Italy, I was focusing on mostly seeing the main sights I'd heard about - going to Rome, seeing the Colosseum, going to Florence, seeing the duomos. So, I was focusing on the tourist activities that'd I'd already heard a lot about. Jared: OK, and was that fun? Christine: It was excellent, yes. Jared: OK, so now, are you thinking of going back? Christine: Hmmm, I think, you know the answer to this question; but yes, I am. In fact, I'm going back in November. Jared: OK, so now, is this trip going to be different? Christine: It will be different. Now, I'm branching off of the beaten track a bit. I'm planning on going to the Amalfi Coast, which is the coast of Italy, which I actually hadn't even known was there. I've heard it's actually beautiful, so that's where I'm planning to be for most of my trip. Jared: Have you already started to make plans? Christine: I have. I've booked my plane tickets. I've also bought another book, another book about Italy, where I can read up more about the Amalfi Coast and places I may want to go. Jared: Is there anything right now that's on your list of things to research before you get there? Anything you want to be prepared about? Christine: Yes, I specifically want to see if I can find an amazing B&B in one of the towns along the Amalfi Coast just to make sure I have something planned before I go to Italy. Jared: OK, well excellent. Well then, let's do that. Here's what I just did; I'm going to step out of the role here. Thank you for playing along. Christine: You're welcome. Jared: A couple of things. First, if I had just created a scavenger hunt task I might have just tried to get you to go to Rome and Venice and Florence and all the big tourist traps, right? Because that's what I think tourists do when they go to Italy. In fact, I'll be honest with you, my knowledge of Italian geography is so poor, I don't even know where the Amalfi Coast is. Christine: Right, right. Jared: OK? So, here I've learned something. Now I can start talking about the Amalfi Coast as if I've been there a thousand times. I can work with you and we can go find that B&B. That will be our first task. Instead of trying to book you into a Marriott in the center of Rome we're going to go off and find the B&B on the Amalfi Coast. Maybe, our site handles that scenario well and, maybe, I realize, as soon as I say it that it's going to handle it very poorly. But, I'm going to get a lot of information from this study. Christine: And I can tell you, as the user who's already thinking about this trip, that I'd be really engaged during the usability test to find this information. Jared: Absolutely. If I've got an hour to run a usability test with you and I've already spent 10 minutes or 15 minutes getting this information that only leaves me with 45 minutes left in the study, and I want to do three or four things on the site. I want to book your trip. I want to find some restaurants. I want to, maybe, see if there's a day bike trip that you might want to rent bicycles for. I might want to do that. I'm going to have to allocate time, right? I'm going to budget, maybe, 10 minutes for this initial task. It may be the case that you get so into this that I literally have to say, "You know, Christine, you're going to have to finish this at home because we have other things we need to do. Christine: Right. Jared: I've had users actually ask if they can email a link to their home email address because they don't want to lose their place on the page because what they just found right before we ended was where we want to go. Sometimes, as a facilitator, I'll just let them continue and I'll sacrifice those later tasks because we're getting such useful stuff. Part of my job, as a facilitator, is to have to decide, "OK I need to stop this now and we need to move onto something else, but at least we will have learned something." Again, because we're using this in an interview format I'm using your terms and your interests. Now, we're working as a partnership. It's not me delegating tasks to you and seeing if you can jump through the hoops that we've created. It's more of, "I'm getting a chance to see how you really would use this site for things that are important to you" Christine: I can't end our conversation today, Jared, without asking you one last question. It's really the question, the most common question I get, and I'm sure it's one of the most common questions you get when you're teaching interview based tasks to folks. Without fail, when I'm introducing this concept to clients, someone will raise their hand and say, "Well, with interview based tasks if every single user is doing something different how do you insure that your data is statistically significant or how do you insure that you're finding patterns in the data?" How do you typically respond to that? Jared: Usually, with a hardy laugh, mostly because... Christina: [laughs] Like that? Jared: Yeah. Ha ha ha ha ha. Statistically significant is just one of those great buzz phrases that people throw out. You know, that's another topic we should do a podcast on is "what does it actually mean to do something statistically significant?" I can tell you that if they didn't do interview based tasks, if they did standard scavenger hunt tasks the odds of those being statistically significant are so low. Nothing that we do is statistically significant because it's not an important thing to be. Given that, does that mean we can't learn anything from this? Absolutely not! Because we're dealing with qualitative data not quantitative data. Statistical significance comes with quantitative data, that's when it's most important. With qualitative data, you're getting a lot of data from a given individual, and what you're looking for are things that make you go, "Huh, we need to consider that". So, whenever you get a result you're asking the same questions. You're basically asking, "Is what the user just did something we think other users will do? If it's something we think other users will do how important is it that we design for that thing?" Those are the same two questions you're going to ask no matter how many users you see or what tasks you ask them to do. What we're doing here is we're asking those questions for everything we see, "How likely is this?" It may be the case if we test with, let's say, eight users in a study. We have them all do interview based tasks. We have one user who on his trip to Italy decides that what he wants to do is visit all the chocolate factories of Italy, OK? We could ask ourselves, "Well, how likely is it that someone is really going to visit chocolate factories. We had eight people. This was one out of eight. Do we think that chocolate factories are something people care about, versus, let's say, wine or cheese or biking the hills or visiting vineyards or whatever. Why chocolate factories?" So, we can ask that question. Then, we can go out and do some research independent of the study and find out if we think this is something that's really important for us to deal with or something that we can just chalk up to this person being a little off the beaten path. Christine: Another interesting point is, if our clients are conducting interview based tasks it doesn't mean that they can't still incorporate scavenger hunt tasks into a test. Jared: Oh, no, completely. In fact, one of the things that we often do is, we'll do around a test and maybe we'll be testing, let's say, ten users or we'll be doing what we talked about before, inherent value testing, where we'll be doing probably 12 or 16 users. What we'll do is, from the early interview based tasks we'll turn those into scavenger hunt tasks later on. We'll do things like bring someone in and say, "Your boyfriend has said that for your anniversary he really wants you to go to Italy, and he wants to do something sort of off the beaten path. He doesn't want to go to Rome and to Venice. He wants to go to the coast of Amalfi. You guys happen to like bed and breakfasts, see what you can do." It's basically the same task now. You're not going to be as passionate, but we can at least see how you might tackle that problem in this more scenario-based way. We can see if there's a difference when we put it in this scenario-based version versus when we do it in the interview-based version. If we do it in the scenario scavenge hunt version, if we do it that way, and we see that people do, in fact, behave completely differently, then we know that in our tests we don't want to as many scavenger hunt tasks for those types of things. But, yeah, absolutely. There's no reason why you can't hand someone and say, "OK, for the next ten minutes I want you to do "x," and give them a piece of paper with a task written out. Just like you would do for any other scavenger hunt task. Christine: Right. If people wanted to learn more about this topic, we do have a virtual seminar that I taught a few months back, "Demystifying Usability Tasks," where I touched upon the different types of tasks which I think could be valuable for people. But, I also believe you wrote an article about the topic a few years back. Jared: Did I? [laughter] Jared: In fact, I did write one called, "Interview Based Tasks Learning From Leonardo DiCaprio" back in March of last year. Christine: Well thanks, Jared, this was fun. Jared: Oh, there's one more resource... Christine: Yes. Jared: ...before you get rid of me. Christine: Oh, I wanted to get rid of you. Jared: I know, but one more resource which is... I highly recommend a book that was written by Janice Redish and JoAnn Hackos, it's on task analysis, "User and Task Analysis for Interface Design." This is a fabulous, fabulous book that talks about these problems and talks about what your options are, while it doesn't talk about interview- based tasks specifically. It was written just about the time that we were doing this, which was 2000; so it sort of predates it. But, it's really an excellent resource in terms of the types of things that you get when you start to look at tasks, and how people work with it. It should be on everybody's bookshelf and I highly recommend you look it up. Christine: Another book, just in terms of talking about general usability testing and how to recruit users and how to facilitate tests, would be Jeffrey Rubin's book. Jared: Yes, "The Handbook of Usability Testing." The problem with The Handbook of Usability Testing is that it's out of print. Christine: But, they're coming out with a new version soon. Jared: Right, right. Oh, you know what? It's interesting. I just looked at Amazon right now and they're selling it for $39. It's been selling for $90 for the last month. This is good; go buy it now. Go buy this book. It's an excellent book. Yeah, they are coming out with a new... Dana Chisnell and Jeffrey Rubin have sat down and are writing the second edition. I've actually been consulting to them about the book and I'm very excited about it. I think it's going to be an excellent book. I believe Dana and Jeffrey are talking about task design in far more depth, about this and so that... absolutely. Christine: Excellent, well thanks, Jared. Jared: Thanks, Christine. Christine: For those listeners out there who have specific topics that they'd like us to discuss in an upcoming "Usability Tools Podcast," I highly encourage you to send us an email with your topic ideas at, mailbag@uie.com. Since we seem to be enjoying these podcasts, Jared, I think, we're still planning on doing these every Monday. Jared: Yeah, absolutely! Christine: So, if anyone has any topics that they'd like us to discuss for a few minutes, I highly encourage you to send those along. Also, we mentioned Jared's article about interview based tasks. We first published this article in "UIE Tips," our email newsletters. If you'd like to be reading about our upcoming research, the latest research coming out of UIE you can sign up for our email newsletter for free at www.uie.com/uietips. That's it for today. Thanks a lot, Jared. Jared: Thank you, Christine. [music] Announcer: We hope you've enjoyed this "Usability Tools Podcast." If you're interested in more of UIE's research sign up for our free email newsletter. You can subscribe easily at www.uie.com. If you'd like to attend our next virtual seminar for free just fill out our short podcasting survey at www.uie.com/audio. We'll give away free admission to one lucky respondent each week. We love hearing from you. Send us your comments at, mailbag@uie.com. That's all for this week; thanks for listening. Goodbye. [music]