SpoolCast: Jared Spool with Dana Chisnell Podcast Transcript Brian Dana Chisnell interviewed by Jared Spool, recorded June 7, 2008. Christiansen: [music] Brian: Welcome. I'm Brian Christiansen, producer of the UIE Podcast. In this week's episode, Jared speaks with Dana Chisnell, usability expert and principal of Usability Works, a consultancy based in San Francisco. Dana is the co-author of the recently released second edition of the "Handbook of Usability Testing". Dana will be presenting "Usability Testing Guerilla Techniques: Collecting User Data on a Shoestring" at our User Interface 13 Conference, which will take place October 13-16, 2008 in historic Cambridge, Massachusetts. You won't want to miss it. And now, here's Jared. [music] Jared Spool: Welcome, everyone. This is another episode of the SpoolCast. And, I am very pleased to be sitting here in the UIE studios with Dana Chisnell, who is the recent co-author of the "Handbook of Usability Testing", of the second edition. And, she's also going to be presenting at the User Interface Conference in October. She is a principal at Usability Works and welcome, Dana. Dana Chisnell: Thank you. I'm delighted to be invited. Jared: I'm glad you were able to make it. So, what I wanted to talk about, you and Jeffrey Rubin just came out with this great new book. "The Handbook" has always been a resource ever since the first edition came out in 1994, but we went off and you did this whole new version. And, in doing the research for this book, did you get a chance to think about what the difference is between someone who's good at user research, and someone who's like really great at user research? Dana: Yeah, in fact, I thought about it a lot as I was writing the book, and talking to Jeff about the differences, between what was going on in 1994 and what's going on in 2008. In 1994, when the first edition came out, Jeff wrote a book about usability testing, which was the main tool, and just doing a usability test would make you a great designer of what was unknown about experiences then. But now, it takes a lot more than that, consumers are a lot more demanding; design is a lot more involved. Technology is everywhere, so that means that you need a lot more tools and techniques than just the hammer that usability testing can be. Jared: So, what are some of these techniques that make a difference? Dana: Well, it takes a bit more research. You have to know a lot more about who your users are. You can't just assume that they're novice technology users anymore. They have much broader experiences and technology fits into people's lives in a much bigger way. So, there's a level of vision that an organization has to have to be able to look out and do some homework about who their users are, who the buyers are, who may not be the users, and how those things fit together. Jared: So in terms of putting these visions it's just like corporate vision, Corporations have missions: We want to be the best doing the thing that everybody else does, so that we can return a huge profit to our shareholders. Are you talking about that vision, or are you talking about something different? Dana: Well, it is part of the mission. Of course, the mission of most corporations is to make money, but they couch that in some careful way, but it ends up translating to design of user experiences, and talking about how the company wants to interact with their customers. Enterprise Rent-a-Car, for example, spends a lot of time and energy thinking about every step of the process where they have an interaction with their customers. So, it's not only usability testing of a website, but standing around in the stores, as they call them, and observing people getting their cars or turning their cars in, and seeing how those experiences fit together. Jared: So at looking at all the different touch points and understanding what is it that we really think the ideal experience could be. So, what role does user research plays in the formulation of that vision? Dana: Well, it's part of a strategy. So, you create a whole plan for the things that you need to find out, and all the ways that you want to try to find those things out. Once you have a vision of what you want the user experience to be like, you have to create a strategy for reaching that vision. And, that incorporates any number of methods and techniques at the hand of any good designer, but there are lots of people in the organization who can do design. So, it's about understanding the context what the user is and doing research about that, and what their goals are, and how those match up to the business goals and objectives. Jared: Let's say I've been working on conducting usability tests in my organization and we've been bringing people in and testing out a particular function, and saying OK this function works better now because we've been doing this testing, and all that's been happening. How do I then start transitioning to something where I get more of a focus on vision versus just a spot check of functionality? Dana: Right, yeah, a lot of companies do that, just look at, kind of, very local issues on their site, fixing niggling problems like sewing a button on a shirt. But to back up, then you need to probably go out into the field where the user does what they normally do. Get out of the lab - that's one of the recommendations that we make in the book it's to do as much as possible on the site where the person is doing what they're doing with your organization. Jared: So, going out and seeing, if they are entering information, you know, if I'm booking a trip, how do they decide what city they were going to go to? Maybe they were business user and they're being told to go to this. Or, let's say I'm renting a car, and one of the things that every car rental website asks for is the flight number. Do they have that handy? Can you look it up for them? You know, stuff like that. Dana: Right;. Jared: One of the things you can see when you go out on site is, "Oh my gosh" I don't know, I just booked the flight five minutes ago and I can't remember the flight number. I'm going to have to go and go back to the website and bring up the itinerary for that flight, then type in the flight number and the time that it's going to get there - those are things computers could do. So, you could figure out that there maybe there's a chance to optimize the experience at that point. Dana: Right, but you're also looking at the decision-making process that the user goes through. Do they shop around for good prices? Do they care about scheduling? Or, if you're booking a flight does it matter what airport you're going out of? Jared: Right. Dana: If, people make different decisions based on whether it's a business trip or a leisure trip, and who is paying? So, whether they are using points or whether they are paying for it themselves, those kinds of things, and all of those factors figure in to what the experience is like. Jared: So, vision is one piece. What's another thing that separates good from great? Dana: Well, there's the vision, then creating a strategy for reaching that vision, but neither of those things is any good if people throughout the organization are not involved in making those things happen. First of all embracing that strategy being part of creating that strategy but then also carrying it out. It is fine to have a few people in the organization who are knowledgeable about methods for doing user research. But it is really great if everybody who is on the design team for whatever the product or the service is to be involved in doing that research and because they learn firsthand that way. Otherwise, there is this concentration of knowledge that doesn't get transferred very carefully or very clearly. And the value doesn't get implemented for the organization or for the user if it is just stuck there in the usability lab or on a shelf with a bunch of tapes and recordings. Jared: But, isn't that of concern that if you have designers going off testing their own stuff that they are going to have this crazy bias that is going to cause them to somehow ignore the realities that users are struggling or coated over or say that was just stupid users or.. Dana: Yeah, sure. We talk about that all the time in this community, but there really is no research to support that, that is a bad practice to have a designer evaluate or observe someone using their design. I think it is a good idea for the designer to see it firsthand. You can discount that, but after you have seen a few people have the same kinds of issues or questions, suddenly it stops being such a recipe for defensiveness and more a situation where you are actually beginning to learn because you can see the nuances of what is causing those issues. Jared: Right, I have never understood where that notion came from because every time I am sitting with designers and they are testing their own stuff that theirs is to not make changes before the users left the room, "Hold on, I just want to recode this and then we'll try it again" Dana: Do it again. Yeah. [laughs] Jared: You can recode when they leave. Let's keep going. Dana: Let's wait, at least, until the end of this session and go on. Jared: Would you say that the folks who are best at doing user research are really involving the teams tremendously? Dana: Oh, yeah, absolutely. Whenever there is inter-disciplinary coordination then it is a much richer exercise. You get much richer data out of people who work in different groups in different departments who have different specialties or interests when they are observing the same thing. It's sort of a Roshamon effect, right? Everybody saw the same thing, but they see it from different points of view. Some, the stakeholders or the business people see it from the bottom line point of view whereas the technology people see it from the feasibility of implementing improvements. There has got to be a match up there at some point or, at least, a discussion about how those things can match up. It works much better if individuals from those different groups can be part of the research firsthand seeing than does if some one or two people go out and do the research and come back and report. You lose a lot in the translation. Jared: So, we've got vision. We have involvement. Dana: There is strategy in there as well. Jared: Oh, and there is strategy. How does this play out in terms of when we see teams that are doing these things? What stands out to you in terms of the results that they are getting? Dana: Teams who do this have much deeper and much more holistic understanding of how their products fit into the worlds that their users inhabit. They don't make the mistake of thinking that their thing is the only thing in their user's world. So, there is a much smoother fit for users into what they are doing and what their goals are when companies can do this. The other thing is that companies just tend to turn out better product and do better service when they take this approach. Jared: Yeah, so they just get a better product, better service which then yields into higher loyalty to the company, to the brands which then they start to see the passion and the pride in the users because they are getting that higher brand engagement which then turns into devotion and that crazy cult-like behavior that you see with, for example, with Apple customers. Dana: Yes, for example. Jared: So, absolutely. In terms of what it takes, if I am good at what I do but I really want to get to that great point, what are some sort of starting tips? What are some things I can do to push the envelope for me a little? Dana: One of the things you can do is share the love inside the company. Don't own user research yourself but teach other people, coach other people in the company how to do it and be willing to let go of ownership. Jared: What if they do it wrong? Dana: They are not going to do it wrong. Honestly, rigor can be good and there are many people in the community who are rigorous, super, super important, but I think much more important than that is that people on the design team get the experience of observing users, talking to users, listening to users about the products and the services that the designers are working on. OK, so the interview is not exactly the same from one to the next, so what? They are still learning things. It's all good. Jared: Yeah, but they don't know how to calculate a confidence interval. Dana: I don't either. I've got to look it up every time. [laughs] What do you need that for? Jared: But if everybody does usability testing then you won't need people who do user research and the staff, right? Dana: Right. Yeah. Jared: So, then, I am out of a job. Dana: No. You have a better job then because you are not tied to the lab and doing mundane things like recruiting participants constantly. You are now the center of truth revealed for informing design for your organization. I think that's a much better fit. For me, that's a much more interesting job. Jared: So getting out there and teaching other people how to do this and then once that is done, what becomes that next piece? Dana: Well, this is where the strategy comes in again. You have to start creating a sort of rolling plan, a rolling forecast of what's coming into the company. What is the company strategizing about? What is in the pipeline and working with managers and executives who are involved in those areas and getting them to sign up? You know, working with them, partnering with them to do the right thing from the beginning of their product. Jared: Would this be like an activity such as looking at a given set of usability tests that you've planned out and figuring out, "Well, if we take those users and we add ten minutes or fifteen minutes to each session, could we find out something about some features we're planning that are a year out or two years out?" Could we start to collect data now about things that are in the way future at the end of a test because we just happen to have the users there? Dana: Oh yeah, yeah sure. There are a lot of ways to do that. There are a lot of ways to make use of having participants in the room that you're already paying. So absolutely; there's no reason that you have to do a staid kind of usability test in that little time frame that you've got. Jared: But you really need to know what the strategy is long term. You really need to be involved in that so you can know, "What information are we going to need six months from now?" I think a lot of good usability folks are reactive in that they're collecting information that the team needs right away, and the teams says, "I need to know this information." So OK, we'll construct a test, we'll go get users, we'll bring them in, we'll watch them, or maybe we'll go out in the field and we'll watch them do that thing. But to be able to say, "Gee, you know, there's been talk about us going in this direction. What research can we do on competitors' systems, on prototypes? Can we go into the field and, while we're out at people's houses, just collect a little bit more information about their use of this other thing and bring that information home so that when that team is assembled... Those teams may not even exist yet; you just know that that's something people are thinking about. Dana: Right. It's when you have some insight into what's coming down the pipeline that you can start to do those things and build that into the research that you're doing. So even if you're doing summative testing you can tack on an exploratory thing to do a proof of concept or to explore some potential want or need in coordination with doing usability tests or doing other kinds of research. Jared: Makes perfect sense to me. So, in October you're going to be doing a session on guerilla usability testing. Dana: Yeah, I'm really excited! Jared: This is really about looking at things on a shoestring. Can you talk a little bit about what people are going to get out of that? Dana: Well, the idea is that you don't need to build a hundred-thousand dollar usability testing lab to be able to do user research in your corporation. If you have five bucks, you can do some research. If you have fifty-thousand bucks, I'm going to talk about what you should be doing with that. Jared: You should be hiring us. Dana: [laughs] No, you should probably be hiring headcount so you can do it internally. Jared: Oh, oh, oh, doh! Dana: [laughs] Yeah, well you know I'm an independent consultant too, so I'm selling my soul here really. Jared: Don't tell them, don't tell them. It'll just be between you and me. Dana: OK. Jared: They didn't really need us ever, did they? Dana: No they didn't. Sometimes they don't. Jared: Right. [laughs] Dana: So it'll all be about usability testing; from recruiting to conducting the sessions and analyzing the data, then reporting findings. But the theme throughout will be, what's the budget? And if you have a little bit of money, how can you make the most of that? And if you happen to have a bucket of money, then what's the most value you can get out of that at different stages? Jared: And so this is a good session for people who haven't done any testing before, right? Dana: Yeah, I think so. Jared. And for people who've done it before, are they going to get value out of it? Dana: I think so because they'll be able to start building their strategy, right? So, they'll be able to think about, "OK, I have a budget cycle coming up. I need to make my budget requests." They should be able to get some ideas about what's feasible at different stages in their studies, or across the budget year, and build that into their requests based on what they hear here. Jared: You were telling me earlier; you're going to actually bring out some actual budgets and pricing. Dana: I'm not making this up. Jared: You're not going to make them do spreadsheets, are you? Dana: Yeah. I might. [laughs] Actually, we do have a worksheet that we'll go through that will show a spread of, if you do a couple usability tests a year, what that looks like at a very guerilla level, stealth, just under the radar... How many clich's can I use? How many metaphors can I use on this? [laughs] Nobody's noticing you doing usability testing or doing user research yet. You're just paying for your own gas to go do visits to doing exploratory studies that are much bigger, that involve a lot of contextual research, to doing summative usability testing when you've got a design that you can gather some serious quantitative data on. We're going to look at those across some spreadsheets, so yes, you'll have to do some math. But don't be afraid. It won't hurt. Jared: OK. Dana: We'll try to demystify those things so you can go talk to your manager and feel confident. Jared: There we go. Well thank you Dana. We've been talking with Dana Chisnell. She is the co-author of the newly revised second edition for the Handbook of Usability Testing, an absolute must-buy book if you are considering doing any usability testing or even if you've been doing it for a long time, there is all sorts of new. Did you guys update it for this millennium, right? Because the original one was like 1994 and the web was not even out, were there any web examples in the first book? Dana: There were no web examples in the first book. In fact, most of the computing examples were DOS. Jared: Wow. Dana: GUI was not really out. Jared: That's amazing. Dana: Yeah. Jared: So, you have such a great book. I got to read an early draft. They asked me to do the Forward. Dana: I appreciate it. Thank you for doing that. Jared: You're welcome. It turns out to be just a wonderful resource. I know people are going to want it. Dana: You know, for a guy who says that he doesn't like usability testing you seem to like the book OK. Jared: [laughs] Yeah, I like the book a lot. And you're going to be teaching the full-day workshop, "Usability Testing Guerilla Techniques: Conducting user research on a Shoestring, " at the User Interface 13 conference in October in Cambridge, Massachusetts. Thank you again for talking to us today. Dana: Well thank you.