On-Site Search Usability Tools Podcast, Episode Eleven, "On-site Search", recorded at the studios of User Interface Engineering, November 20, 2007. [music] Brian Welcome. I'm Brian Christiansen, Producer of UIE Podcasts. Each Christiansen: week, we'll be sharing tools for improving your site's user experience, based on our research at User Interface Engineering. If you're interested in the topics we discuss in the podcast, sign up for our popular free newsletter, "UIE Tips". I'll have more details on how, at the end of the podcast, and now, this week's episode. [music] Brian: Welcome back to another episode of the Usability Tools Podcast. This is Brian Christiansen again, here with Mr. Jared Spool. Hello, Jared! Jared Spool: Hey, Brian! Brian: What do you say we talk about On-site Searching today? Jared: That sounds excellent. Brian: All right. We don't talk about Search too, too often, but it's still a pretty large topic in our field. Why don't we talk about Search that often? Jared: We don't really talk about it because basically, we see it as a symptom of a larger problem. We've talked in the past about how our research has shown that people don't actually go straight to the search engine, except in very specific situations. Instead, they use the Search only because whatever links are available on the page are just not working for them. Basically, what they're trying to do is compensate for a bad design, so we don't talk about Search because talking about Search would be fixing the symptom and not fixing the problem. The problem is the navigation, so we spend a lot more time talking about navigation. Brian: Now, we taught in a previous show that there are no users that naturally Search, so that would mean we should focus on improving navigation, I would take that. Jared: Yes, that's basically it. It's the design that forces users to Search. People don't wake up in the morning and say, "Oh, today's going to be a great day, because every website I'm going to go to, I'm just going to use Search!" [laughter] It doesn't work that way. They're looking for something, and they only use Search because the design of the site has forced them into doing that. Brian: So what designs force users to Search? Jared: There are some places where users will more naturally Search. They're very rare. They're places where people are referring to things by what we call "uniquely identified items". They are things like books, and CD's and movies, so Books and CD's and Movies are where people search. They go and they type in the name of an author, or they type in an actor, or they type in the title of a book or the name of a CD they want. That's basically the situation. Everyplace else, people try to use the links first. They don't always succeed, but they try. Even a site like if someone's looking for a book, if they're looking for a Harry Amazon: Potter book, they're going to type in "Harry Potter". But if they're looking for a DVD player, they're going to try to click on something that indicates Electronics or something first. Rarely, they'll go to Search. They want something that gives off a clue to the design. Basically, what happens is: people come to the page; they scan the page for their "trigger words," their words that they want to click on. Then, if they don't find those "trigger words," at that point they go to Search. When they go to Search, what they type in is, in essence, their "trigger words". They're basically creating their own link for that page, and then clicking on it, and crossing their fingers and hoping they get the results they want. Brian: So if designers were to dig into their server logs for the sites they're working on, they could, perhaps, improve the content on their page by finding these "trigger words". Jared: Absolutely. The server logs are filled; the Search logs in particular, are filled with "trigger words". You can figure out from your logs not only what the user typed in, but what they typed it in from. They're basically telling you what links were missing from that page, and probably the page before it that they visited too. So you're getting clues all the time. That's something people don't look at. IBM, at one point, went through their Search logs, and found 80 Search terms, found out what pages to put those in, and saw a 300% increase in revenue the next day in their laptop section of their site. Brian: We've conducted the studies watching people use on-site Search. Can you talk about how we went about doing that? Jared: Yes. We started by watching a lot of people shop online. In the process of shopping, we noticed that sometimes people Search, sometimes they don't. We watched what they searched on, how they looked for stuff. That was where it all started. We collected a whole bunch of terms. We saw when they would type things in wrong. They'd type in stuff that the search engine didn't expect, things like that. I watched someone on the Macy's site, for example, Search for the word "slipper" and Macy's came back and said they don't have any "slipper". [Chuckle] Turns out you had to make it plural, as "slippers". Brian: No one buys just one slipper! [laughs] Jared: That's right. We collected all these terms, and then we decided to put together what we called an "obstacle course". We went out and did a study that didn't involve users, but used the terms that users had used, and we created standardized Search queries, that we looked at 80 different sites with. What we did; we would go in and we would type in various Search terms in these sites. Now, some of the Search terms were terms that were for things that we knew were on the site. In fact, in some cases what we did was: we just copied the text that was on the site, and pasted it into the Search box, to see if the site could find stuff that was there. So we did stuff like that. We would type in things like "slipper" to see what would happen, on a site that sold clothing. We had a variety of different things we would do. What we found was that there was a whole spectrum. Some sites did really well, some sites did really poorly. The best performing site was Office Max, actually, of all the sites. Its Search engine really performed the best of all the others. Office Max did really well! What was interesting was that there was a huge spectrum, from the best to the worst. I mean there was a huge variety, that allowed us to go in and start comparing, and see what attributes made the best the best and made the worst the worst. We could then start to identify what really makes a good Search engine. Brian So what did separate the best from the worst? Christiansen: Jared: The biggest thing that separated them was the ability to come back with what's called a "match relevant" response. "Match relevant" is something that is pretty much what the user's looking for. The example that I like to use is: we typed into the version of Office Max that we were using "shipping tape". It came back with six items, all of which you could use for shipping tape. What was really amazing about those six items was that not a single one of them was called "shipping tape", or was "shipping" used in the description anywhere. They were packaging tape, or masking tape, or boxing tape. They had different names on them, but none of them were called "shipping tape". Brian: So somebody had to go in and manually associate that. Jared: Exactly! Someone paid a lot of attention. On one of our worst- performing sites, a site called Reliable Office Supplies, they weren't nearly as reliable. [laughter] when you typed in "shipping tape," they came up with "no results". None whatsoever, even though, if you looked under "packaging tape" and the other types of tape, they actually had 24 different products on their site that could have been useful as shipping tape. The Search engine didn't produce any of them. What really separated the difference was: on the one end of the scale, the best sites were best at producing "match relevant" results, and didn't give "no results", and the worst sites gave a lot of "no results", and the results that they gave weren't relevant at all. Brian: There's lots of talk nowadays about which search engine is best, and everybody loves Google, and you see Ask on TV and all this stuff. Can the Search vendor you use, that you plug into your website, make up the difference? Does it matter what the platform you're using is? Jared: We found that it didn't. We thought at first, maybe some vendors are great. So we contacted all the different sites and we said, "What vendor are you using?" We found that for everybody who was at the top of the list, every vendor at the top of the list had as many sites, or more, at the bottom of the list, and that it wasn't specific to the vendor. It's tied specifically to the implementation. To give you an example of this: one of the sites we tested was Eddie Bauer. We were excited to test Eddie Bauer because the folks at Endeca had implemented the Search engine. Endeca has a very good product. We've worked with them for years. Brian: They're right down the street, practically. Jared: We know those guys really well, and they're really sharp. They've got really nice products. So we were very excited to test it. The Eddie Bauer site didn't do so well in our study. In fact, we typed in "women's pajamas" and we got things like men's jeans, wallets, and flannel shirts. It wasn't clear at all why these products were coming up. So we called up our friends at Endeca, and a guy named Pete Bell on the phone, who we've worked with for a long time. We asked Pete why we were getting all these wacky returns from this query on Eddie Bauer. He said, "Hold on," and he put me on hold for about 35-40 seconds. He came back on the phone, and said, "OK, try it now." Sure enough, when I tried the second time, it came back with only women's pajamas. Brian: huh. And that's service![laughs]. Jared: Yes, that was great. I asked him what he did. He said, "Oh, there was a typo in the configuration file." There was like one character change and boom. That's what we found was the big difference between the best sites and the worst sites; was that the best sites were configured properly, and the worst sites were configured poorly. It seems, with all of these vendors, it's much more of an issue around the configuration and the installation, than it is around the specific vendor. Brian: So it's implementation and execution, basically? Jared: Exactly. Brian: So you just can't show up and boom; it's better. Jared: Exactly, this implies that you're going to have a ton of work when you put one of these things in. It's not a simple task. You're buying a lot of work when you spend big money on one of these search engines. Brian: Sure. So, how can you tell if your search engine is working on your site, or whether you need to put some more effort into it? Jared: Basically, there are four types of things that come back when you type a query in. The first one is "no results", right? So you type in something, and it says, "I'm sorry, but we don't have that.", actually, most of the time, the sites don't say, "We don't have it." They say something like "We couldn't find it," and suggest you type in something different. Brian: Right. You made the mistake. Jared: You made the mistake, right? That's probably the worst thing you could do. If it's something you actually don't have, you just want to come out and say, "We don't have that." But, in our obstacle course, we knew that 80% of the things were already on the site. We'd found them and identified them on the site, so we knew for a fact, they were there. Yet, the worst sites would have 40-45-50 percent "no results". So, "no results" is the worst thing that you can do to somebody. The second-worst thing is to give them what we call "whacko results". [laughter] We call them "whacko results" because we had a user who was looking at something that just didn't make sense, and she just kept saying, "My God, this thing keeps giving me these whacko results," and that name just stuck. Brian: It's a technical term now. Jared: Yes. It just stuck. So, we called them "whacko results". What they really are, are completely irrelevant things, things that have nothing to do with the search term. If you look at them, you just can't understand why they're there. What we found is that having those things show up in the top ten search results: bad thing! As soon as a user finds a whacko result, they stop looking beyond it. Brian: Right. Jared: So, you don't want them to run into it, because that, to them, means, "OK, we've hit the bottom of the usefulness of the search engine. There's nothing beyond this point." The next result could be exactly what they're looking for. They never look at it. Then there's the third type. We have "no results" and we have "whacko results". The third type is what we call, "related results". I type in, "twin sets" and I get results for sweaters. If I know that a twin set is a type of sweater, that result will have meaning to me. But, if I don't know that, I could consider that result "whacko". Those are border-line, and we see that a lot, where the user doesn't understand why the result was presented. What you end up seeing is then the user has to "pogo stick". They have to click on the result, and see if it's what they want. "No, it's not what I want." They go back to the search engine results. Choose the next one. So, you see that "pogo sticking" thing going on. As soon as that happens, the chance that the user's going to actually find what they're looking for drops tremendously. So you have to be very careful with "related results". The fourth type are "match relevant" results; results that actually match. With those four types, you can then start to look at what's going on, on your site. The first thing you should be doing is going through, let's say, the last month's search engine results. If that's too many, just do the last two weeks or the last weeks, whatever. Take some number of results, let's say, 100-150 results or queries. Go through the logs, find the queries, then, type them in and see what you get for a return. See what your competitors get. We actually go through and count the number of results that show up in the top ten or in the top twenty, and classify them as "no results" queries, "whacko results", "related results" and "matched relevant" results. We count them up, and then we produce charts that show what you do versus your competitors. Instantly, you get this picture of what's going on, and then, what you can do is start to tweak. You can start to tune your site, and look for your most common search terms, and make sure they always produce "match relevant" results. Brian: Which brings up the question that I'm sure is burning in the minds of our listeners right now: how do I do that? How do I improve the Search? Jared: That's going to depend on your tools. There aren't any specific ways, but almost every search engine will allow you to forgive a keyword, to be able to insert a particular result in that keyword. Some companies call them "Best Bets". There are all different terms for it, but basically, it is the ability to hard-code things in. You should start by looking at your most common terms, making sure that you have hand-tailored the results page for those terms. If you do that, you're far more likely to get qualified results that mean something, than if you don't know what people are searching; if you don't know what terms are coming in. What's really interesting is that the top terms at an electronics vendor for the last three years have been iPod and Wii, for the last year and a half. It's not hard to know what should happen when you type in those things. And they're finding that if you go to a lot of electronics vendors, because the Wii is very hard to get, with many of them, if you type in Wii, you don't get any mention of the game system. It doesn't say, "We occasionally have the game system, but it's currently not in stock." Instead, the user has to go through and search through these long listings of all the Wii games, and controllers and accessories. Brian: Where's the Wii? Jared: They can't find the Wii, and they don't walk away with a positive thing. That page for the Wii should be hand-crafted. It should say, "It's not in stock right now. It's going to be in stock later. Would you like to sign up on an email list? We'll let you know when it's in stock." Imagine what you could do to create that experience, so that people are then part of that situation. That is something that you could do, but nobody does it. So, you want to look at it. The whole idea is that you want to reduce the "no results". Try to eliminate them for your top 200 terms. Increase "match relevant" results. Look at where the "whacko results" are coming in. You want to do things like look for common typos that people make. Make sure your searching for things where people have transposed letters, typos. Brian: Did you mean...? Jared: Right. Spelling mistakes also. A spelling mistake is different from a typo. If I look at a typo I can instantly see that I've made a mistake. If I look at a spelling mistake, I may not realize that I've made a mistake. So, you want to look for both of those things. You want to deal with plurals. What happens when someone looks for a slipper? At one point, on the J.C. Penney site, if you searched for "maternity" with maternity spelled wrong, it would actually find the right thing. But, if you searched for "maternity clothes" with it spelled wrong, it wouldn't. Those are the types of things. You want to be going through your logs, looking and seeing what people are typing in, and what the results they're getting are, then tuning your engine to account for that. When you do things like that, you're going to see a dramatic improvement. Brian: It sounds like you can really tune your experience by going through and finding the "trigger words", addressing these few areas, and turning some possibly bad experiences into good customer experiences. Jared: Right, but the key thing is to make sure that the links in the first place are there. People shouldn't be searching on Wii and iPod at these electronic vendors. They should be the top links on the Home page, because that's what people are coming for. Brian: Avoid the search altogether. Jared: Exactly! Brian: Well, great! I think this has been very enlightening, that's all for this week. Thanks, Jared. Jared: Thank you, Brian! [music] Brian: We hope you've enjoyed this Usability Tools Podcast. If you're interested in more of UIE's research, sign up for our free email newsletter. You can subscribe easily at UIE.com. If you'd like to attend our next virtual seminar free, just fill out our short Podcasting survey at UIE.com/audio. We'll give away free admission to one lucky respondent each week. We love hearing from you. Send us your comments at mailbag@UIE.com, that's all for this week. Thanks for listening. Goodbye. [music] [End of recording.]