Outsourcing Your User Research Is Like Outsourcing Your Vacation

Jared Spool

August 2nd, 2011

Hang around me long enough and you’ll hear me say this:

Outsourcing your user research work is like outsourcing your vacation.
It gets the job done, but probably won’t have the effects you were seeking.

I usually say this when someone is asking me to do their user research for them. This is something we did quite a bit in the early days of UIE, but don’t do any longer.

Usually, they are asking us to do this to save time, because they don’t have trained folks, or because they are afraid of bias. All these reasons are rational, but there are better ways to deal with them than hiring someone else to do the research on their behalf.

As I said, I founded UIE as a company to do just this. I felt the rational reasons where why companies weren’t conducting their own research. I thought we could offer cost-effective, inexpensive research services to help. User Interface Engineering, in 1988 (it was our 23rd birthday yesterday!), was one of the first companies to make user research services available to other companies.

However, after working with hundreds of teams and providing their research, we started to looking at how effective we were. Were the teams’ designs getting better? Were they doing more research? Were they creating better user experiences?

We were sorely disappointed with our results. While every team told us they really got a lot out of our work, most weren’t improving their designs. They were appreciative of our reports, but hadn’t read them. They enjoyed our presentations, but weren’t really adopting the recommendations. And, most importantly, their culture didn’t change — they weren’t integrating users into their design process any more than before.

It wasn’t only UIE’s clients with this problem. We reached out to organizations using other outsourced user research services and discovered the same results. Hiring the work out wasn’t getting the job done.

We realized that we were missing an important variable in user research: the team’s direct exposure to their users.

When we take a team on a field research project, we introduce the team members to their users and having them spend time seeing them use the product and doing their work. In doing this, we’ve accomplished 90% of the work of the project.

It’s the exposure that changes the way people work. The same is true for usability testing or interviewing users. The direct exposure is the most valuable part of the project.

When you hire out your user research, even to the most competent of user research professionals, you’re losing 90% of the value. The research becomes a game of telephone, where the “away team” (to steal a Star Trek term) learns all about the users and somehow has to communicate back what they’ve learned. No mount of report writing or presentations can replace that lost experience.

Some UX service companies will tell you that they’ll remain part of the team, integrating the knowledge they learned into the design as the project continues. However, that creates an imbalance, where some people on the team know the users well and others have no idea. Those others, who will eventually own the entire design, are working at a disadvantage and won’t be making their design decisions using this critical knowledge.

This is why we now refuse projects where the team wants to outsource their research. We still do plenty of field visits and usability tests with our clients, but only if they come along to every session. If the client team isn’t there, we won’t conduct the session – there’s no point.

For the folks that think they don’t have time to do their own research: You’re better off taking the money you’ll spend on hiring someone and burning it in the back yard. You’ll get the same value in your product.

Seriously, if you want someone else to do your research because you don’t have time, you’ll need to dedicate twice as much time to spend with the researchers, extracting every little thing they learned about your users. Otherwise, you won’t get the value you paid for. It’s not a time saver to go this route at all.

For the folks who feel they don’t have the skills onboard: That’s an easy problem to fix. Training on user research methods is pretty easy. This is the bulk of our consulting work these days. We use a “Watch one, Do one, Teach one” approach. (We stole it from the medical training world). Most teams pick up the skills pretty quick and do a damn good job in just a few weeks.

And for those folks who feel doing your own research introduces a bias: You’re right, but it doesn’t matter. There’s always a bias in research, even when you get a third party to execute it. There’s nothing wrong with biased research, as long as you understand your biases and how to counter act them.

If there’s anything you can outsource, it could be participant recruiting. However, make sure you work with someone trained in UX recruiting, not market research recruiting. UX trained folks (we use Usability Works – they’re awesome!) know how to deliver the information they learn about your users in the process.

That said, you should even try to resist outsourcing your participant recruiting. You learn a lot when you talk to your potential users, even if they don’t qualify for the study. When you’re outsourcing it, you’re flushing a lot of great source material down the toilet.

Once you’re in the habit of doing your own research, you’ll never want to go back. It’s just too awesomely addicting and useful.

9 Responses to “Outsourcing Your User Research Is Like Outsourcing Your Vacation”

  1. Steph Beath Says:

    Hats off to you Mr Spool! This is so well said I have put it on my wall and printed copies to hand out to the next person who (a) is too busy to be part of the research we’re conducting for their project (in house) or (b) wants to relieve my workload by offering me help from an external agency. These are common challenges I face and I get tired of repeating the mantra of DIY learning. Fantastic to have this coming from you – now I have back up!

  2. Kris Says:

    Happy birthday for yesterday!

    This article is timely as I was considering having someone else do it for me. After reading this I am going to go and do it myself!

  3. Jens Says:

    Loving it.

    I totally subscribe to the “go and see for yourself” approach. There is even a phrase for it: Genchi Genbutsu: http://en.wikipedia.org/wiki/Genchi_Genbutsu

    Allright, it is a Japanese phrase, but nonetheless!

    :)

  4. Dave Malouf Says:

    I generally agree with the total premise of the article, but there is something that I have found as an innie that there are 2 reasons I have found that outsourcing research does that you can’t do inside many organizations. 1 of the reasons is not a good one, but a real one, and other I have seen be truly effective.

    1) The outsider is always right
    Companies bring in consultants to say exactly what the internal teams have been saying all the time. Yes, this is a horrible reality, but I think it is still a reality. Sometimes I have used this to validate my existing research and give it more weight like an independent certification of sorts.

    2) The outsider sees things you can’t
    A clear set of eyes without corporate bias goes a really long way in getting to insights you may have glossed over yourself. This is the reason I use outside researchers more than any other.

    3) Yup, I said 2 but I’m going with 3 … More people
    Sometimes you pipeline is just bigger than your team’s capabilities. Sometimes saying slow down is not an option when you have quarterly reports to give your investment community. Not a great reality, but reality nonetheless

    Again, I get the spirit of what Jared is saying and it goes right in line to use talk about what teams have in common that create great experiences and one big one is contact. I would offer that just because you hire someone to do research, it doesn’t necessarily mean they are doing it “for” you. It could also mean they are doing it “with” you. They bring in expertise, and support resources that aid your team. They don’t come in and throw a report at you, but facilitate your participation and emergence with the process of doing research.

    I think it is fool hardy to be absolute about this topic. Your great adage Jared is “it depends” and I think on this topic, it really does. Not every organization is ready to jump in. They need to get there and outside help gets them there. Consult to the context, not to the ideal.

  5. Rina Says:

    Great article and so very true – have more than once changed web content and navigation after research and feedback from users, only to have to defend these decisions from colleagues within the organization who have no contact with the users!

    Have to agree with Dave Malouf’s Point 1 though, sadly the higher authorities often won’t take it from their own staff but will pay a contractor to say exactly the same thing for a lot more cost and then accept it as though it’s a new idea.

  6. StartupDigest Says:

    [...] Outsourcing Your User Research Is Like Outsourcing Your Vacation By Jared Spool, User Interface Engineering [...]

  7. Ruth E Haworth Says:

    Very very well said! I agree completely… I have been saying this for years and I appreciate having a catchy quote to use. In fact I’d make it a little pithier: “Outsourcing user research is like outsourcing your vacation. It gets the job done, but won’t have the effects you were seeking.”

    Having said that, this issue exists not just for outsourcing but also for researchers within organizations. It’s not easy to translate research results into behavioral change. I regularly have the problem of reports that don’t get read, presentations that are quickly forgotten, etc.

    The trick seems to be (as you say) to involve the people you aim to change. Exposing them to users is a big part of it, but I think we also need to involve them in posing questions and creating a loop in which they see what users want them to do and then see the effect of doing it.

    Really effective graphics are also a great boost in getting across both qualitative and quantitative data. I’m struggling with that one.

  8. anonymous Says:

    Nicely said, and as a 12 yr veteran in the field, I mostly agree with you. However I think the crux of the issue is not so much exposure to the customer, it’s much more insidious. You said it, actually

    “…most importantly, their culture didn’t change — they weren’t integrating users into their design process any more than before.”

    Incorporating user feedback, and understanding your user base are not enough to make great software. A team has to have a user-centered design and development process. Every individual in the product development cycle has to understand what it means to think about the problem in a user centered way – i.e., who is this product for? What problems is it trying to solve? How can we prioritize these problems so that we can solve the most important ones effectively, efficiently. Most important, I find that the business owners – the people who will actually determine how your product makes money – must clearly understand the user and what will motivate that user to part with their money.

    As long as product development teams are rewarded for the number of features they build, the speed with which they build them, and the ability to put a product in the market that meets some bare minimum, they will not be able to incorporate user insight no matter how good it is.

  9. Eddie James Says:

    Ditto what Dave Malouf said. Every one of his points rings true.

    One area that UXers lack in is getting stakeholders to want to show up to observe research sessions. I don’t think we always position our research in a way that makes stakeholders get excited.

    Stakeholders need to know what’s in it for them. If we can sell the importance of observing users AND make it easy for them to observe AND make them want to give up time in their day to observe, then our research findings will have more impact.

    In my experience, the folks observing, and what they walk away understanding from observing users, often has more impact on the project than my final report. And their observations often impact future projects.

    I try to encourage stakeholders to observe by:
    1) always having a live stream of the sessions available to anyone who can’t be there in person.
    2) inviting important stakeholders individually, telling them what I think they will find valuable. I also cc their assistants (and stop by their desks), who often control the calendar for many busy execs.
    3) Offering a highlights video just for stakeholders
    4) Making reports as short as possible
    5) Once they are in the room observing, I make sure that they understand who the participant is and what we are trying to find out. Anything I can do to engage them I do. I don’t want them doing work on their laptops during a session or playing with their phone.

    Making a special invitation to each important stakeholder has yielded the best results for me. I usually find even the most high up stakeholder will at least send one of their direct reports to view the sessions live — if I’ve made a good enough case that it’s worth their time.

    If anyone has more tips for engaging stakeholders, I’d love to hear about it.

    Thanks Jared for writing this. I totally agree with diy usability research, but outsourcing definitely has it’s place and innie research has its challenges.

Add a Comment