jay

You can get whatever results that you want from usability testing - simply by the scenarios you write. And you can turn around and say - wow - we had these fabulous results and our interface rocks. And maybe it doesn’t.

Let me make an analogy - I have done electrical work in my house and have managed not to electrocute myself. However, I wouldn’t go out and call myself an electrician by any means. I am not discounting those individuals who are out there trying to make a difference. But some of these issues are due to exactly this.

Likewise, individual skill and talent is not primarily what differentiates good artists and their art, I don’t believe. They must all be technical masters, and understand what they can and cannot do with their medium - that is craft. It is each artists’ unique preferences in what they want to *produce* with their craft skills that differentiates them, and elevates one above another, I believe.

If Jared’s asking whether usability testing is art or science, I’m inclined to say that yes, it is science - but it’s a soft science, just like literary or music criticism. The problem for us as people interested in usability is to know into which canon it fits so that we can know which tools we are using to work with it. Donald Norman and Alan Kay (to take two names off the top off my head) are scientists, and their work is scientific. I say that our work is science too - the science of good design.

So I’ll try to add something a little different.

Usability is inherently multidisciplinary: the art and the science are two ends of the same stick. We need qual and quant methods. Quant methods are generally taught in a scientific setting, and ‘look’ very scientific and rigorous. But when we seek to understand a participant’s mental model which even they are fully conscious of - we have use empathy, intuition, and inference.

So, my experience - having practiced and taught usability in universities and corporations - is that we improve our skills by enjoying both the artistic and scientific side of usability. Letting both rip! Find the methods that turn us on and come easily to us, and then try to develop the one’s that don’t.

In my case, I love the loosely scripted interview, and relying heavily on my sense of empathy. I can get that to work for me quite easily. I have to work harder on putting together testing scripts that will be reusable over a whole series of tests. But both types of activity support each other.

I think what was missing from Rolf’s landmark study was a follow-up to see what resulted from any implemented changes (i.e. could it be proven that one company’s high priority proven a big issue, and another’s proven less impactful?). And is there any way to do this based on method alone, or does the skillset of the test team members play a more significant role than the methodology?