Measuring the Productivity of Designers
November 24th, 2006
Over at the SIGIA discussion list, the always interesting Ziya posted a link to a post by Joel Spolsky that discussed Joel’s experience with measuring the productivity of programmers.
Joel doesn’t think too highly of the process, saying
Software organizations tend to reward programmers who (a) write lots of code and (b) fix lots of bugs. The best way to get ahead in an organization like this is to check in lots of buggy code and fix it all, rather than taking the extra time to get it right in the first place. When you try to fix this problem by penalizing programmers for creating bugs, you create a perverse incentive for them to hide their bugs or not tell the testers about new code they wrote in hopes that fewer bugs will be found. You can’t win.
Joel’s point, a good one, is when you measure the productivity of individuals, you have to be careful of gaming. He implies that you can’t build a measurement system for programmers that can’t be gamed.
That may be true, however, I’m not sure gaming is always a bad thing.
Part of the problem with the measures Joel spoke of was they ignored any valuation on quality. They focused on function points and lines of code, two easy-to-measure quantities. Code quality is not so easy to quantify, so we tend to not try to measure them. (After all, who wants to get into arguments on whether something is quality or not.)
Someone really smart (not me and I don’t remember who) once said, “Once you remove ‘quality’ as a requirement, everything gets a lot easier to build.” Well, once you remove ‘quality’ as a measure of success, then the measures become a lot easier, but they don’t necessarily measure what you’re looking for.
In Ziya’s post, he posited the same problem would occur when designers try to measure their own productivity. After all, what do we measure our design team on?
Maybe I’m an optimist, but I believe it’s possible we can arrive at productivity measures that work (meaning, when gamed, they produce the results we were hoping for) for design teams. These measures would quantify the results the team produced and compare them to the effort it took to produce it.
From there, we could determine what practices and skills are most efficient at gaining the best results. We can also use these measures to help us make budgets and predict timelines.
Right now, we probably don’t know enough to put these measures together. But, just because we can’t effectively measure it right now doesn’t mean it’s impossible. History has shown many examples of things which took a long time to learn how to measure, but someone eventually does.
How would you measure the productivity of your designers?
November 26th, 2006 at 2:41 pm
I’d start with defining the design deliverables and the customer’s success criteria, so you have “stuff” to measure.
IF you know how long it took a team to deliver a design using a certain set of deliverables AND you know how happy the customer(*) was with the end-result THEN you have a situation where analysis is possible:
- did we use the right deliverables?
- why did we use more time than average?
- did we overpromise or underdeliver?
Of course this requires, as Jared said, a long time to measure; you will need to do similar projects that require similar deliverables and aim for similar client goals to have a set of comparable figures.
Hey, maybe we could set up a set of design process standards(**) for this!
(*) yes, the customer’s client should also be happy, but it’s up to the client (and his consultants, which may be you) to determine how to measure this.
(**) Read more about this idea here:
Adding design process attributes to patterns
http://www.peterboersma.com/blog/2006/08/adding-design-process-attributes-to.html
November 27th, 2006 at 12:38 pm
The problem with Peter’s suggestion (and he hints at it) is that deliverables will (almost) always vary in breadth and complexity. That’s where quantifying the design deliverables even in a fuzzy manner might help: “Wireframe with 10 controls and Level-2 complexity.” And yes, you would have to add in a measure of quality or else you wind up in the same boat as developers with function points.
December 4th, 2006 at 10:56 am
To overcome breadth and complexity, we could start with comparing the quality of parts of the design process.
Currently I started with measuring ideation effectiveness. How effective was a certain idea-generation session? The answer would help at least in the beginning of a design session.
My research focuses on working with children, but I have based my research setup on an article by Shah and Vargaz-Hernandez in Design Studies 24 (March 2003). They propose to measure the effectiveness by the novelty, variety, quality and quantity of ideas produced in a design session. It feels like demystifying creativity, inspiration and quality terms of the like. In short setting the success and weighing criteria for quality and the importance factors for the variety of ideas to generate is the most painful exercise. But in the end, I am able to say something about the kind of method that delivers ideas most effectively with 10-year olds.
I can imagine someone could perform this exercise in designing with adults. I am curious to learn what you think of the article and their approach.