This process started to break down when they began to hire engineers at a faster clip. Whereas in the past there was time to bring an engineer up to speed, now the team began to let new eng hires work on features due to pressures to fix bugs and get features out the door. These not-yet-up-to-speed engineers became responsible for many of the kinds of bugs they were hired to help fix. Fixing bugs instead of working on features meant more new engineers needed to get hired. Rinse and repeat.

Not being on the engineering team, I was able to view this from the sidelines and noticed that net productivity was nearly the same with almost double the team!

I think this approach to development works well with small and well managed groups. At this job, the team had grown too big too soon with no growth in engineering management.

Can you tell me what some of the best tools are to track the effects of a redesign. Obviously, I want to focus on conversion rates and watch where people click, but are there any more specific tests etc?

I myself am a (Cooperian) Interaction Designer in the field of web applications with a background in webprogramming. But I find myself developing ever greater interest for the iterative development process.

I have been reading more and more about this way of developing. One thing I tend to not find in these (mostely) success stories is an explanation of how these companies guard their code from ’scar tissue’ because of the many changes (also see: http://www.fawcette.com/interviews/beck_cooper/default.asp).

Do any of you have more information about this because culture and attitude is one thing, but keeping your code healthy is another.

One question that comes to mind as I read the excellent UIE article The Freedom of Fast Iterations: How Netflix Designs a Winning Web Site the question comes to mind, what would have Netflix looked like if a big interactive agency had been involved? I …

On2Biz is a living breathing application. It is permanantly “under construction”. We keep adding and modifying features that help users improve their comfort with the tools. And the process never ends. Compared to traditional product develo…

I wonder if you can elaborate on how you specifically measure the success or failure of a change and how that fits into the design cycle.

It has one of the largest observation spaces I’ve ever seen, with huge plasma displays for everyone to see what’s going on. Apparently, there are so many people interested in observing that it frequently fills up.

Do you know if they do traditional user research or usability testing? I’m exploring whether some of these successful companies do so - I’m guessing no (I know you have looked at this before).

Recognizing this, Red Hat has a number of initiatives underway that attempt to combine a fast-iteration, user-centric, prototype-driven “design thinking” approach with the open source development method. Red Hat’s open community site Mugshot (mugshot.org) is developed using a design thinking approach — see developer.mugshot.org for background. Design Thinking and its applicability to open source was also featured in a recent edition of the online Red Hat Magazine.

I just saw Bill Buxton give an inspiring closing keynote at the Computer Supported Cooperative Work conference, in which he shared some related observations:
“The worst thing in the world is a precious idea. The worst person to have on your team is someone who thinks his idea is precious. Good ideas are cheap, they are not precious. The key is not to come up with ideas but to cultivate the adoption of ideas.”

(more notes from Bill’s talk can be found here … near the bottom)

Unwieldy legacy systems? Not to be flip, but if you are afraid of changing your code, any of it, you are already dead. I’ve seen it happen too many times.

Do fast iterations work better with smaller, skilled, and general teams? It is easier with small teams. Skilled people keep teams small. Generalists have nothing to do with iterations, though it may be helpful to have one around to cover holes in your expertise (see Jared’s work on teams and desgn for a scary list of useful skills).

Outsourcing? I would only outsource slowly-changing parts of the system. That doesn’t mean you write everything. Use libraries and third-party stuff like crazy.

What about “good” software development practice? Netflix handles credit cards, so I think you have your answer. Good process fits the business requirements. A practice-based process, like many agile ones, may be a better fit than a milestone-based process. Test-driven development is another process than can work well. Some things don’t change — you must know exactly what code you have running on the site.

Where I am now, the QA people sit with the developers. We are actually in alternating offices, QA, HTML production engineer, QA, Java developer, but I don’t think that is by design.

Don’t people hate it if you change the pages every two weeks? Of course. If you have enough traffic, test the changes on a slice of users. If you don’t have a lot of traffic, evolve the site, change one part of it, use visual changes to signal functional changes.

This stuff isn’t new. We had a two-week push cycle at Infoseek nine years ago. Also, think of much shorter cycles than two weeks. For an emergency push (site down, security vulnerability) the whole process might run in a few hours. With full SarbOx.

Fast iterations would probably not work well with an unwieldy legacy system. In fact, most processes probably won’t work well with an unwieldy legacy system…

Netflix doesn’t have a particularly small team. They do have different design arms concentrated on different parts of the site, so they may act as small teams as necessary. Some changes, however, are bigger than a single team. In general, however, it doesn’t take more than a few people to make a new feature (or tweak an older one) for testing.

Outsourcing would definitely make this a bigger challenge than not, because there would be more communication necessary.

Certainly, some apps should change more than others. As I hinted in the article, the designers at Netflix aren’t satisfied with their already amazing site, and want to continually improve it. Other businesses would have different constraints…especially if their website isn’t the only conduit to success as it is with Netflix.

Enjoyed the article and the approach sounds great…
however I have some questions:
- How would small fast iterations work if Netflix had an unwieldy legacy system to interface with?
- Does the concept of small fast iterations work better with smaller, more highly skilled and generalist teams?
- I assume having an internal team with control over all aspects of the technology is essential? Outsourcing aspects of the workflow to third parties is probably less likely to work…?
- What about “good” software development practice? How do the QA folks get on with this approach? (If we changed the credit card payment page or the search results page every 2 weeks where I work, we wouldn’t be the favourite people that we are!)

Perhaps I need to drop some of my pre-conceptions about how best to manage this web stuff but appreciate your thoughts on this.

All the best
DJ

You can track when and how often Netflix updates their site using the Internet Archive Wayback Machine

It appears that the fast iterations really started cranking out from October 1st 2005.

Cheers, Rich.