Productivity Metrics

02 Sep, 2005

A few days ago, on the XP mailing list, "PierG" mentioned that Mary Poppendieck's book Lean Software Development recommends software product revenues as the best metric for software teams. He went on to say:

In my opinion this can be a very good metric for the product team (dev team + marketing + sales ...) but ... what if the dev team has developed 'running tested features' respecting time, costs and quality ... and the problem is the perception of the market of the 'internal' customer (product marketing, sales .. or so)?? Does poor revenues mean that the dev team has bad performed? Whell ok, it didn't produce 'value' ($) but ... it produced the value he was asked for.

I agree with Mary 100%. I've been looking for a good metric of software productivity for some time. All of the ones I've found are subject to gaming. There's the infamous lines of code, feature points, stories, velocity--none of these actually help us measure productivity. That's because

productivity = features / time

...and we don't have an objective measurement of "features." What is a feature, anyway? How do you define it? A feature is something that the business values. It can be huge or it can be teensy. There's no objective comparison that I've found, and believe me, I've looked. I'm no expert in this area, so I could have missed something (I keep waiting for somebody to tell me I'm wrong about feature points), but so far, no luck.

Okay, so let's switch gears. We can't measure features / time. Why do we want to? Well, so we can measure the productivity of the software developers. Presumably so we can jump around and (a) crow about how cool we are, or (b) fire those lazy bastards.

Wait a second. That's not very agile. Why would we want to measure just the programmers? Aren't the testers part of the team? Aren't the business experts? What about that guy who developed a 500-line program that resulted in millions of dollars of sales just because it copied order status from a mainframe to a webpage? What about the team that hired an army of temps to do data entry rather than writing a program, and saved the company hundreds of thousands and three months of programming effort?

The bottom line is... the bottom line. That's the only objective measure of productivity I've found.

productivity = $ / time

But wait! We don't want to punish the programmers if they have a lousy business expert on the team! The business expert has a much bigger impact on this metric than the programmers do!

...oh, really? Huh. THAT'S interesting. What does it tell us about the important things for a programming team to do?

In the discussion that followed this post, many people came back to the question of not wanting to be punished for bad performance by poor marketing. I can understand this point. But... it doesn't matter how good you are at producing quality software on time and on budget if it's not selling. In fact, I would say that "on time" and "on budget" are totally irrelevant. Who cares if the software is a month late if it has twice as many sales? Conversely, who cares if it's a month early if nobody buys it? As they say in the games industry, nobody cares that Half-Life was a year (or two) late--they just care that it's a kick-ass game.

You may ask, "what can the software team do if they don't get good requirements?" There's lots you can do! Break down the chinese wall. Create interim releases and give it to friendly customers. Create a paper prototypes and run them past users. Try out the software and see what people think.

In the Lean Software Development course the Poppendiecks put on last week, they talked about Toyota's use of "Chief Engineers." Toyota's chief engineers are responsible for the success of the product (vehicle). They lead in product vision and in design vision. Mary told one memorable story (which I may be mangling, but it's close enough) about a chief engineer for an minivan who drove 30,000 miles around the US in order to understand his market better. He came back and told his team that the minivan had to fit a 4x8 sheet of plywood/sheet metal/whatever in the back. They scoffed. "It's not possible!" So he pulled out a video he had recorded showing clip after clip of Americans driving up to Home Depot loading a sheet of 4x8 plywood into the competitors' cars, and driving away. You can bet that the next year's model could hold a 4x8 sheet of plywood.

Now that's taking responsibility for the product.

If you can't do these kinds of things--I'm not talking about the chief engineer bit, which is a big change for some companies, but the simpler things like creating interim releases and trying out paper prototypes--if you can't do these things, then your team won't do well on the $ metric. But that's okay... companies like that won't put it into place anyway. They're too concerned with schedule/scope/budget to think about $. If they do put it into place, you'll have a good excuse to start breaking down those walls. And that would be a good thing... not just for your team, but for the quality of your software and the success of your company.