Metrics and Money

Thursday, February 26, 2009 2:04 PM

I've been trying to stick together a few notes about what I've learnt about process design and implementation over the years.  The "short notes" just keep getting longer and longer.  However, this story is worth repeating and should illustrates one rule that everyone should know by now is: never link a metric to cash.  It seems like the most sensible thing in the world to give out bonuses on the basis of deliverables.  It's not, it will damage your department and your firm.  If you're lucky, you'll know how.  If you're unlucky, you'll never find out.

Let me give an example, a friend of mine worked at this firm.  We'll call him Joseph Developer.  Management, having finally ditched performance-based metrics a year previously (for reasons that would make a good post of their own) had decided that it was time for re-introduction.  With a new and popular CTO keen to make his mark, they sat down and came up with a new plan.

OK, I can hardly tell this story without laughing.  I'm actually laughing right now typing this.  I can assure you that it wasn't as funny for anyone directly affected by this.  Let me just remind you that this was the best plan the CTO, the COO, the CFO and CEO locked up in a room could come up with.  This was what they pitched to the developers.

  • They were concerned about the quality of their systems.
  • So they thought about linking bonuses to the number of bugs assigned to someone.
  • But they knew that didn't work.
  • So, rather than that, they decided to target the number of re-opened bugs.

I can pretty much pinpoint that as the point were some people's nagging suspicion that the CTO wasn't as good as they hoped transformed into an unshakeable certainty he didn't have a clue.  Here's what they'd done: they'd taken a system they knew didn't work, taken the first derivative and decided to implement that.

Labouring the point

Clearly, this announcement would never have occurred if the managers had actually understood the problem.  So let's talk for a bit about why you shouldn't incentivize people on the basis of bug count.

  • What is a bug and what isn't is ill defined.  That isn't a problem until you start making dollar amounts (which are about as concrete as you can get) dependent on the distinction.
  • You've just created an adversarial relationship between the bug reporter and the developer.  These can develop anyway, but they're never productive.  By putting money on the line, you've guaranteed it.
  • You've re-incentivized people to work in a negative, rather than a positive way.
  • Your developers will figure out a way to game the system.

The first two points look like they're unique to bug tracking, but they're not.  All of the metrics that you've got are indicators of what's going on, not the unvarnished truth.  All that the bug metric tells you is that a certain number of bugs have been entered into your system.  It's not a measure of quality and it sure as hell isn't a measure of productivity.  That's not just true of bugs, it's true of the actual P&L of your company.  Don't believe me?  Ask an accountant.  Or a market analyst who has to decode earnings announcements.  Not even the bottom line is the bottom line.

People get misled by examples from manufacturing and construction, in which well-defined metrics produce well-defined outcomes.  Go and read that Joel article again.  I'd go a bit further and say that such incentives do work in very limited circumstances: where you want the guy to do that and only that.  My old project manager once killed a round of testing simply because it meant he would hit his bonus targets.  Fixing bugs would not.  Sales commissions are brilliant at getting salesmen to sell.  You'd better keep a fairly tight eye on exactly what they're selling, though.

The adversarial point is equally general.  You've replaced a metric which helps you tell what's going on with a salary negotiation mechanic.  I hope you're not planning on that metric being used for anything else.  Such as, for instance, bug tracking.  I think there have been enough options scandals by now to emphasize that this is equally true of earnings numbers.

The final point is the one that should make you pause, even if the others didn't.  Let's take a look at our four guys locked up in a room.  They were employing more than 50 developers.  Bright as they were, they weren't brighter than 50 guys with degrees and training in a profession that emphasizes logical thinking.  Actually, it doesn't even matter that they were bright, anyone could have come up with a way of circumventing it.  Since you're paying them to do so, you can be reasonably guaranteed they will.

So how many of those objections didn't apply to the "penalize re-opened bugs"?  None of them.

  • What was an unfixed bug and what was a separate issue was ill-defined.
  • They made a fairly adversarial relationship between QA and development ten times worse. 
  • People regularly worked in a negative way.  Often, more time was sent arguing about the exact status of a bug than fixing it.

And as for gaming the system, well, that's where the fun really began.

Gaming the system

When I was first told of this, it took me five seconds flat to figure out how to game it.  Just fix easy bugs.  If a label's wrong, fix it.  Avoid anything involving a nasty interaction or an ill-defined behaviour.  Never mind that those are where the value is.  In fact, the truly pernicious thing about this whole process was that it actively penalized senior, responsible developers who took on hard problems.  Now, as I say, I was lucky, I didn't have to put up with such a stupid system.  My friend did.

So, I met up with Joseph a couple of months after this had been put in place and asked him how the firm was going.  He told me that the atmosphere was very negative (not solely caused by this decision) and that he'd already given up on getting a bonus this year.  He was two months into the bonus cycle.  This was an extremely talented and conscientious guy, regarded as a star at the firm.  And he'd given up on getting a bonus, simply because he was behaving like a responsible developer and not gaming the system.

Two months later, he left and joined a much better firm.  Yes, eventually he figured out a way to circumvent the policy he was happy with: he quit.  He wasn't the only one.  And, for the reasons I've already outlined, it was the best staff who jumped.

Now, management at this firm clearly forgot Evil Overlord Rule #12, but just because the example's extreme it doesn't mean that the point isn't general.  Incentive structures distort behaviour and de-motivate staff.  Metrics-based incentive structures distort metrics as well.  At my current workplace, I receive a bonus based on how well the firm did and how well my managers think I did.  Yes, I still have targets, and I still try to hit them, but I don't let my targets interfere with serving the business.

So how's Joe now?  He's still at his new firm and very happy.  I tried to tempt to a job that would pay significantly better, but he's not interested.  Now, that's a firm that knows how to motivate its staff.

Technorati Tags: ,,
Comments
No comments posted yet.
Something to add?

Talking sense? Talking rubbish? Something I'm missing? Let me know!

Fields denoted with a "*" are required.

 (will not be displayed)

 
Please add 7 and 7 and type the answer here:

Preview Your Comment