Keeping Honest with Metrics

Last weekend I purchased the Nike + iPod device. It's a small sensor that fits into my running shoe plus an interface unit that plugs into my iPod. It keeps track of how far I've run, my pace, even calories burned. It's really pretty effective and accurate.

What the tool is really doing for me is keeping me honest. I used to go out and run for 45 minutes or an hour and not worry about the pace. "I'm going around 8:00 minute/mile pace" is what I would tell myself, knowing a lot of times I was really going slower. With my new toy, I know exactly what pace I run. The end result is I am running harder so that I do keep at 8:00 minute pace or better. The proof is here.

This same theory applies to managing business processes. Maybe you know it takes about 5 business days to do end of month processing and get your invoices out the door and your customers take around 30 days to pay the invoices. Is this good? If it used to take you 8 days to do invoices, this is an improvement, but do you really have the metrics to make good decisions? Should you work to improve this process, or is there another process that would be more important to focus on?

However, let's say you knew that by cutting your invoice processing to 4 days and getting your money into the bank sooner, you would earn another $21,000 in interest a month. Now there seems to be more value in capturing detailed metrics on your process and using those to optimize the process.

Like my running, businesses need good metrics in order to run the business effectively. In my case, the results will be in my race times, not increased profits, strong stock price, or increased customer satisfaction.

If a tree falls in the woods...

I came across the Copenhagen Interpretation in a book I'm reading. This idea came about in quantum mechanics by Niels Bohr, Werner Heisenberg and others, working in Copenhagen in the early 20th century. In simple terms, the Copenhagen Interpretation says that something doesn't exist until we observe it. So if a tree falls in the woods and no one is there to hear it, it doesn't make a sound.

So what are we trying to observe on our projects? What do we try to capture by our reports, metrics, SLAs etc? The Copenhagen Interpretation supports the idea that we get what we are looking for.

I was doing work at a call center one time. There, they were looking at what percent of time the representatives were available to take calls and when they were "logged out" meaning they were taking a break, doing documentation from a previous call etc. This data point was pretty consistent across the representatives. However, when they dug a little deeper, they found a small set of people that answered significantly fewer calls. This didn't make sense since their availability was the same as others.

As it turns out, some representatives figured out how to scam the system. They watched the representatives around them getting calls and could predict when they would get a call. They logged out for a few seconds, the call went to the next representative over, and they logged back as available, knowing they wouldn't get another call sent to them for a while.

So you're going to get the behavior you look for and reward. If you reward the number of bugs the coders find, they'll find a lot of bugs. If you recognize the person that can fix the crisis, you'll get a lot of fires that need to be fought. So don't think lightly about what you measure.

Metrics

I was working with a new client yesterday and heard about an unusual metric they are tracking on their project.

They have a running tote board showing the names of the team members. They are tracking the number of negative comments that are made be everyone; things like sarcastic wisecracks, cynical comments or pessimistic opinions. They want to track if this metric goes up if the project becomes challenged.

Like any metric, the key will be what they do with it to manage and change behavior. If they see the number go up on a weekly or monthly basis, will they take action to bring the metric back down? This could be a good indicator it's time for a team building activity or some other break in the routine.

Measuring Success

I received a $30 gift certificate in the mail this week for taking 5th in my age group in a race I did this spring. I was surprised, because it was a big race (around 10,000 participants including walkers) and I didn't think I would do that well. This got me thinking about how we measure success.

In project management, it's always scope, schedule, and budget that determines if a project is successful. But a good project manager knows that isn't enough. The customer also has to be satisfied, but how do you measure that?

One approach I've taken for some time is to define the measures of success as the project is going through the chartering process. This is where key metrics can be identified that can later be used to measure if the project has delivered. In process improvement, the measure can be reducing the number of defects or decreasing the time to complete a process.

So as you're starting off your project, think about all the goals the project should accomplish and how to measure them. With running, my goal is to finish in the top 5% of any race. Anything else is a bonus.