Notice the black line? That's flow!

Can You Measure Your Agile Transformation?

A friend recently posed this question to me and some collegues and suffice to say, I don’t think there is any real data out there that talks about how to figure out the ROI of hiring a coach.

I have my own thoughts and it comes down to the reason(s) why you’re hiring a coach or consulting firm to help you through your Agile Adoption or Agile Transformation.   Here’s methods I like to suggest.

Let’s assume you’re bringing in Agile because you want “better, faster, cheaper”.   Typically that’s my experience, management isn’t happy with what’s being produced, customers aren’t happy or the organization is in a whole pile of trouble and at risk of going under.

Let’s start by looking at some of the problems with figuring out the ROI of hiring a coach for your Agile transformation:

  1. Any metric would be a lagging indicator: change takes time and you are not going to see measurable business results within days, weeks or even months depending on the size of your company.
  2. Progress will actually slow: the coach bringing in the change can often be seen as the one who’s creating the problems.  A good coach will explain this in advance.
  3. It’s not only the coach/consultant’s cost: a coach needs time from your people, that takes them away from their day jobs in order to learn new skills.  I guess that’s the reason why progress is slow but the coach/consultant’s cost is one piece.  Chances are you will need some tools at some point that align with a new way of working plus other training.  The most significant cost is your time.

Before I go into some methods I think are useful, here’s a few measurements to avoid:

  1. Team velocity: Velocity is not a measure of productivity and is easily gamed.  To set a metric of “let’s increase velocity by 30%” is ridiculous, don’t do it.  A team will inflate their points sub-consciously,  I’ve seen it happen and I actually tried it once, big mistake.
  2. Any other ‘Throughput” increase:  If you’re producing the wrong thing it doesn’t matter if you produce more of it.  Agile isn’t a code word for “produce crap faster”.
  3. Any metric associated with time tracking:  Isn’t it amazing that when you track time people work exactly 37.5 hours per week?  I once had an executive tell me they tracked time to know when it was necessary to hire more support people.  Seriously?
  4. Number of Defects: Any type of defect or QA metric is not helpful and easily gamed.  “In process” defects don’t matter. What matters is how many times you get a phone call or email for training or support.

Here are some alternatives:

  1. Net Promotor Score: At the end of the day, what matters is whether or not customers will recommend your companies product or services.  It’s a difficult metric to game and it focuses on customer value.  The problem is, like any metric to measure the ROI of a coach, is that it’s a lagging indicator.  You aren’t going to see the affect of this for months.  In a large organization you can still use Net Promotor Score (I use it for personal performance reviews) to see how likely your business stakeholders would recommend IT to deliver software for them.
  2. Cost per Call: If you have a consumer facing product or any product that has a helpdesk, you should be lowering your cost per call and reducing the number of calls to your helpdesk because a coach is going to help you figure out how to make better software.  Ideally it has less defects, better usability (meaning less time spent training customers) and actually solves the problems customers have.
  3. Escaped Defects:  Customer reported incidents matter.  Customer should be reporting less problems.
  4. Increased Sales:  Better software means more sales although the danger with this one is that correlation isn’t necessarily causation.  Increased sales could be the result of a market shift, a marketing campaign or other factors.
  5. Failure Demand vs Value Demand: ideally you’ll be spending less time on re-work (defects and support problems) and more time on valuable features.  The flip side is that it’s easy to start labelling “defects” as “features” to game this metric.

From my experience, measuring ROI for a coach is difficult, but not impossible.  It does imply a level of trust between you (the customer) and the coach because in shorter term engagements you’re not likely to see tangible, bottom-line business results until the coach is gone.  Again, this assumes a few months of engaging with a coach in a medium-ish sized organization.  There’s many more scenarios and engagement options that would make this post go on far too long.

Measuring progress, however, is a different story.

Before engaging with a coach try:

  1. Gallup’s 12 Questions: measure your employee engagement before the coach and after the coach.  Ideally an Agile Transformation improves the quality of work-life for people, challenges them with learning new skills and fosters trust and collaboration.
  2. Happiness Index:  Is there less stress in the air?  Are people happier at work?  Happy people are productive people.
  3. Behaviours and Gamification:  We are using character sheets and gamification ideas to energize people around building their skills, people self-measure based on behaviours we typically see when bringing in Agile/Lean practices.  For example, we’ll see collaboration happening between departments that typically didn’t talk to each other at all before the transformation.  For more, check out this post from Jeff Anderson….an hopefully soon my experiment with an online behaviour tool.  We were pushing this but have since changed tactics.

These are only a few, there are more such as lower in-process defects.  Yes, I suggest not tracking them because they are not valuable to your customer and it’s easily gamed but ideally your testers will report less defects.  Once you understand why in-process defects are a useless metric you’ve realized your metric of progress!  Get it?

The most important point I wanted to make was that your measurements of your transformation should evolve over time.  Initially you’ll want to collect process-related metrics like throughput, cycle time etc and that will help you find what your other problems are and eventually you’ll discover the right business metrics to determine the ROI of hiring a coach or consulting firm.

What are your thoughts about measuring your transformation or ROI for hiring a coach?

  • Anonymous

    Good post Jason.  As I am leading an agile trasnformation that is just ramping up I have been thinking about this quite a bit.  I really like your advice on what to avoid and your list of alternatives are almost exactly the same list I had come up with.  I was also looking at something like comparativeagility.com or some other sort of “skills” assessment to take on a regular basis just to track high level progress.  Do you have any advice for or against something like that?

  • Bill Schaller

    Jason this post brought back a real life story about a large corporation undergoing an agile transformation.  Teams were formed, given initial training and asked to operate within a scrum framework. I was asked to help the teams and found them mostly in disarray.  One month after the training an email comes out that the executive wants some tangible metric to show that the transformation was succeeding. Hmmm… I responded back with my observations from the ground floor but they were not impressed.  They decided to measure the time it took to fix a bug as their metric.

    I whole heartedly agree that the stakeholders must understand what to expect within an agile transformation.  However, as you point out, these quickly vaporize in light of problems and customer issues.  

    The effectiveness of an Agile Coach impacting the business bottom line is a long stretch to cover.  It depends on what was set out at the beginning.  If you walk into a dysfunctional situation, there is little you can do to measure success or failure.  A simpler situation would be to help the organization solve their continuous integration build/automation systems.  There are tangible metrics that you can put in place.  

    If the organization is responding well then they would be pushing themselves to expose the rocks-in-the-lake and have an understanding that exposing impediments is the right thing to do.  More importantly, stakeholders have an understanding of these impediments and are working them.  So one metric I would propose is to measure the organizations understanding and handling of their impediments.  As for the Agile Coach part of this — they have succeeded at developing a greater awareness and way to handle impediments.

  • http://www.agilecoach.ca Jason Little

    Hi Matt,

    I hadn’t seen that site before. I think it’s a good way to get some data back about how your organization stacks up. We are trying to Gamify our transformation by using character sheets for employees to track their own progress.

    I’ll post more about that later as we’ve switched focus a bit on the character sheets themselves, the cool thing is because this is such a large organization, there is an existing Learning and Performance group that is helping out.

  • http://www.agilecoach.ca Jason Little

    Yikes, time to fix a bug! Thanks for the reply, I like the tactic of finding a tangible metric that, while not effective for the org, can be effective at building a level of trust with management.