A friend recently posed this question to me and some collegues and suffice to say, I don’t think there is any real data out there that talks about how to figure out the ROI of hiring a coach.
I have my own thoughts and it comes down to the reason(s) why you’re hiring a coach or consulting firm to help you through your Agile Adoption or Agile Transformation. Here’s methods I like to suggest.
Let’s assume you’re bringing in Agile because you want “better, faster, cheaper”. Typically that’s my experience, management isn’t happy with what’s being produced, customers aren’t happy or the organization is in a whole pile of trouble and at risk of going under.
Let’s start by looking at some of the problems with figuring out the ROI of hiring a coach for your Agile transformation:
- Any metric would be a lagging indicator: change takes time and you are not going to see measurable business results within days, weeks or even months depending on the size of your company.
- Progress will actually slow: the coach bringing in the change can often be seen as the one who’s creating the problems. A good coach will explain this in advance.
- It’s not only the coach/consultant’s cost: a coach needs time from your people, that takes them away from their day jobs in order to learn new skills. I guess that’s the reason why progress is slow but the coach/consultant’s cost is one piece. Chances are you will need some tools at some point that align with a new way of working plus other training. The most significant cost is your time.
Before I go into some methods I think are useful, here’s a few measurements to avoid:
- Team velocity: Velocity is not a measure of productivity and is easily gamed. To set a metric of “let’s increase velocity by 30%” is ridiculous, don’t do it. A team will inflate their points sub-consciously, I’ve seen it happen and I actually tried it once, big mistake.
- Any other ‘Throughput” increase: If you’re producing the wrong thing it doesn’t matter if you produce more of it. Agile isn’t a code word for “produce crap faster”.
- Any metric associated with time tracking: Isn’t it amazing that when you track time people work exactly 37.5 hours per week? I once had an executive tell me they tracked time to know when it was necessary to hire more support people. Seriously?
- Number of Defects: Any type of defect or QA metric is not helpful and easily gamed. “In process” defects don’t matter. What matters is how many times you get a phone call or email for training or support.
Here are some alternatives:
- Net Promotor Score: At the end of the day, what matters is whether or not customers will recommend your companies product or services. It’s a difficult metric to game and it focuses on customer value. The problem is, like any metric to measure the ROI of a coach, is that it’s a lagging indicator. You aren’t going to see the affect of this for months. In a large organization you can still use Net Promotor Score (I use it for personal performance reviews) to see how likely your business stakeholders would recommend IT to deliver software for them.
- Cost per Call: If you have a consumer facing product or any product that has a helpdesk, you should be lowering your cost per call and reducing the number of calls to your helpdesk because a coach is going to help you figure out how to make better software. Ideally it has less defects, better usability (meaning less time spent training customers) and actually solves the problems customers have.
- Escaped Defects: Customer reported incidents matter. Customer should be reporting less problems.
- Increased Sales: Better software means more sales although the danger with this one is that correlation isn’t necessarily causation. Increased sales could be the result of a market shift, a marketing campaign or other factors.
- Failure Demand vs Value Demand: ideally you’ll be spending less time on re-work (defects and support problems) and more time on valuable features. The flip side is that it’s easy to start labelling “defects” as “features” to game this metric.
From my experience, measuring ROI for a coach is difficult, but not impossible. It does imply a level of trust between you (the customer) and the coach because in shorter term engagements you’re not likely to see tangible, bottom-line business results until the coach is gone. Again, this assumes a few months of engaging with a coach in a medium-ish sized organization. There’s many more scenarios and engagement options that would make this post go on far too long.
Measuring progress, however, is a different story.
Before engaging with a coach try:
- Gallup’s 12 Questions: measure your employee engagement before the coach and after the coach. Ideally an Agile Transformation improves the quality of work-life for people, challenges them with learning new skills and fosters trust and collaboration.
- Happiness Index: Is there less stress in the air? Are people happier at work? Happy people are productive people.
- Behaviours and Gamification: We are using character sheets and gamification ideas to energize people around building their skills, people self-measure based on behaviours we typically see when bringing in Agile/Lean practices. For example, we’ll see collaboration happening between departments that typically didn’t talk to each other at all before the transformation. For more, check out this post from Jeff Anderson….an hopefully soon my experiment with an online behaviour tool. We were pushing this but have since changed tactics.
These are only a few, there are more such as lower in-process defects. Yes, I suggest not tracking them because they are not valuable to your customer and it’s easily gamed but ideally your testers will report less defects. Once you understand why in-process defects are a useless metric you’ve realized your metric of progress! Get it?
The most important point I wanted to make was that your measurements of your transformation should evolve over time. Initially you’ll want to collect process-related metrics like throughput, cycle time etc and that will help you find what your other problems are and eventually you’ll discover the right business metrics to determine the ROI of hiring a coach or consulting firm.
What are your thoughts about measuring your transformation or ROI for hiring a coach?