Managing Multiple Team Releases

Agile42 posted a fantastic case study about the use of SAFe with one of their clients recently. I’m no fan of SAFe, but I recognize some big organizations need a “thing” to help them deal with the uncertainty of scaling Agile.

This post is designed to show an alternative to opening up the SAFe cookbook. Namely, open up the Stone Soup cookbook.

Context: 600 person organization, 5 core teams in IT consisting of ~140 people. The problem these teams was facing was that one team’s changes would break another team’s changes because of tight coupling. They would usually find out after a production release or way to late in the development cycle to do anything about it…other than to stay up all night and weekend fixing stuff.

I was talking to the release manager and suggested we implement a release wall to highlight the problem. Sorry for the horrible quality picture, but this was 2010 when all I had was the lowly iPhone 3.

release-wall

  • Each swimlane represented a team
  • Across the top of the wall we displayed the next 3 months of releases (major, minor, patches) This organization had EA (Early Adopter releases) and GA (General Availability releases) so they had the ability to do the equivalent of what is known today as feature toggling.
  • On the right of the release wall was a production support/maintenance chart so we could see the impact each release was having
  • Our fancy build siren notified people when the build was broken and the release team responsible would lock out the ability to check in until it was fixed.

How this wall worked:

  • At the start of a release, the release manager (SAFe’s release train engineer I suppose) would get the team lead’s and managers together and populate the BIG GREEN stickies for the next major. These stickies simply showed the major features being delivered. Here they would discuss how to co-ordinate work between the teams
  • When necessary, the release manager would bring people up to the wall to discuss minor releases and patches.
  • The release manager owned the wall and nagged people to keep it updated.
  • Sometimes we’d do a ‘scrum of scrums’ in front of the wall, but not all the time

How it helped:

  • It helped all the teams understand what was being worked on so they could fix problems sooner
  • We measured the impact of the implementation of the wall in terms of value versus failure demand. We saw a 15% reduction in failure demand which wasn’t necessarily attributed to the wall or Agile…what mattered more was that people ‘felt’ it was more effective to manage complex releases this way.

Other Things We Did

  • implemented Sonar and put some ‘heath’ measurements in place (cyclomatic complexity, code duplication etc)
  • Socialized the wall with teams
  • Implemented a quarterly improvement initiative and measured results (we were accomplishing 20% of the improvements each quarter…for 2 quarters at least, I was there for 8 months)
  • Some teams were using Scrum, others Kanban

What We Didn’t Do

  • We didn’t create feature teams, they were still grouped by application or application area (IE: administration tools versus client facing data entry applications)
  • Use any “agile scaling technique”, we simply facilitated conversations in front of a wall
  • Implement an Agile Release Train…because they already had a regular cadence of releases. Majors were quarterly, minors were monthly and patches were daily when needed.

I presented this at Lean 2011, here are the details

 

And the other presentation…

Another Case Study

In another organization, we implemented a smaller-scale version of this release wall:
release-wall2
Left-most column: The application being released
Across the top: Dates for each release, this organization scheduled releases every weekend. They wouldn’t release every weekend, but they did all their production pushes on the weekend.
Colours: Blue tickets were non-project related work, Green tickets were project related work
Flags: Red flags meant something done got busted! Or it was late or not implemented.
The IT Change Manager owned this wall. After our Monday enterprise standup, she would ask anyone involved in the release that happened on the weekend to stick around and update the wall. She would then ask anyone to update her on what releases were coming over the next week or two.
What Worked
  • 3 times a week we’d run enterprise standup meetings to talk about issues and risks
  • weekly release co-ordination in front of the wall
What We Didn’t Do
  • implement “scaled agile anything”
  • implement any new tooling (we tried, but none of it stuck)

The Moral of the Story

SAFe isn’t a bad thing. I prefer DAD myself because, to me, SAFe is for cooks and DAD is for chefs but, to each their own!

That said, you don’t need an Agile scaling framework to co-ordinate work in big organizations. 

It’s more important to figure out how to do scaled facilitation before scaled Agile. Don’t underestimate the power of well-facilitated conversations in front of a big visible wall.

It’s a low effort, possibly high-return experiment, what’s the worst that could happen?