This is the final part of our Release Planning articles, visit Part 1, Part 2 and Part 3 for more information.
In Part 4 of Release Planning, we will go into the details for:
If you want to improve anything, then you must measure and observe where you are at, what is working and what is not. This is true whether you are growing corn in a field, or implementing an enterprise Software Application.
By identifying your Key Performance Indicators (KPI’s), you are better able to tell over time if your progress towards your ultimate goal is on track or not, and make adjustments as needed. Within a Software Development Project or Program there are several KPI’s that we typically use to track our progress and performance.
After you have Configured the Release and Epic Stories within your Agile Tool of choice and started your Sprints or Progress towards the Release, you will want to be monitoring your Release Burn-Down and Burn-Up Charts. Some Agile Tools combine these into one Report, while others have them separated into two different Reports. Either way, the Release Report is designed to show you 3 key perspectives:
To give you a very clear idea of what a Release Burn-Down / Burn-Up Charts look like, we are going to use Charts captured from Jira for an actual Client and Agile Development Team. We will cover what is working well and what needs more scrutiny or explanation.
This particular Agile Development Team tracks Epic Stories and Items based on an hourly estimate, rather than using Software Points. Because they do so, there will often be differences between the original estimate and the actual estimate, particularly if an item is shifted from one Team Member to another.
While analyzing these charts there are a couple of other caveats to keep in mind:
Observations
Pros:
Cons:
So, how do we solve this Problem and get better at our Release Estimated End Date? You have to look for Patterns in the Data over Time.
An Approach:
This is another way to show the same information over time (2 Week Sprints).
Regardless of which Chart Style you want to use, what is obvious is that we Underestimated the amount of Scope Change when the project first started. But now that we know this, we can better plan for the next Project / Release from this Product Management Group. The goal is that over time our Estimates become better and better.
Your Velocity Chart is one of the most important KPI’s for every Agile Software Development Team, because it not only lets you know how Productive they are, but also how Predictable the Team is. In addition, if the Team is fairly Predictable, then your Estimates will be more accurate in terms of how long it will take to complete a Release. In order for the Team to be fairly Predictable you want to see a long term Velocity Chart with only a slight variance from Sprint to Sprint. Ideally it should vary no more than 10% from the Moving Average.
However, if the Velocity Chart looks like a roller coaster, then you know this Team needs Help. And typically, it is because they are trying to do too much work, and allowing their Client to add a significant amount of Scope to the individual Sprint, without taking anything out. By doing so, the Team ends up transforming from a Lean / Agile Team, into a “Push” manufacturing model, where everything is being thrown at them and they simply can’t keep up.
Seriously, avoid the Lucile Ball scenario at the Bon-Bon Factory, because it doesn’t end well.
And this is what it looks like in Real Life, when an Agile Team has gone off the rails. Yes, this is a real Team and they are all over the place. What is worse is that this is a fairly large Team, of 10 individuals. And you can see in one Sprint they only completed 5 Software Points worth of work. And if you look at their Sprint by Sprint Burn-Up Charts it becomes obvious, every Sprint they are adding between 35% and 50% more Scope during the course of the Sprint. So, they are allowing themselves to become overwhelmed and taking on way too much work.
Similar to the Release Burndown Charts, we can also look at individual Epics to see which Epic Stories are either under or over estimated in terms of Scope and thus Time. This is important because it can help tell us which Sub-Teams or Individuals are more accurate in their Estimations; and thus how to plan accordingly.
Everyone instinctively either tends to underestimate or overestimate things. And the other problem we have is that as the item, process, or project gets larger our “Estimates” become less precise and more inaccurate. If you ask anyone to estimate the weight and dimensions of say an iPhone and they are able to hold it, in general most people will be fairly accurate.
But ask them to estimate the weight and dimensions of a Boeing 737 airliner, and unless they happen to be a pilot or aeronautical engineer, they will be not even close and often by a lot. The key here is that when we are estimating an entire Release, we want to break things down into smaller more manageable Epic Stories. Which will also give us better estimates.
The Epic Burndown Chart works very similar to the Release Burndown, as you are tracking progress towards the completion of the Epic Story over multiple Sprints. Likewise, you can see that at times additional Scope is defined or added to this Epic.
The good thing is that after the first couple of Sprints, there is not a large change in the Scope of this Epic Story. The changes in Sprints 4 and 6, once you drill into them further are mainly due to Minor Changes and Bugs that were found by the Subject Matter Experts (SME’s). So, in this case the Team is actually doing a pretty good job of controlling the Scope Changes for this Epic Story.
This next report is just a version of the Epic Burndown Report. But you can easily tell at which points were there major changes or spikes in the Scope of the Epic Story. I’ve circled these in red. Again one of the approaches that you can take is that at the Epic Story Level to create a placeholder User Story that is used to estimate the expected Delta or Change in the overall Scope of the Epic Story. And just like in the Release Plan, you can either do this using Story Points or Hours.
Of course, if you do this at the Epic Level, you can reduce, but not eliminate your Risk Factor at the Release Plan Level. You will still need some Risk Factor at the Release Plan Level to account for Full Integration Testing of the Release and Bugs or Minor Changes that are uncovered once any form of Beta Testing begins.
A Control Chart is used to measure how quickly the Team is able to address or resolve issues over time. This is a measurement of when the Team starts working on an Item and when they complete it. The “Start Time” only begins when it is pulled into a Sprint. Ideally, what you want to see is a relatively flat line or average in Time to Resolution, and also a relatively small Standard Deviation. Naturally, the tighter your Standard Deviation is, it means that the Team is more likely to complete their User Stories or Issues within a predictable period of time.
If you have a very wide Standard Deviation, then that means that:
Here is a real world example of a Control Chart. You can see that this Teams long term average for Time to Completion is very good. And their Rolling Average does not deviate by much. In addition, their Standard Deviation is pretty tight with only a few outliers.
Looking at this Control Chart, you can reasonably say that this is a High Performing Team, especially since a Sprint for this team is 2 Weeks or 10 Business Days. This tells us that most of the items that they commit to completing are actually done on time, with only a few exceptions.
The Software Performance Index Chart was originally developed by Rally Software. What they did was analyze over 19,000 different Agile Software Development Teams using their SaaS product to manage their Projects and Programs.
They selected 4 different criteria for evaluating every Software Team:
Each measurement includes a very extensive analysis of your Team’s metrics, and then compares this to 19,000 other Development Teams around the world. Thus you are able to get a benchmark comparison on where your team ranks overall and also in specific areas where your team needs to make specific improvements.
In this example, this team is in the Upper Quartile (top 25%) for both Productivity and Predictability, which is great. However, they are only in the 2nd and 3rd Quartile for Responsiveness and Quality respectively. So, as a leader, if there is one area to focus on - you would probably start with Quality, as it is the lowest rating.
In this blog article, we have covered several important Key Performance Indicators of managing a Software Release and tracking whether the Team(s) are on track or not, as well as what to watch for. This included:
For more info on release planning, visit Part 1, Part 2 and Part 3 of our article series.
We hope that you have enjoyed this Blog Article and found it useful.
Thank you, David Annis.
