Making it tangible: What metrics can I use with the “Manifesto for Meaningful measuring” in mind? This is an overview of metrics you can use depending on each level in the organization and each maturity of the team. And remember: outcome is most important, with a special place for the north star metric 

In our previous articles we offered a “cookbook for meaningful metrics”, followed by an article “Acting on key Agile metrics with an Agile mindset” where we also introduced an Agile manifesto for meaningful measuring.  In this last article we want to bring it all together by combining organizational levels with maturity levels and finding the right balance. 

 On the horizontal axis we recognize maturity levels measuring: 

• Effort: focused on quality, stability 
• Output: focused on productivity & predictability 
• Outcome: focused on value & growth 

On the vertical axis we recognize organization levels measuring:  

• Team level: mostly a Scrum team but most items are also useful for a team that works Scrumban or Kanban  
• Program level: could be a value stream, SAFe program level or group of teams working on the same product (team of teams) 
• Organization level: in some organizations this is portfolio level, executive leadership team (ELT) or a Management Team

When we visualize this, this leads to the following graph: 

Though the metrics used in this graph are not a complete set, we think many of them are valid, if applied in the right context, with the right mindset and in the right levels of both maturity and organization. Therefore we’ve included the definitions of the metrics below. 

We hope these articles will help you to implement meaningful Agile Measurements, whilst being sensitive about the right context and maturity- & organizational levels. We also hope that the set of definitions above inspires you not just to blindly use what we have offered here, but to add your own when the context requires. Have a great meaningful measurement roll-out!
Part 1: Cookbook for meaningful metrics
Part 2: Manifesto for Agile meaningful measuring with the right mindset
Part 3: Overview of Agile metrics on each organization level and maturity phase
Written by David Warnink & Jeroen Stoter 

BONUS: Definition of the metrics in the graph

Description of all measurements on each level in the organisation and on the different maturity levels 

Team level

Measures the value of teams 

Teamlevel – Effort 

• Burndown: chart for tracking the completion of different tasks during a sprint. The X-axis refers to the time. The Y-axis represents the work left. burn-up chart shows how much work has been completed and the total scope of the backlog. 
• Velocity: determines the ability of a team to work through backlogs. As time passes, velocity tends to evolve. The X-axis counts the sprints. The Y-axis represents the amount of points. 

• Predictability (Planned to done ratio): how much work the team commits to doing at the start of the sprint versus how much they have completed at the end of the sprint. (deliver reliability)  

Teamlevel – Output

• Lead time: the period between the moment of making a request for delivering a product and the actual delivery. 
• Defects: all problems with the products (software) after releasing: Error, defect, Bug and Incidents. Also the calls from customers 
• Cycle time: Cycle time can be defined as how long it takes to produce a software release, from concept to completion. 

• Sprint costs: trend in the costs of sprint or the cost for a story point 
• Grow improvements: Improvements that the team is realizing to grow in their maturity using a grow- or maturity model  

Teamlevel – Outcome 

• Delivered value: the trend in delivering value for the customer over time.  
• Customer NPS (net promotor score): shows how much the customers are willing to recommend the product or service to others. Customer loyalty is an important leading indicator to determine the success for the future 

• Team Morale: measures the enthusiasm and persistence of the team engagement  

Program / value stream level 

Measures the value Program performance, team of teams or in SAFe the program level measured mostly in program increment. It is often a rolled up aggregation of team metrics  

Program level – Effort 

• Release burn down: chart is for tracking the completion of different tasks during a sprint. The X-axis refers to the time. The Y-axis represents the work left. burn-up chart shows how much work has been completed and the total scope of the backlog 

• Program velocity: determines the ability of a team to work through backlogs. As time passes, velocity tends to evolve. The X-axis counts the sprints. The Y-axis represents the amount of points 
• Program predictability: (Planned to done ratio): how much work the Program commits to doing at the start of the sprint versus how much they have completed at the end of the sprint. (deliver reliability) 

Program level – Output

• Cumulative flow diagram:  ensures consistency in workflow across the team. Graph to visualize amount of work waiting to be done (backlog), work in progress (started), and completed (validated). The X-axis represents time. The number of issues is on the Y-axis. Ideally, the diagram should be smooth from left to right 
• Program lead time: the period between the moment of making a request for delivering a product (feature) and the actual delivery. 
• Program cycle time: Cycle time can be defined as how long it takes to produce a software release, from concept to completion 
• Release costs: Trend in the costs of every release  
• DevOps Health radar: The health radar was built to help Agile Release Trains assess their ability to Release on Demand. 

Program level – Outcome 

• Delivered value: the trend in delivering value for the customer over time.  

• Customer NPS (net promotor score): shows how much the customers are willing to recommend the product or service to others. Customer loyalty is an important factor to determine the success for the future 
• Employee NPS (net promotor score): shows the employees level of motivation (mastery, purpose, autonomy)  

Organization 

Performance organization level or portfolio level. Depends on the scale of the organization. 

Organization level – Effort 

• Epic burn down (epic progress): Status view of all epics in a portfolio. Chart is for tracking the completion of different tasks during a sprint. The X-axis refers to the time. The Y-axis represents the work left. burn-up chart shows how much work has been completed and the total scope of the backlog 
• Investing horizon: Overview of the investment horizons in short – middle – long term in the current backlogs (on the horizon levels described by McKinsey).  

Organization level – Output 

• Lead time: the period between the moment of making a request for delivering a product and the actual delivery 

• R&D over revenue costs:  This metric compares the overall R&D cost with the revenue that results from it.  The ratio between euro’s earned and time spent indicates the rewards that the efforts turn out. The ratio gives an indication of how successful the investment of resources in products are over a certain time. The ratio between actual revenue on contract-, maintenance- and SaaS licenses as mentioned in the Product P&L and actual internal R&D hours spent on that product in any full year. 

• Business agility: helps portfolios evaluate their progress toward business agility 
• LPM self-assessment: Structured, periodic self‐assessment to continuously measure and improve portfolio processes. Aligns strategy and execution by applying Lean and systems thinking approaches to strategy and investment funding, Agile portfolio operations, and governance  

Organization level – Outcome 

• North star metric: is the key measure of success for the organization. It defines the relationship between the customer problems that the organization is trying to solve and the revenue that the business aims to generate by doing so whilst acting as an overarching “mother”-metric, for all organizational levels to relate (their metrics) to. 

• Delivered value: the trend in delivering value for the customer over time.  
• Customer NPS (net promotor score): shows how much the customers are willing to recommend the product or service to others. Customer loyalty is an important factor to determine the success for the future 
• Employee NPS (net promotor score): shows the employees level of motivation (mastery, purpose, autonomy)  

Specific DevOps metrics 

We’ve tried to be generic for all product development organizations with the examples above, but have included a set of specific DevOps measurements for Software Organizations. 

Within a software driven organization using DevOps there some additional metrics available 

• Continuous Delivery pipeline efficiency:. Measures efficiency of steps in terms of touch and wait time, i.e., analysis, backlog, build, validate, deploy, release, etc Some of the information can be sourced automatically from tools, especially Continuous Integration, and Continuous Deployment, while other data requires manually recording in a spreadsheet. 
• Deployment frequency: tracks the frequency of deployments. The objective of DevOps is to develop and deploy smaller deployments more frequently, as reducing the size of deployments and the amount of changes for each cycle makes it easier to test and release the deployment. 
• Total test duration: the time it takes to run the automated tests 

• Requirements coverage: measures what features are tested, and how many tests are aligned with a user story or requirement. 
• Unit test coverage: Shows how much of your source is tested. It helps you assess the quality of your test suite. Combination of function, statement, branches, condition & line coverage 
• Path coverage: measures the linearly independent paths covered by the tests 
• % automated tests: the percentage of automated test related to all tests 
• Test execution: measures total tests executed as part of a build. 

• Recovery over time: measures the number of rollbacks that occurred either physically or by turning off feature toggles. 

Metrics in an Agile organization are not magic in itself. The magic is in the behavior of people and especially leadership acting on the metric. Our Agile Manifesto for meaningful metrics helps in making the preferred behavior visible by focusing on dialogue, the trend and the coherence in the numbers. 

Following our first article on Agile metrics “Cookbook for meaningful Agile metrics”, we promised to break a lance for an Agile Manifesto for meaningful measuring. We believe that Agile metrics are an essential component of a high performing Agile organization. Agile metrics help Agile teams and their management measure the effort, output and outcome. Value delivered to customers – instead of measuring “what” or “how much” we are doing, is the most powerful.  

By measuring, agile metrics help keep the different levels in the organization in check whether there are any problems or opportunities for improvement. They also give a clear assessment of the current state.  

We have seen companies where management wants to use for example cost per story point as a standard measurement across teams; or management tries a one-size fits-all and forces all the teams use the same point measurements.  

This invariable leads to anti-patterns which always prove to be harmful. Remember, like velocity, it’s never valid to compare the cost per story points between teams. Every team is composed of unique individuals with their own set of capabilities, work habits, strengths, and weaknesses. As a consequence, teams will invariably have different results in their estimates, velocity, and cost per story point. Fixed reports that are not tailored towards the specific needs of the team, it’s maturity and organizational level will not help all teams to learn, grow and improve. This does not imply that such a metric cannot be helpful for one specific team if the context is right! 

This is where we want to introduce a Manifesto for Agile meaningful measuring; while there is value in the items on the right, there is more value in the items on the left. 

Agile Manifesto for meaningful measuring 

Dialogue about the why over Judging People 

To enable learning and growth, a safe environment to make mistakes and learn from them is key for teams to open up and face the opportunities to improve. Data without context can turn out to be nothing more than noise and distraction. The role of a leader is to support the teams to understand the direction and facilitate them and help remove their impediments. 
Nothing beats a good conversation with the team; a metric can be a starting point, not an accountability gate for the team to pass 

Coherence and Trends over Maximizing one metric 

Viewing all metrics in conjunction is important to avoid increasing fixation on improving one number. This creates more space to research the core problems in order to arrive at solutions that positively improve multiple metrics. The last number of the metric does not say much, following trends gives room to react to the pattern Don’t sail blindfolded on your dials; be aware of context and as such not just one metric will tell the full story. 

Realtime and Transparent over Fixed Reporting 

Transparency is one of the pillars in an Agile organization; this also applies to measuring. Therefore make all metrics public, irrespective of their level. This is a clear indication of transparency and underpins what you are steering towards as manager. Working with the Obeya room methodology can help. With the current techniques this can be done in real time and on a daily basis, aimed at supporting the employees. It prevents you from steering on monthly reports afterwards because you are then too late or the picture is not up to date. The value of a fixed report is therefore low in a complex, fast organization. 

Outcome over Effort 

A North star metric can be a great metric for all organizational levels to define and measure their own contribution to the overall goal. When a team or organizational level matures, the basic “hygiene” is in order, the “right to play” has been established. Growing through “Output” towards “Outcome” oriented performance and related metrics is a rewarding journey for all stakeholders.  
In general, we prefer leading indicators over lagging indicators. 

Bonustip

To close this episode, here are five tips how a good metric looks like (source) 

1. The metric is used by the team – Agile metrics should not be imposed or measured by management, they should be used voluntarily by agile teams to learn and improve. 
2. The metric is surrounded by conversation – Metrics should not just be numbers, they should be the starting point of a conversation about process and roadblocks affecting the team. 
3. The metric is part of a specific experiment – Metrics should be used to answer a specific question about agile processes, not just measured for the sake of measurement. 
4. The metric is used in tandem with other metrics – Even a great metric, if used alone, might lead to tunnel vision, and incentivize teams to maximize that metric at the expense of all else. Using several metrics together provides a balanced picture of agile activity. 
5. The metric is easy to calculate and understand – Metrics that are overly complex or not fully understood, even if they provide good insights about a team’s work, are not useful in guiding day-to-day activities. 

In the next episode, we will bring it all together by linking organizational levels to maturity.

Part 1: Cookbook for meaningful metrics
Part 2: Manifesto for Agile meaningful measuring with the right mindset
Part 3: Overview of Agile metrics on each organization level and maturity phase

Written by David Warnink and Jeroen Stoter

Agile metrics are an essential component of a high performing Agile organization. A great metric challenges teams to be better if surrounded in the right mindset. Metrics are there in different phases effort, output, outcome and on different levels in the organization.  

Creating agile organizations is the default answer to today’s increasing complexity and product development velocity; so much that required transitions become a goal in itself, rather than a means to an end. As such some best practices are implemented at will, without proper context. 

The challenge we see, is that throughout such transitions, it is imperative that for all sorts of stakeholders, there is value to be delivered captured in relevant metrics, to assess progress. 

As the transformation journey progresses, the metrics need to support the maturity phase of the organization at hand. If this sounds complicated, we do this in practice all the time. If you think about the ability to for example “light matches” is not something we want to measure with young children. But when they grow up, we love it when they want to cook us a meal, by lighting the stove as a first step. So the metric “ability to light a fire” is not one with a relevant meaning in the early stages of maturity. 

And then there is the element of stakeholders, often assessing “value” on different axes. From a market or customer perspective, the ability to adopt a product early to gain efficiencies may be more important than whether the team that delivers is planning sufficient improvements to remain efficient themselves, or even technologically relevant. And other stakeholders include internal staff or teams, portfolio, process improvements, innovation. 

We believe there are no shortcuts here, an organization is a living species, and its parts, its people, need to walk the transformation path for them to learn and progress in the process. There is no on-size fits all “check-list”, but we believe there are three generic phases: 

• Effort 
• Output 
• Outcome 

In the effort phase, it is about “the right to play”, more focused internally, requiring metrics to demonstrate a level of stability. Examples include “predictability” on a portfolio level, or “epic burndown” on a team level. 

In the output phase, as maturity is growing, customer focus is increasing. Metrics include “Defects”, or even “Business Agility”. 

When the rhythm is there, the next logical step is focusing on outcome. NPS and Employee engagement are typical metrics in this phase. 

One needs to understand the introduction of a metric that is suitable for the outcome phase, does not serve well if the basics are not in good order. You don’t want to measure the child on his ability to light a match.  

So what does a great metric look like? A great metric challenges teams to be better. Though such metric can set a big hairy and audacious goal (BHAG) – the goal needs to be achievable to be motivating; on an organization level, an overarching North-Star metric to which the underlying metrics contribute, can serve as a means for teams to contribute to the overall organization goals. 

Metrics are not blind dials to sail by; leaders need to provide context and safety for teams to learn and grow. By having a conversation standup or Obeya, supporting the teams to find and solve their impediments. If leaders use metrics to judge, this will lead to undesired behavior of the teams.  

So here are five tips: 

1. Recognize the maturity phase and apply metrics that fit that phase; 
2. Don’t sail blindfolded on your dials; be aware of context and as such not just one metric will tell the full story; 
3. Emphasize the opportunity to learn by having the right conversations; metrics are not a managers tool, they are for teams to support their maturity growth; 
4. Nothing beats a good conversation with the team; a metric can be a starting point, not an accountability gate for the team to pass.  
5. Teams need to feel safe to be able to learn and grow 

The result will be beneficial to all stakeholders.  

In a next episode, we will break a lance for an Agile Manifesto for meaningful measuring.

Part 1: Cookbook for meaningful metrics
Part 2: Manifesto for Agile meaningful measuring with the right mindset
Part 3: Overview of Agile metrics on each organization level and maturity phase

Written by David Warnink & Jeroen Stoter