Balancing metrics on organizational level & maturity

by | May 17, 2021 | Scaling | 0 comments

Making it tangible: What metrics can I use with the Manifesto for Meaningful measuring” in mind? This is an overview of metrics you can use depending on each level in the organization and each maturity of the team. And remember: outcome is most important, with a special place for the north star metric 

In our previous articles we offered a “cookbook for meaningful metrics”, followed by an article “Acting on key Agile metrics with an Agile mindset” where we also introduced an Agile manifesto for meaningful measuring.  In this last article we want to bring it all together by combining organizational levels with maturity levels and finding the right balance. 

 On the horizontal axis we recognize maturity levels measuring: 

  • Effort: focused on quality, stability 
  • Output: focused on productivity & predictability 
  • Outcome: focused on value & growth 

On the vertical axis we recognize organization levels measuring:  

  • Team level: mostly a Scrum team but most items are also useful for a team that works Scrumban or Kanban  
  • Program level: could be a value stream, SAFe program level or group of teams working on the same product (team of teams) 
  • Organization level: in some organizations this is portfolio level, executive leadership team (ELT) or a Management Team

When we visualize this, this leads to the following graph: 

Though the metrics used in this graph are not a complete set, we think many of them are valid, if applied in the right context, with the right mindset and in the right levels of both maturity and organization. Therefore we’ve included the definitions of the metrics below. 

We hope these articles will help you to implement meaningful Agile Measurements, whilst being sensitive about the right context and maturity- & organizational levels. We also hope that the set of definitions above inspires you not just to blindly use what we have offered here, but to add your own when the context requires. Have a great meaningful measurement roll-out!
Part 1: Cookbook for meaningful metrics
Part 2: Manifesto for Agile meaningful measuring with the right mindset
Part 3: Overview of Agile metrics on each organization level and maturity phase
Written by David Warnink & Jeroen Stoter 

BONUS: Definition of the metrics in the graph

Description of all measurements on each level in the organisation and on the different maturity levels 

Team level

Measures the value of teams 

TeamlevelEffort 

  • Burndown: chart for tracking the completion of different tasks during a sprint. The X-axis refers to the time. The Y-axis represents the work left. burn-up chart shows how much work has been completed and the total scope of the backlog. 
  • Velocity: determines the ability of a team to work through backlogs. As time passes, velocity tends to evolve. The X-axis counts the sprints. The Y-axis represents the amount of points. 
  • Predictability (Planned to done ratio): how much work the team commits to doing at the start of the sprint versus how much they have completed at the end of the sprint. (deliver reliability)  

Teamlevel – Output

    • Lead time: the period between the moment of making a request for delivering a product and the actual delivery. 
    • Defects: all problems with the products (software) after releasing: Error, defect, Bug and Incidents. Also the calls from customers 
    • Cycle time: Cycle time can be defined as how long it takes to produce a software release, from concept to completion. 
    • Sprint costs: trend in the costs of sprint or the cost for a story point 
    • Grow improvements: Improvements that the team is realizing to grow in their maturity using a grow- or maturity model  

    TeamlevelOutcome 

      • Delivered value: the trend in delivering value for the customer over time.  
      • Customer NPS (net promotor score): shows how much the customers are willing to recommend the product or service to others. Customer loyalty is an important leading indicator to determine the success for the future 
      • Team Morale: measures the enthusiasm and persistence of the team engagement  

      Program / value stream level 

      Measures the value Program performance, team of teams or in SAFe the program level measured mostly in program increment. It is often a rolled up aggregation of team metrics  

      Program levelEffort 

      • Release burn down: chart is for tracking the completion of different tasks during a sprint. The X-axis refers to the time. The Y-axis represents the work left. burn-up chart shows how much work has been completed and the total scope of the backlog 
      • Program velocity: determines the ability of a team to work through backlogs. As time passes, velocity tends to evolve. The X-axis counts the sprints. The Y-axis represents the amount of points 
      • Program predictability: (Planned to done ratio): how much work the Program commits to doing at the start of the sprint versus how much they have completed at the end of the sprint. (deliver reliability) 

      Program levelOutput

      • Cumulative flow diagram:  ensures consistency in workflow across the team. Graph to visualize amount of work waiting to be done (backlog), work in progress (started), and completed (validated). The X-axis represents time. The number of issues is on the Y-axis. Ideally, the diagram should be smooth from left to right 
      • Program lead time: the period between the moment of making a request for delivering a product (feature) and the actual delivery. 
      • Program cycle time: Cycle time can be defined as how long it takes to produce a software release, from concept to completion 
      • Release costs: Trend in the costs of every release  
      • DevOps Health radar: The health radar was built to help Agile Release Trains assess their ability to Release on Demand. 

      Program levelOutcome 

      • Delivered value: the trend in delivering value for the customer over time.  
      • Customer NPS (net promotor score): shows how much the customers are willing to recommend the product or service to others. Customer loyalty is an important factor to determine the success for the future 
      • Employee NPS (net promotor score): shows the employees level of motivation (mastery, purpose, autonomy)  

      Organization 

      Performance organization level or portfolio level. Depends on the scale of the organization. 

      Organization levelEffort 

        • Epic burn down (epic progress): Status view of all epics in a portfolio. Chart is for tracking the completion of different tasks during a sprint. The X-axis refers to the time. The Y-axis represents the work left. burn-up chart shows how much work has been completed and the total scope of the backlog 
        • Investing horizon: Overview of the investment horizons in short – middle – long term in the current backlogs (on the horizon levels described by McKinsey).  

        Organization levelOutput 

        • Lead time: the period between the moment of making a request for delivering a product and the actual delivery 
        • R&D over revenue costs:  This metric compares the overall R&D cost with the revenue that results from it.  The ratio between euro’s earned and time spent indicates the rewards that the efforts turn out. The ratio gives an indication of how successful the investment of resources in products are over a certain time. The ratio between actual revenue on contract-, maintenance- and SaaS licenses as mentioned in the Product P&L and actual internal R&D hours spent on that product in any full year. 
        • Business agility: helps portfolios evaluate their progress toward business agility 
        • LPM self-assessment: Structured, periodic self‐assessment to continuously measure and improve portfolio processes. Aligns strategy and execution by applying Lean and systems thinking approaches to strategy and investment funding, Agile portfolio operations, and governance  

        Organization levelOutcome 

        • North star metric: is the key measure of success for the organization. It defines the relationship between the customer problems that the organization is trying to solve and the revenue that the business aims to generate by doing so whilst acting as an overarching “mother”-metric, for all organizational levels to relate (their metrics) to. 
          • Delivered value: the trend in delivering value for the customer over time.  
          • Customer NPS (net promotor score): shows how much the customers are willing to recommend the product or service to others. Customer loyalty is an important factor to determine the success for the future 
          • Employee NPS (net promotor score): shows the employees level of motivation (mastery, purpose, autonomy)  

            

          Specific DevOps metrics 

          We’ve tried to be generic for all product development organizations with the examples above, but have included a set of specific DevOps measurements for Software Organizations. 

          Within a software driven organization using DevOps there some additional metrics available 

          • Continuous Delivery pipeline efficiency:. Measures efficiency of steps in terms of touch and wait time, i.e., analysis, backlog, build, validate, deploy, release, etc Some of the information can be sourced automatically from tools, especially Continuous Integration, and Continuous Deployment, while other data requires manually recording in a spreadsheet. 
          • Deployment frequency: tracks the frequency of deployments. The objective of DevOps is to develop and deploy smaller deployments more frequently, as reducing the size of deployments and the amount of changes for each cycle makes it easier to test and release the deployment. 
          • Total test duration: the time it takes to run the automated tests 
          • Requirements coverage: measures what features are tested, and how many tests are aligned with a user story or requirement. 
          • Unit test coverage: Shows how much of your source is tested. It helps you assess the quality of your test suite. Combination of function, statement, branches, condition & line coverage 
          • Path coverage: measures the linearly independent paths covered by the tests 
          • % automated tests: the percentage of automated test related to all tests 
          • Test execution: measures total tests executed as part of a build. 
          • Recovery over time: measures the number of rollbacks that occurred either physically or by turning off feature toggles. 

           

          Related Post(s)

          Pin It on Pinterest

          Share This