Any change initiative should be accountable, which means we need to measure progress and impact, and be prepared to adapt our strategy based on feedback.
Business goal
Your business goal serves as your ’true north’. This is the reason you are looking to optimize your organization in the first place. We work with leaders to make this goal explicit so it can be the ultimate arbiter of success.
Too many change programmes lose sight of the reason they started, getting stuck in the weeds of change for its own sake, and fail to meet their original objective.
There is surely nothing quite so useless as doing with great efficiency what should not be done at all.
— Peter Drucker
At we recommend using Objectives and Key Results (OKRs) to define the goal and its success criteria. For a more ambitious business goal that will take several quarters to realize, you adjust the key results each quarter, setting an ambitious stretch target that will move you towards the final outcome.
Operational metrics
To measure operational effectiveness, we look at how work items are moving through the system of work. This is based on the lean principle of ‘measuring the work items, not the workers’. We want to know how much work is getting done and how long each item is taking, so we focus on three key metrics, lead time, throughput and work-in-process.
For technical delivery work, we augment this with release metrics based on the DORA Accelerate research: lead time from change to production, deployment frequency, change failure percentage and failure time to recovery
We implement these delivery metrics as part of turning the lights on.
Behaviour change
The purpose of any product is to effect observable changes in behaviour in its target audience, which we call outcomes. If you think of a change programme as a product, then the audience is your staff and and the goal is to measurably change their behaviour.
One effective way to measure behaviour change is through Likert surveys. These ask how much you agree or disagree with a series of statements, using a five- or seven-point scale from ‘Strongly disagree’ through ‘Neither agree nor disagree’ to ‘Strongly agree’.
Designing surveys is hard. There are two criteria for a survey to be considered robust: it needs to be both reliable and valid.
Reliable means that if you carry out the same survey multiple times, with different representative samples, you will get ‘similar’ results. They will not be identical but the results should share a similar profile. When this is not the case, you simply have a random number generator!
Valid means that if you can derive a result through other means, say by direct observation, then these results should converge. In other words, the survey is not reliably telling you something that is demonstrably false!
Designing a robust survey is a skill in itself, and we recommend working with professional researchers who will be more than happy to help. Most surveys, including many corporate ‘pulse’ surveys, are littered with bias, which immediately invalidates the results.
Survey design needs to account for factors like the wording of statements (framing bias), their order (anchoring bias), inverting the sense of some statements to compensate for people just ticking all the ‘Strongly agree’ boxes (pleasing bias), the people you choose (sampling bias), and so on.
Robust survey data can act as a strong indicator of organizational behaviour. Dr. Nicole Forsgren and her team applied these techniques to the State of DevOps Report surveys over several years to provide the research for DORA’s seminal Accelerate book, which demonstrates a direct (‘inferential predictive’) relationship between various technical and management practices and elite-level organization performance.
This combination of operational and behavioural monitoring provides a rich and reliable basis to measure the progress and impact of change. On larger-scale engagements, the team will use these and other methods to guide the programme of work.