Any change initiative should be accountable, which means we need a way to measure progress. takes the view that you should track both operational metrics and changes in people’s behaviour and attitudes.
Measuring delivery
To measure operational effectiveness, we look at how work items are moving through the system of work. This is based on the lean principle of ‘measuring the work items, not the workers’. We want to know how much work is getting done and how long each item is taking, so we focus on three key metrics, lead time, throughput and work-in-process.
For technical delivery work, we augment this with release metrics based on the DORA Accelerate research: lead time from change to production, deployment frequency, change failure percentage and failure time to recovery
We implement these delivery metrics as part of turning the lights on.
Measuring behaviour change
The purpose of any product is to effect observable changes in behaviour in its target audience, which we call outcomes. If you think of a change programme as a product, then the audience is your staff and and the goal is to measurably change their behaviour.
One effective way to measure behaviour change is through Likert surveys. These ask how much you agree or disagree with a series of statements, using a five- or seven-point scale from ‘Strongly disagree’ through ‘Neither agree nor disagree’ to ‘Strongly agree’.
Designing surveys is hard. There are two criteria for a survey to be considered robust: it needs to be both reliable and valid.
-
Reliable means that if you carry out the same survey multiple times, with different representative samples, you will get ‘similar’ results. They will not be identical but the results should share a similar profile. When this is not the case, you simply have a random number generator!
-
Valid means that if you can derive a result through other means, say by direct observation, then these results should converge. In other words, the survey is not reliably telling you something that is demonstrably false!
Designing a robust survey is a skill in itself, and we recommend working with professional researchers who will be more than happy to help. Most surveys, including many corporate ‘pulse’ surveys, are littered with bias, which immediately invalidates the results.
Survey design needs to account for factors like the wording of statements (framing bias), their order (anchoring bias), inverting the sense of some statements to compensate for people just ticking all the ‘Strongly agree’ boxes (pleasing bias), the people you choose (sampling bias), and so on.
Robust survey data can act as a strong indicator of organizational behaviour. Dr. Nicole Forsgren and her team applied these techniques to the State of DevOps Report surveys over several years to provide the research for DORA’s seminal Accelerate book, which demonstrates a direct (‘inferential predictive’) relationship between various technical and management practices and elite-level organization performance.
This combination of operational and behavioural monitoring provides a rich and reliable basis to measure the progress and impact of change. On larger-scale engagements, the team will use these and other methods to guide the programme of work.