The context
- Major UK broadcaster
- The boring-but-critical tech that does broadcast, not the cool mobile or web apps. If anything goes wrong, millions of TV sets go blank rather than a website being unavailable for a while, so there is a much more conservative mindset.
- 120 people looking after a portfolio of legacy apps, some internal, some vendor, some hybrid (heavily customized vendor products)
- 80+ active branches of the same codebase, with unknown versions of apps in production
- Testing outsourced to offshore tech partner with poor automation skills; thousands of slow SpecFlow tests
The impact
- Reduced average lead time of feature delivery from 12+ months to a few weeks
- Created shared culture of quality, testing, and testability across the department
- Simplified path to live for most critical systems
- Slashed change failure rate (deployments causing a problem in production)
- Massively simplified automated test suite and up-skilled testers to be able to maintain and evolve it
The approach
We paired with individuals and groups to apply new techniques to real problems.
We measured lead time for features; huge variation of 2-14 months.
We reduced releases from monthly to quarterly(!), but delivered what was committed in that quarter, so lead time came down to a guaranteed 12 weeks!
Then we increased release frequency as we improved engineering practices, until we were back to monthly releases but with a lead time of only 4 weeks.
We developed and ran a software testing class because we were unable to find anything that covered the ‘why’ of testing and testability.
We ran 4 classes of 30 people each, which included the external test consultants.
We collapsed 80+ code branches down to single trunk for the core app codebase.
We grouped the thousands of SpecFlow BDD scenarios by functional area and behaviour, then replaced most of them with a handful of small, fast, targeted PyTest tests.
We introduced stubs for testing against third-party dependencies, and found and reported bugs in some of the services, where we shared our automated tests.