Target setting and ‘outcome measurement’ are part of a vast industry of checking and ‘holding to account’ but basically it doesn’t work, and probably causes more problems than if you didn’t bother in the first place.

That’s a bit controversial…, particularly if you are in the game of; measurement, checking and ‘holding to account’. Let me explain, but first have a look at these images from the Foresight Tackling Obesities: Future Choices Report from 2007 (via the GOV.UK website).

Measuring outcomes in Adult Social Care. This week I was at a session organised by Monmouthshire Council Social Services and Nick Andrews from the Swansea University, School of Social Care Research, where Toby Lowe from Northumbria University introduced the The Foresight Obesity Map.

The point Toby was getting across was that in a complex system, like society wide obesity, specific interventions on discrete areas are unlikely to have the outcome you hope for.

There’s too much going on; the system is not predictable and it is constantly changing. This means that targets and carefully measurements of progress against them is pretty much pointless. You are trying to define the unpredictable (by setting targets) and measure things over which you have no control. A complex system is controlled by many things beyond what you are concerned with.

The point that Adult Social Care operates in a complex system was not lost on most people in the audience – even the people from regulation/audit/performance management and data. We had an existentialist crisis in a group discussion on our table… ‘What is our purpose…?’ Always a good one to ponder just before lunch.

The key point from all of this is that – we live in a complex world and attempting to control and measure it by breaking it up into smaller parts (reductionism) doesn’t work like you think it does.

If you want to get deeper into complex systems have a look at Toby’s blog, Embracing Uncertainty and what Dave Snowden has to say in posts like; ‘the banality of measurement’ and,‘the myopia of metrics’.

Campbell’s Law and Corruption Pressures. This is my second big point. Basically, people will change their behaviour to achieve targets that are being used to measure their performance. Many people will have heard of the expression, “What gets measured gets done” – attributed to many people – and basically true as far as I’ve observed.

Campbell’s Law describes this change in behaviour as ‘corruption pressures’ which in extreme cases can be ‘corrupt’. Deliberately acting to falsify results or in less extreme cases ignoring some things (like a small act of human kindness) whist you focus on getting the job done on time (because that’s what you are measured on).

If you search the internet you’ll find many examples of people and organisations changing behaviors and ‘corrupting’ measurement processes to make sure they deliver against what is being measured. Examples include; education in schools, hospital waiting times, police records of crimes solved etc etc etc. The evidence for this practice is so widespread that I’m a bit surprised that we don’t talk about it more – it’s a bit of a problem in my view.

You can read more about the work of Donald T. Campbell (a Social Scientist) here. Also, there are people like Charles Goodhart (an Economist) and Marylin Strathern (an Anthropologist) who pretty much said the same thing which you can read about in this post by Dave Snowden, The Strathern Variation. In the post Dave offers his own variation: Anything made explicit will sooner or later be gamed for survival purposes and that need will corrupt practice and people (Snowden)

Dead Cobras (finally). My third point – the law of unintended consequences. When people are ‘gaming’ (a less threatening word than corrupting) measurement system, you often get unintended negative consequences.

Known as the Cobra Effect one of the best known examples comes from time of British Colonial rule in India. The Authorities wanted to reduce the numbers of Cobras so offered a reward for any dead Cobras delivered to them. People saw the financial opportunity in this and started to breed Cobras so they could claim the reward for the dead ones. The Authorities got wise to the practice and stopped offering the reward. The result was Cobra breeders releasing the now worthless snakes and the problem was much worse than when things started.

I suspect that if most people sit and think for a while they will be able to find an example of The Cobra Effect from an aspect of their working life. Maybe not as dramatic as a plague of poisonous snakes, but an unintended negative consequence of people chasing targets.

Outcome Measurement is a Dead Cobra. I mentioned that just before lunch these was a bit of an existential crisis in the discussion at the regulators/auditors/performance and data people’s table.

Basically this is where I was at: “in a complex system target setting and measurement of progress against those targets is pointless, and people will corrupt the system (change behaviour or full on cheat), and you are probably going to get unintended negative consequences – like a plague of poisonous snakes”. Oh happy days!

The Story is The Measure*. Like any good story format I’m offering you a bit of hope here. There was an afternoon session and we came away with some ideas on what you can do to ‘measure progress’ in a complex system.

I promise I will write that post soon (on the train to Bangor on Wednesday) and walk away from the pit of poisonous snakes. All I’m going to say for the moment is that core to it is the idea that ‘the story is the measure’ of what is happening in a complex system. Oh, and by the way I don’t mean manufactured ‘corporate stories’. I mean real stories told be real people in their own, unfiltered words – a bit like what I’m talking about in this post; Cool Story Bro… be cautious of Organisational Storytelling and Silver Bullets

*I’m attributing ‘The Story is the Measure’ to Matt Wyatt @ComplexWales if he didn’t invent it, he will certainly have a view on who did.

So, What’s the PONT?

1. We live and work in a complex system. Simple, traditional linear models do not work in complex systems.
2. Campbell’s Law is a real thing – people change their behavior to meet targets. These ‘corruption pressures’ often have unintended consequences.
3. Unintended consequences are often negative like the Cobra Effect – things are far worse than when you started.