How many times have you heard a funder or charity talk about the need to demonstrate impact? It’s one of those stock phrases that rolls off the tongue in any conversation about evaluation. I’ll confess up front to having dropped the DI-bomb on a number of occasions. However, the more time I’ve spent grappling with questions around measuring impact the more I’ve become convinced that the concept of demonstrating impact is deeply unhelpful.
The reason the idea is damaging is because of what it implies. In short, it assumes an answer to the thing it should be setting out to to test. From this perspective your organisational impact always exists, all you need to do is find it. To see why this is wrong you only need to accept that proper impact measurement should be - at least in part - scientific in nature. Scientists conducting research usually start with a hypothesis, but the research itself is designed to test that hypothesis. When research is conducted where we suspect that this testing is far from neutral, for example when a sugar company funds a study on the link between sugar consumption and obesity, we quite rightly treat it with scepticism.
We also prefer to keep an open mind in less scientific endeavours. You do not, for example, approach a staff appraisal with the intention of demonstrating high-performance: the level of performance is what you are trying to ascertain. It’s surely right, therefore, that when talking about the impact of a particular programme or intervention we should make no assumptions about what that impact might be.
Does talking about demonstrating impact really preclude genuine questioning of what your impact is? You could argue that the phrase just embodies a rational response to a climate where funders and supporters expect organisations to do more, and that many organisations have become adept at demonstrating value to external stakeholders whilst maintaining the ability to critically examine impact internally. This may well be true in some cases. But in many others the phrase is symptomatic of a culture that sees impact measurement as a tool for validation rather than improvement. Anyone who has worked in the sector has probably seen this culture in action. It’s exemplified by the board meeting a friend recently told me about where a trustee responded to an impact analysis showing only slight gains by saying “we must be having more impact than that” and asking if they were sure the analysis had been done properly. It’s exemplified by the organisations where the bulk of impact work sits within the fundraising team and data goes straight from the frontline into a glossy external-facing impact report. And it’s exemplified by the CEO who wants to cut certain data slides out of a presentation because they are not “on-message” enough.
Challenging this culture requires a number of things. First, charity leaders need to work hard to combine belief in the potential of their organisation with a healthy scepticism about whether it is yet meeting that potential. This is a difficult balance to get right, especially when so much of these roles involves advocating for the organisation and the work it does. Nonetheless, ensuring those at the top of an organisation lead the way in treating impact as something that needs to be questioned rather than assumed is vital to improving the effectiveness of the sector.
Second, we need to reset expectations about the impact organisations are expected to have. This starts with a recognition that charities are often working on the toughest of social problems. The gravity and scale of the problems alone should ensure that we start from the assumption that having an impact is extremely difficult. It should be perfectly acceptable for a charity in the early stages to be able to say that it’s not yet sure what impact it is having. We should expect them to have a credible theory as to why their intervention can have impact, but expecting them to be able to produce proof at that stage is unrealistic. Funders have a massive part to play here. Frankly, funders that demand evidence of impact in funding bids for anything other than very established charities are part of the problem. All this does is force charities to engage in a justification exercise, often producing dubious numbers just to satisfy the funder. Instead funders should be asking more open-ended questions about what charities know and don’t know about their impact. This would be a far better way of identifying the organisations that are most likely to have an impact because they see impact as something to question and constantly strive for rather than a given.
So next time you find yourself about to use the phrase “demonstrate your impact” stop for a second and consider an alternative. Pick something instead that reflects the kind of critical, improvement-driven impact culture that we want to cultivate. I’m going to go for “monitor and question your impact”. It’s less succinct but much more on-message.