Is there such a thing as evidence-based delivery?

I started writing this blog post in frustration at hearing once again the mantra in the aid world, and particularly in the UN of need for us to be doing “Evidence based policy and programming” and discussions and laments about why we don’t yet have this and what to do about it. But then this morning I saw two excellent blog posts by Kevin Watkins of ODI and Owen Barder of CGD on Jim Kim’s “Science of delivery” which also seem to touch on some of the same frustrations.

It seems to me that at least part of why this discussion remains frustrating is that we don’t agree on what evidence is, and what it’s role should be in policy or programming.

At first glance it might seem obvious that our policies and programmes should be “firmly based on the best available evidence” and that wherever the ‘best available evidence” isn’t good enough we should be doing more research to fill the gaps. And it can be quick to jump to the conclusions that we need to invest more in research, carefully designed experiments (such as RCTs) and independent evaluations and on the other to put in place incentives or rules than encourage or even force policy makers and programme managers to use the evidence that is available.

While all this might be useful, I don’t think it will be the silver bullet that some think it will. If this is so self-evident then it begs the question as to why we don’t already practice what we preach. Why isn’t aid work primarily if not exclusively evidence-based?

Firstly, however much we invest in research and evaluation there are limitations to what we will know (see my earlier post the truth is out there for a more detailed discussion of the limits of what we can know). Some interventions lend themselves to rigorous experimental design or data analysis (e.g. cash transfers, medicines) while others are harder to do as experiments (e.g. policy reforms) or to research cost effectively.

Secondly incentives to use, or not to use evidence also matter. It’s not enough to improve the packaging and dissemination of new knowledge, or to make using knowledge part of procedure – there are many other factors that govern whether or not knowledge is used both individual and institutional e.g. focusing on implementation rates reduces the incentives to take the time to learn from research or other experiences. (See my earlier post “creating a demand for knowledge?” for more ion the challenges of getting people to use what is already available)

Thirdly, context matters. In most cases it’s simply not enough to exactly copy a “best science based practice” from one location and directly replicate it in another. At best the approach will need to be tailored or adapted to a new context, at worst the approach may not work at all. Kevin Watkins mentions the importance of politics and how different political structures and power relations can have an important impact on how effective a “scientifically sound” approach might be in practice, and you can’t factor this into an experimental design. Owen Barder also highlight how the traditional “evidence-based approach” also fails to deal with complexity i.e. that there are complex and dynamic relationships between the context and the programme which evolve together in unpredictable ways for which a static approach to using evidence and best practice is ineffective.

But I’d add an additional dimension to the issue of context and complexity. Programmes are implemented by individuals, not just by institutions, and individual people bring their own layer of complexity to a problem which is hard to measure or control. Different project managers performing the same role can have different levels of technical skills, but also different personal motivations, and different personal relationships with other key players in the project – and these can interact with the project in unpredictable ways. What’s more we tend to overlook the fact that someone may perform well in one context with one team yet perform poorly in another situation – so it’s impossible to standardize how exactly a project team will work on the project unless we reduce all elements of judgment and unpredictability – transforming programme management to a production line function.

Yet I don’t think all is lost. There are some additional things we can do to improve how we learn and implement apart from investing more in research. Here are a few of them;

1. Take the broadest possible view of what “evidence means”. Evidence might be rigorous research, but it might also be case studies, stories, ethnographic studies, analysis of “big data”. Take a broad view but recognize what each type of “evidence “ us useful for and what its limitations are. don’t overlook key areas such as political context and power analysis.

2. Do more pilots and dynamic experiments – encourage more experimentation – not only in the sense of control-design experiments but also in the sense of coming up with lots of possible ideas, then trying them out, adapting them as you go and building on those that are yielding results.

3. Start from, but don’t blindly copy past practices – encourage use of existing knowledge and experience – but as a starting point that will be built upon and deliberately and continually adapted.

4. Encourage ongoing collaboration and sharing of tacit knowledge between practitioners as well as sharing of explicit research or evaluation reports and results.

5. Work with and learn from beneficiaries  – they will often have insights into why something does or doesn’t work, or what might be done to address a problem that an outsider cannot see.

6. Keep a diary with detailed ongoing records of what is happening on the ground both internal and external factors to help generate enough material to get a useful insight into what is working and why. Encourage self-reflection by those involved in the project based on this information. But you can also use this as a key source for more rigorous independent analysis.

7. Work on incentives  – make sure that incentives are there to generate and to use evidence but also personal insights and experience.

8. Adapt – rather than looking for an idealized approach to solving a problem once and for all  – keep searching for improved ways to solve it that work in the context where they are applied and keep modifying and varying your approach based on what you are learning.

I’m not sure if delivery can be entirely evidence-based – but evidence informed and learning-based would be a good start.