In measurement we trust?

I’ve been reading with interest Bill Gates “Annual letter” on why measurement matters in aid and development – and finding myself both nodding and shaking my head at the same time.

Of course there is a lot to be said for better measurement in development work. A few of the important reasons why we should care more about measurement include:

1. Accountability – donors, (and beneficiaries) are increasingly demanding to know where the aid money is going and what is being achieved with it – and greater transparency and accountability makes it easier to see how aid agencies are doing, to compare them and to invest wisely. It also creates powerful disincentives against poor management and corruption (hey you! – you know we can see what you are doing – right?)

2. Knowing how you are doing – If you have a clear plan in place with identified outcomes and a good monitoring/ measurement system to support it then you can keep tabs on your programmes and see how well they are progressing, quickly identify problems and take steps to get back on course. You know where you are going – and how far along the way you are. While it seems obvious that every programme should have this in practice too many programmes have unclear objectives and poorly formulated monitoring indicators – often because there isn’t sufficient thought and resources put into ensuring the indicators can be collected and analyzed regularly and in a timely manner.

3. Testing hypotheses – if you want to know whether a particular intervention will work, or how impactful it will be under what circumstances, or if you just want to understand how different aspects of development are interrelated – then to do this “scientifically” you will most likely need to use research methodologies that are heavy on data collection (such as randomized control trials).

4. Learning – and if you are running a project you can also use measurement to learn from your programme as you go. Not only to identify small course corrections on your plan – but also to adjust your plans based on actual experience and to identify positive and negative spillovers from your project which you might not have anticipated.

5. Bottleneck analysis – this involves looking at each element of a system and monitoring to see which elements of the chain are most responsible for failing to reach development goals (kind of a six-sigma or quality circle approach to aid) and thus help decide where to focus your attention to get the best results. An example would be for primary education you might look at i) education budgets ii) school construction iii) teacher training iv) curriculum development v) cost barriers and incentives such as school fees, uniforms etc. vi) culture and attitudes (e.g. on girls’ rights to education). Looking at each aspect of the system you can see which of these are the most critical barriers (or bottlenecks) to achieving greater coverage and focus on addressing those first.

Most of these issues are not new, but the possibilities to do measurement have never been better what with big data, real-time monitoring, SMS reporting, opinion polls, really simple reporting, beneficiary storytelling, satellite imagery, RFID tracking etc. in addition to better traditional data as statistical capacity and evaluation methodologies continue to improve.

So why the head shakes? For me the Gates letter while advocating measurement and science seems a little “faith based” in its level of belief in what better measurement can achieve – as if better measurement alone would be enough to change the world. But there are a few pretty strong reasons why that’s not the case:

1. Not everything worth doing can be easily measured. In fact there is a danger of focusing on what can be relatively easily measured (e.g. vaccination rates) while not focusing on things which are important but hard to measure (like human rights). There are already some calls in the post-2015 discussions to stick with the MDGs because we already more or less know how to measure them. (For more on what can be measured and what can’t see one of my first and most popular blog posts “The truth is out there”)

2. Complexity – Measurement is often used in too much of a linear or reductive way looking at each step in a programme as an independent issue to model, manage and benchmark. But in reality different elements of an environment are interrelated in unseen ways and these also might vary between different contexts that on the surface appear similar. Taking the issue of education – increasing the number of schools and teachers might for example also decrease the quality of teaching and thus have a negative impact on enrollment and even outcomes.

3. Having the data is not a guarantee of acting on it – failure – a strong focus on measurement and accountability in a difficult funding environment means that there is little willingness to admit and learn from failure in the fear that this will mean that organizations and programmes that are unsuccessful – but are learning from those mistakes and providing valuable lessons for others will be defunded. Ed Carr has an excellent analysis of this issue in his blog on the Gates letter.

4. Political will – knowing is a first step, but without a will to address the issues identified it won’t take you far. Too much monitoring, evaluation and research goes unread since it isn’t in a language or format decision makers can understand and use.  Communication of results is one part of this. Mobilization is another. Sometimes more anecdotal evidence is better for persuading decision makers and the public – or at least hard data  – but backed up by real life examples. Sometimes despite the data being clear, there just isn’t the political will to address the difficult issues raised or the leverage to put pressure on decision makers to take the hard choices they need to. Looking at evidence without looking at power and politics and who is influenced by what and why won’t take you very far. Unfortunately just because you can create or access knowledge isn’t any guarantee it will be used (See creating a demand for knowledge for more on this)

5. Evidence isn’t neutral – a follow on point to the last one is that the use of data and knowledge or even its creation is very political. Deciding what is important to collect, who will use it and how – or deciding as a funder or senior aid agency manager which data “matter” or what evaluations really mean about the success of a project are highly subjective and often depend on other information than what appears to be being discussed objectively (such as who is involved with the project as funder, manager, evaluator and beneficiary and the relationships between those actors).

So measurement is important and its good that Gates raises it and promotes it. But while it may be necessary to improve measurement to improve aid, and while we may have unprecedented opportunities, it’s far from sufficient – and without thought to the limitations of measurement – and the necessary complementary actions in the area of people, politics and power it may not achieve very much.