When data is not enough
I love data, and am a huge supporter of all the work being done by various groups to make more data freely accessible, whether it is development statistics such as World Bank data or Child survey statistics, or figures on aid and open government data. My academic background is in statistics, so the idea of all these publicly available numbers to crunch makes me positively salivate.
That said I’ve seen a fair bit of effusive commentary indicating that with all this data we no longer really need analysts and experts to tell us what to think, and that we are in an age of data rather than expertise where web developers are the new truth tellers.
I think we are getting a bit ahead of ourselves. It’s useful to remember a few things:
1. Not everyone has the skills required to manipulate, analyze and draw valid conclusions from all these datasets. Even people who have the skills, usually don’t have the time, so they still need to rely on others to do this work for them. These people would be “experts” – perhaps now from a much broader pool since the producers of the data don’t hold a monopoly over its analysis – but a group of experts nonetheless. In fact this greater availability of data arguably creates a greater inequality between those who have the capability to make use of it and those who do not. Also the more data we have, the more we can feel overwhelmed and the more we need “experts” to help us filter what we should pay attention to.
2. Context is key – even with great analytical skills it can be hard to understand what data actually mean without knowing something about local context. We might now about teacher absenteeism rates, or funding bottlenecks, but this isn’t all that helpful if we don’t understand why they exist – something the numbers alone can’t tell you.
3. The data are not the truth. Despite how numbers are often presented in the media – they usually come from some type of estimation process – either from imperfect and incomplete administrative systems, or from sample surveys, or from non-statistical research techniques. They may not cover our whole population of interest, or might be based on imperfectly formulated questions, or people might give deliberately misleading responses. In some cases when we are looking at relatively rare events (a good example being maternal mortality) the estimates can be so poor that there are extremely wide error margins, and the methodology used means that data collected today actually only tells about the situation some years in the past.
3. Data collection is also an expensive and time consuming business. Despite technological advances that can reduce the cost and time needed for data collection – getting accurate data is still an expensive business and the cost has to be weighed against the benefit, and against competing demands for limited resources. This means that the data we are able to collect only covers a part of what we would usefully like to have, and is collected much more infrequently than we would ideally need to really track change.
4. Perhaps most important of all, there is some real concern to great a focus on numbers is dehumanizing. It creates a greater distance between us (researchers, policy analysts, policy makers, aid workers) and the real experiences and travails of those people we are trying to help, whose lives become reduced to data points. The IDS story project was set up to help remedy this – in their own words “Concern that an obsession with numbers is leading to development donors distancing themselves from those they seek to help has led to the creation of a new initiative which seeks to bring stories and story-telling back into the heart of development communication” (Hat tip to Aidnography for pointing out, and who has a great post questioning whether better aid data will really improve aid which partly inspired me to write this). It is important to remember that we are trying to help people, who each have their own stories, aspirations and motivations, and that it is the sum of their individual actions rather than “explanatory variables” that really achieve change.
I’m a firm believer that we need to invest in more and better data, and that we need to make it as open and available as possible – to broaden the range of people who can effectively make use of it. But I think we need to recognize its limitation,s and that it is only one part of the knowledge we need for effective development. There is still a role for experts to both analyze and interpret data, to understand the context and politics in order to make sense of it and use it in policy and programmes. There is particularly a need to try harder to listen to the voices of those we seek to help to understand their lives and perspectives and to ensure that what we are doing helps them and is valued by them – and this on a personal level not only a statistical one.