KM on a dollar a day

Musing on knowledge management, aid and development with limited resources

When data is not enough

with 6 comments

I love data, and am a huge supporter of  all the work being done by various groups to make more data freely accessible, whether it is development statistics such as World Bank data or Child survey statistics, or figures on aid and open government data. My academic background is in statistics, so the idea of all these publicly available numbers to crunch makes me positively salivate.

That said I’ve seen a fair bit of effusive commentary indicating that with all this data we no longer really need analysts and experts to tell us what to think, and that we are in an age of data rather than expertise where web developers are the new truth tellers.

I think we are getting a bit ahead of ourselves. It’s useful to remember a few things:

1. Not everyone has the skills required to manipulate, analyze and draw valid conclusions from all these datasets. Even people who have the skills, usually don’t have the time, so they still need to rely on others to do this work for them. These people would be “experts” – perhaps now from a much broader pool since the producers of the data don’t hold a monopoly over its analysis – but a group of experts nonetheless. In fact this greater availability of data arguably creates a greater inequality between those who have the capability to make use of it and those who do not. Also the more data we have, the more we can feel overwhelmed and the more we need “experts” to help us filter what we should pay attention to.

2. Context is key – even with great analytical skills it can be hard to understand what data actually mean without knowing something about local context. We might now about teacher absenteeism rates, or funding bottlenecks, but this isn’t all that helpful if we don’t understand why they exist – something the numbers alone can’t tell you.

3. The data are not the truth. Despite how numbers are often presented in the media – they usually come from some type of estimation process – either from imperfect and incomplete administrative systems, or from sample surveys, or from non-statistical research techniques. They may not cover our whole population of interest, or might be based on imperfectly formulated questions, or people might give deliberately misleading responses. In some cases when we are looking at relatively rare events (a good example being maternal mortality) the estimates can be so  poor that there are extremely wide error margins, and the methodology used means that data collected today actually only tells about the situation some years in the past.

3. Data collection is also an expensive and time consuming business. Despite technological advances that can reduce the cost and time needed for data collection – getting accurate data is still an expensive business and the cost has to be weighed against the benefit, and against competing demands for limited resources. This means that the data we are able to collect only covers a part of what we would usefully like to have, and is collected much more infrequently than we would ideally need to really track change.

4. Perhaps most important of all, there is some real concern to great a focus on numbers is dehumanizing. It creates a greater distance between us (researchers, policy analysts, policy makers, aid workers) and the real experiences and travails of those people we are trying to help, whose lives become reduced to data points. The IDS story project was set up to help remedy this – in their own words “Concern that an obsession with numbers is leading to development donors distancing themselves from those they seek to help has led to the creation of a new initiative which seeks to bring stories and story-telling back into the heart of development communication” (Hat tip to Aidnography for pointing out, and who has a great post questioning whether better aid data will really improve aid which partly inspired me to write this). It is important to remember that we are trying to help people, who each have their own stories, aspirations and motivations, and that it is the sum of their individual actions rather than “explanatory variables” that really achieve change.

I’m a firm believer that we need to invest in more and better data, and that we need to make it as open and available as possible – to broaden the range of people who can effectively make use of it. But I think we need to recognize its limitation,s and that it is only one part of the knowledge we need for effective development. There is still a role for experts to both analyze and interpret data, to understand the context and politics in order to make sense of it and use it in policy and programmes. There is particularly a need to try harder to listen to the voices of those we seek to help to understand their lives and perspectives and to ensure that what we are doing helps them and is valued by them – and this on a personal level not only a statistical one.

Written by Ian Thorpe

December 1, 2010 at 4:30 pm

Posted in Uncategorized

6 Responses

Subscribe to comments with RSS.

  1. […]  He writes about knowledge management, information dissemination, and data analysis.  This particular piece is also very interesting as he talks about what happens when there is a large amount of data […]

  2. […] here’s an excellent post on knowledge management: When data is not enough. I particularly like point #1: not everybody has the skills to manipulate, analyze, and draw valid […]

  3. This is in response to this post and the one x-posted on Sean’s site:

    The one difficulty I see with using data to shape solutions to problems is that then you end up only solving problems that are responsive to data. I’m not sure what the appropriate development example would be, but in medicine, it’s length of life versus quality of life. The first is measurable, the second is not, and so many medical systems are set up to push for gains in length of life, even when there is no (or worse) quality of life.

    Brigid

    December 6, 2010 at 9:29 am

    • Agree – I’m sure you are familiar with the expression “we treasure what we measure”. This leads us to put more value on those things we can easily measure – or to focus on our indicators rather than the underlying problem they are designed to measure. Some of our “results-based” management systems further incentivize this.

      Ian Thorpe

      December 6, 2010 at 11:11 am

  4. Nice post, Ian. Although I’m not a statistics expert, from a semantics approach the same can be said. Data usually doesn’t mean anything. I’m not saying data is not important. But data is input and should be given meaning by computer or, even better, people. That’s one of the reasons I like web 2.0 so much. All these people can share data and information, share their part and together we give meaning to all that data and information. Yes, context is key. And context (or meaning) is given socially.

    Samuel Driessen

    December 6, 2010 at 9:44 am

  5. […] more here: When data is not enough AKPC_IDS += "8253,";Popularity: 50% […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: