KM on a dollar a day

Musing on knowledge management, aid and development with limited resources

The truth is out there (or maybe not)

with 18 comments

In the organization  where I work, like in many other development organizations, there has been a lot of push over the past few years on “evidence-based” policies and programmes. So when I tell people I work on Knowledge Management, they often imagine that I’m working on strengthening academic research, or on building massive all-encompassing databases full of peer-reviewed scientific knowledge.

Although I am working on some databases – this isn’t what I actually do most of my time – nor despite what some of my professional colleagues think – is this what I think we should be doing.

Development is a complex business, if it weren’t we would have gotten further along in solving the world’s problems before now. One common reason cited as to why we haven’t done better is that we don’t have enough data, and we don’t have enough evidence.

A number of remedies are commonly proposed to help address this:

1. Collect more statistical data – more surveys, more administrative data collection. More recently we have started to say that we need more real time data collection.

2. More research – more academic studies, more randomized controlled tests, more papers published, papers published more quickly.

3. More evaluation – we need to more systematically evaluate more of our programmes to understood what worked and what didn’t and what lessons we can learn. We need to use better evaluation techniques.

4. More, bigger and more open databases – we often acknowledge that a lot of research has already been done or data collected, but that it is not easily available as it is stuck behind paywalls, fragmented and not well disseminated or easily searchable. To address this we strive to make big well-organized mega-databases that are the preeminent knowledge sources on their particular topic, and advocate for more free access to data and research.

Guess what – I actually agree that all these things are worthwhile. I mean how couldn’t I? BUT – too many people seem to believe that if we keep collecting more and more data, do more and more research and evaluations and make more and more comprehensive databases, then we will have everything we need to do evidence based development work. Basically, if we look hard enough, the truth is out there…

There are a couple of reasons why I don’t agree with this:

1. There are limits to how much evidence you can collect

2. There are other important dimensions to knowledge that are actionable, yet tend to get overlooked when we take too strong a focus on “evidence”

Firstly the limits of what knowledge you can collect. In developing county contexts in particular, it can be difficult and expensive to come by high quality, timely and relevant data. Existing data collection systems are often weak, and while they can be developed there are still limits in terms of accessibility of marginalized populations and cost of developing surveys to them to such an extent that they can provide the data needed to answer many of the development policy concerns we have.

Similarly for research, there are a large number of potential questions that we would like to address, but availability of data, costs, time and limitations in the research methods themselves mean that there are a lot of questions that can’t be answered in a sufficiently timely manner for the development of policies and development programmes.

Evaluation also is limited in that it can be very costly, yet only tell you part of what you need to know in terms of whether  a programme was effective and why.

One particular challenge for any knowledge related work is that of generalizability. To what extent can the results of a study of evaluation be generalizable to other contexts and other timeframes and how much do they tell you about what you should do and what will work in the future.

Another important limitation of “evidence” is that even when it exists and is fairly clear (which for the reasons stated above frequently isn’t the case), it often isn’t sufficient to motivate policy makers, politicians, families etc. to take action. Any findings or recommendations also need to be contextualized to  the local culture and to the power relations of the situation where you are trying to use the evidence. People often choose to interpret evidence in a way which supports their current beliefs, are not necessarily going to use peer-review in a reputable journal as their benchmark on whether to trust the source, and may not accept advice they don’t lie that they perceive may weaken their current influence or power.

None of this means that data, research and evaluation aren’t needed. But it does mean that they are not enough. So what’s missing?

An important aspect of knowledge transfer and change is personal relationships. Most people don’t have time or the skills to examine all the available evidence first hand. This means they rely on the opinion of others whom they trust. Similarly standard methods for collecting, storing  and disseminating research often have little impact with people being too busy to seek out the evidence they need, or to even develop the skills to do so. Again people frequently ask others rather than access the evidence directly themselves.

Also there is a whole range of knowledge that isn’t captured by research, that of personal experience. Often you can understand a situation, and describe it to share it with others, but you can’t back it up with scientific research (a trivial example is that I’m pretty sure I know the quickest way to walk to the station in the morning- but I have neither measured it nor timed it).  It might be that it would be too expensive and difficult to prove it through research, or that by the time you know the answer, it would already be too late. Some knowledge is in the form of skills or even instinct which doesn’t easily lend itself to being formally captured at all. This type of knowledge is known in the business as tacit knowledge. Here is a handy diagramme that explains the difference between the two (link to original).

 

So, in order to take advantage of the part of knowledge that lies below the surface (the part which isn’t “evidence” in the formal sense), then you need to take other approaches. These can involve using tools to try to capture some of what is currently hidden to make it more shareable (tools such as after action reports, end of assignments reports, self-reflection exercises, lessons learned, story telling etc.) and through approaches that make it easier for people with shared knowledge interests to find each other, trust each other, share with each other and collaborate (through approaches such as knowledge fairs, communities of practice, social networking, cocreation).

In fact I find that the most interesting, and promising work I do in the area of knowledge management is not about evidence at all – but is  about the social dimension to knowledge. What I need to do is make a better case for this with my colleagues  – but then I’m sure they are going to ask me to show them the evidence!

About these ads

Written by Ian Thorpe

November 22, 2010 at 5:27 pm

Posted in Uncategorized

18 Responses

Subscribe to comments with RSS.

  1. Thanks for this post. My latest post also encourages practitioners to consider the appropriate cost and complexity needed for effective M&E, especially given the size, scope, level of intervention, and length of the program. (See: http://www.how-matters.org/2010/11/17/161-indicators/). Regardless, generalizability and causality will continue to plague us in the development sector, making the tacit knowledge and the social dimension to knowledge even more important.

    Jennifer Lentfer

    November 22, 2010 at 11:03 pm

  2. Hi Ian,
    Great post. I strongly agree and think that the same sort of dynamic is true in the use of evidence in non-development nonprofit work.

    Let me know if you’d be interested in having me republish your post as a guest post on Tactical Philanthropy.

    Sean Stannard-Stockton

    November 23, 2010 at 11:05 am

  3. fascinating! thanks. i really enjoyed that thought-provoking piece.

    Rebecca M.E. Pointer

    November 24, 2010 at 3:25 am

  4. [...] The truth is out there (or maybe not) – KM on a dollar a day – The limits of evidence in development. [...]

  5. [...] ii) enough information to be absolutely certain that we are doing exactly the right thing (see related post “the truth is out there”). [...]

  6. Thanks very much Ian, very good post. It clearly shows that the challenge that we currently have is not so much getting the data, but making sense out of it. And what sense someone reads into something is – for better or worse – always connected to people and individual perceptions. Which is why scientific evidence is not the panacea for all questions that we face in our work, as I’ve highlighted also in my recent blog post (http://jschunter.blogspot.com/2011/01/how-we-deal-with-what-we-know-or-dont.html).

    Of course the World Bank’s approach to opening up their data for analysis and interpretation by the global development community hints to one possible solution. To let the crowd make sense of the vast amount of data that we as individuals will never have the chance to pay justice to.

    Johannes Schunter

    January 12, 2011 at 2:16 pm

  7. Very interesting. Ironically, one of the reasons cited by proponents of ever-more research to accumulate evidence is that evidence is objective and avoids personal bias in interpretation. However, in choosing to ignore the value of knowledge captured from personal experience, the evidence hard-liners are demonstrating their own bias!

    Eric Mullerbeck

    April 29, 2011 at 7:48 pm

  8. [...] of my personal favourite posts (which were not always yours) were “The truth is out there“, “Working and living out loud“, “I want to be where everyone knows my [...]

  9. I completely agree. I found it an excellent answer to a question a hear very often from my collegues in social research: why decision makers do not take into account our job (papers, books, research reports) whe they design public policies and make decisions?. I use to respond in the same manner you did in this article, but I think you made it better.

    Pablo Gustavo Rodriguez

    November 20, 2011 at 8:13 am

  10. [...] @ithorpe 7:58am @poverty_action @viewfromthecave @intldogooder @txtpablo @itsjina exchange reminded me of “The truth is out there” http://wp.me/p110pY-12 [...]

  11. [...] A common thread in these approaches is the idea that there is more to knowledge than what you can get from only looking at research and evaluation. There is a whole area of tacit knowledge from experience that lies beneath the surface (see “The truth is out there”). [...]

  12. [...] we need is highly contextual and based on experience as well as on science (See my previous blog “The truth is out there” about the limits of what we can [...]

  13. [...] Tacit knowledge inside people’s head such as lessons learned and good practices from our experience, or person to person exchange of ideas in online and offline communities that can generate new ideas and improved ways of working. [...]

  14. [...] 1. Not everything worth doing can be easily measured. In fact there is a danger of focusing on what can be relatively easily measured (e.g. vaccination rates) while not focusing on things which are important but hard to measure (like human rights). There are already some calls in the post-2015 discussions to stick with the MDGs because we already more or less know how to measure them. (For more on what can be measured and what can’t see one of my first and most popular blog posts “The truth is out there”) [...]

  15. […] invest in research and evaluation there are limitations to what we will know (see my earlier post the truth is out there for a more detailed discussion of the limits of what we can know). Some interventions lend […]

  16. […] be implemented in a standardized fashion across different contexts. (see an early blog of mine “The truth is out there, or maybe not” for more discussion on the limits of applicable scientific knowledge). The most important of […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 12,617 other followers

%d bloggers like this: