Too much learning by doing?

“Action without study is fatal. Study without action is futile.” – Mary Beard

There was an interesting twitter discussion earlier this week on #NPcons (not for profit consultants tweet chat) about the relative merits of thinking and doing and the sense that not for profits have a tendency to take action, but not take the requisite time to think about things first.

This echoes a discussion that has been taking place in house about whether we as an organization depend too much on “learning by doing” and don’t reflect enough on the latest thinking and developments in social science and development studies from outside the organization.

I think its a fair criticism to say that too much programme design and  implementation isn’t taking into account the latest available knowledge.  But it’s important to recognize that the type of knowledge that is needed in programme design is manyfold. In addition to looking to see whether plans are informed by the latest scientific research, we also need to look to see whether there is any relevant prior experience both within the country and in similar contexts. We also need to look at the “political economy” – basically how far can we reconcile what we believe is technical correct with what we can convince others to do.

Perhaps the real problem is not  “learning by doing” so much as doing without any learning at all. Symptoms of this would be basing what we do only on our own ideological views or our past experience and instincts and ignoring evidence and learning from others.  It would be failing to incorporate ongoing learning into what we are doing now if it doesn’t fit our views (because we already know the right thing to do and how it is going to work out). It would be the idea that doing something, anything (and being seen to do it) is better than doing nothing at all.

In a sense though I think there is a bit of a false dichotomy between thinking and doing. In development work you can’t really separate the two, and they shouldn’t be in competition. Instead you need to find ways to integrate the two such that doing is informed by thinking and thinking is informed by doing.

You should  think before you do something of course. But also you also need to think about what you are doing while you are doing it and afterwards (what worked and why, and why what worked in theory might not have turned out exactly as you expected in practice).  And if you are involved in development programming it’s not enough to study and research just because  you want to know – and the idea that study can be neatly separated from experience is a misleading one. To be useful learning needs to be focussed on how to improve policy and take action. Both the design of research – but also how it is written up and communicated need to take into account how it can be used to not only inform but also influence decision making on the ground.

Once we have arrived at the stage of having an evidence informed plan – then we need to get into the real business of learning by doing. This means we need to build in suitable mechanisms to monitor and evaluate our programmes and to reflect on them and learn from them as part of the programme design and implementation. To do this we might need to design the programme and then evaluate it around an explicitly identified theory (or  theories) of change.  It also means doing things like collecting suitable baselines, using appropriate methods to be able to determine the outcome and impact of the programme vis a vis other interventions or changes in the environment. It means seeking feedback from partners and beneficiaries, and it means self-reflection on the experience (including using tools such as after-action reviews). Finally it means feeding back whatever insights were gained from the project into the broader body of knowledge of the organization you work in or better the relevant field of development such that this can be used by others.

This last step of feeding back our experience to others is crucial, but often overlooked. Not all programmes are rigorously evaluated or generate bulletproof data on impact – but with some effort on design and monitoring and some reflection afterwards they should all generate learning which can help inform others. If we did this then we would really be learning by doing – and this would be a very good thing.

I’ve got a theory (of change)

(video: I’ve got a theory – from Buffy the Musical)

Duncan Green just wrote an excellent and detailed post “What does a theory of change look like” which examines the reasoning behind having a theory of change and outlines a range of different approaches and change theories that are in use.

I was planning on posting a few thoughts as a reply – but it got a bit long so I’ve written a blog post instead!

As Duncan mentions, theories of change (toc for short) are all the rage in development now and so you better be able to articulate yours. Yet it’s worth ruminating for a moment on what theories of change actually are and what they are (and are not) good for.

Theories of change are just that – theories. In other words they are mental models we use to explain reality since our brains and our current level of knowledge and understanding of the way the world works are limited. Theories are useful because they allow us to simplify and therefore “understand” the problems we are trying to tackle, and use this predictively to help us decide what to do next. But it’s important to acknowledge that they do not really explain the world – but rather give us a manageable snapshot that we can handle and which is useful to us, and is a close enough approximation to reality that we can overlook those areas where it is a bit off.

This is particularly important when thinking about “wicked” or “complex” problems in development. While we can recognize that we are not truly able to establish linear causal relationships between inputs and outcomes that explain and predict everything that is happening, a theory of change can give us a useful and relatively simple approximation of what is happening in part of a complex system – in particular the part we are planning to do our project on. This is appealing since it breaks down a potentially intractable whole into pieces we can tackle.

But how do we develop our theory of change? I’d argue that the choice of theory we use is, in practice, heavily influenced by our background – our beliefs about the world which can come from our upbringing, or our professional training, or the prevailing organizational culture where we work. For example an economist is more likely to use econometric based models, an anthropologist anthropologically based ones, a religious person might well see a divine presence behind the change.  The point here is that the choice of theory is not something that is impartially decided based on the nature of the problem we are addressing, it is also heavily tied up with our own identity and beliefs.

This means that it can be difficult to get a diverse group of stakeholders to see a problem the same way and get them to agree that a particular toc is the best one to use in any particular circumstance. That said, having an articulated toc is an important way to make the reasoning behind a programme design more explicit to others – and to create a dialogue around different potential ways to look at and address a problem in order to surface and address the different world views that often exist around  a particular development challenge.

But it also points to the need for the toc not only to be useful in terms of helping make decisions to those who “believe” it – but also on the need for the toc to be sufficiently simple, and plausible in order to to be able to be understood by and to be able to convince others. This means that there can be a trade off between having a technically thorough model, and having one that is sufficiently straightforward that you can convince others with differing perspectives to accept it.

Another important consideration is that any toc is useful only insofar as it can be used predictively i.e. to help inform planning and decision making for the future. Although it’s possible, and tempting, to develop a model that fits the data when looked at after the event retrospectively and thus believe it can be applied in the future – this retrospective coherence can be misleading and needs to be thoroughly tested in it’s ability to foresee the future with any accuracy (in stock market parlance – past performance is not necessarily a guarantee of future returns).

From a development practitioner’s perspective, it’s probably wise to be flexible in the use of theories of change  – as Duncan says  -to be prepared to have more than one – in particular to be able to choose tocs that i) seems to work based on the evidence we have ii) are ones around which you can get the different stakeholders to agree, or at least agree to try. They need to be both explanatory AND persuasive. We might also need to consider two or more different explanations concurrently and test them out against the evidence – allowing for the ambiguity both between the theories but also that the results might not clearly support one theory over another.

In some instances you might not need a theory of change at all. Most large aid programmes will require one in order to justify, at least to the donor, why we are investing our money where we are – but many development actors – in particular individual innovators and entrepreneurs might not need a well developed theory. Maybe all that is needed is an idea, a recognition that they can offer something that doesn’t already exist, without a clear idea of its potential benefit. The test of whether the idea is any good or not will be whether people in developing countries make use of it or not, and the benefit will be whatever the benefit was that led to adoption.  In other words, while aid agencies, governments and researchers will find it important to have a theory of change- we shouldn’t necessarily impose this “requirement” on all our counterparts, but rather leave some space for unplanned, untheorized entrepreneurialism.

Development 2.0 – it’s more than tech, it’s remaking an industry

(picture: Villages in Action 2010 – a real life example of applying the principles of Development 2.0)

Back in 2009 Giulio Quaggiotto, then with the World Bank produced an excellent thought piece “A Development 2.0 manifesto” that outlined 13 principles to apply the principles on web 2.0 to the development sector which started off a lot of interesting discussion around the potential this has to offer.

I know in Giulio’s new role in UNDP he has been stirring up discussion on the same issues and how they might be taken forward in the organization’s work. Yesterday Johannes Schunter a colleague of his wrote a great piece  on his blog “Development 2.0 is not ICT for Development. In fact, it is something entirely different!” prompted by the internal discussions, and the reaction of some that Development 2.0 is just a ICT for Development with new branding and some different technological tools.

This is all very exciting stuff, so I wanted to share a few of my own thoughts to add to the discussion. Like Johannes, I’m concerned that some people see the idea of “development 2.0” as being mainly about technology. I prefer to look at it as a new approach based on a set of principles which could potentially transform the way we think about and do aid and development work.

The application of technology (especially new social technologies) facilitates these changes, making possible what might not have been feasible in the past – but I believe the exciting part, and the bit we should always keep in sight are the underlying principles – how they can benefit development and THEN how technology can support them.

After thinking about this some I’ve come up with some high level principles abstracting a little from Giulio’s original manifesto. For me Development 2.o is nothing less than a reworking of our existing approach to encompass the following basic but also in some ways revolutionary ideas:

1. Transparency and accountability – the work of all players in the development sector should be visible in a straightforward manner so that everyone can easily see how much money they spend, where it comes from, what projects they are doing, who they procure from and more. This provides an important basis on which organizations can be held accountable to their key stakeholders whether funders or beneficiaries. Initiatives such as IATI for development agencies is a big step forward – but this transparency also needs to extend to governments with transparent budgets, and needs also to include  effective accountability mechanisms that hold organizations and individuals accountable building on the open access to information on what they are doing.

2. Participation and empowerment – “not for me but with me”. Development work needs to find effective ways to listen to and understand the perspective and needs of “beneficiaries” and include them in programme design, implementation and evaluation, both to ensure that they are meeting real needs an a contextually appropriate way, but also to ensure the elusive concepts of “ownership” and s”sustainability” i.e. that the beneficiaries believe it and are committed to keep things progressing, even when external support is removed.

3.  Distributed and diverse collaboration – there are a wide range of different actors who can work together on development taking on different roles or contributing to a common effort. These can include not only aid agencies and recipient  governments but also beneficiaries, civil society organizations, academia, private sector organizations and private individuals. Increasingly this will also include south to south co-operation. Increasingly this can span locations, and can be more informal and loose than traditional co-operation agreements that are often based on contractual arrangements or partnership MOUs.

4. Innovation and adaptation – rather than tackling a problem with a standardized top down solution there needs to be a more conscious effort to adapt approaches from elsewhere and evolve them based on local context and experience. This will include incremental improvements and changes to respond to changing circumstances. It will also involve innovations and trial and error – and diversity in approaches trying out new approaches to problems or simultaneously trying and comparing different approaches to the same problem.

5. Knowledge driven – with all of this data on accountability and performance, all of this experimentation and adaptation and the involvement of a diverse range of partners, one of the killer-apps in the new development will be knowledge – the ability to capture, share understand and apply new learning whether it be previously unavailable data and analyses, or stories from personal experience. And this is not only about “producing” more knowledge but it is about using it effectively including creating a demand for its use. It will also be about connecting people together who can help each other as much as connecting facts and data to where they can be of use.

6. Open – this perhaps summarizes or overlaps wit some of the above points. For knowledge to be transferred, or for participation to occur, an collaboration to be fostered there needs to be openness. This is both conceptual openness – willingness to accept new ideas, work with new partners etc. but also structural openness i.e. eliminating physical and virtual barriers such as proprietary software systems, restrictive copyright, paywalls, professional cartels.

7. Technology enabled not technology driven.  This goes back to my original point. Technology enables us to do many things on the above list that would be impossible or extremely difficult without it. But the technology needs to be in  service of the goals of development, not an end in itself, as seducing as that may be (we all love our toys!). Technologies can be applied in development in ways which support these principles, but also in ways which distract from them becoming a focus in itself, or even worse worse by undermine them and creating new ways to keep information and ideas locked up.

What do you think of this list – do you agree? Are there any major principles I’ve missed?

Bonus: Owen Barder’s paper “Beyond Planning: Markets and Networks for Better Aid” is an excellent discussion piece on how use of the “technologies” of markets and networks might help remake aid in line with some of the principles I’ve outlined here. It’s a great read and he’s much more thoughtful and articulate than I am.

Getting better all the time

 

Last week we were lucky enough to have Charles Kenny, author of “Getting Better: Why Global Development is Succeeding” come talk to a group of us in UNICEF.

Many of you will be familiar with his book which has been reviewed extensively elsewhere – but I just wanted to share a few salient points from his presentation and the discussion which followed.

His main thesis is that people’s quality of life in the developing world has been steadily improving over time, and that tremendous progress has been made which is often overlooked in the dominant media narrative around disasters and the increasing voices of the aid skeptics.

He points particularly to the fact that if you take a longer term perspective there has been steady progress in indicators such as life expectancy, child mortality rates, literacy, water and sanitation access and so on. This progress is more striking and more evenly spread than progress in incomes. Even those in the most disadvantaged parts of the world have a better life expectancy and better health than those in the richest had 150 years ago. Indeed significant progress has been seen even in those countries whose economies have stagnated or declined. He also argues that while income inequality has grown massively both within and between counties that outcomes in areas such as health and education have actually converged – very relevant and very encouraging for UNICEF’s renewed focus on equity.

He was careful to stress that there are still many problems and a lot of work remains to be done, but that progress is real and the future positive i.e. the development glass is half-full rather than half empty (appropriately since his regular column in Foreign Policy is titled “The Optimist“).

He also made the case that aid has had an important role in catalyzing this change. He argued that aid is often judged against its role in promoting economic growth, but while its far from clear that aid leads to growth, this is the wrong measure by which to judge aid and that its impact on other aspects of quality of life is a better benchmark – and here the role of aid is much clearer.

He explained that a large part of this progress is due to the creation and dissemination of new technologies – here meaning technologies in the broadest sense including both physical technology but also process and cultural innovations. In response to a question from the audience he distinguished between two types of technology – “innovative technology”  – things like cellphones, vaccines and “process technology” such as legal systems and institutions. He explained that innovative technology is much easier to transfer from place to place  than process technology which is highly contextual and evolves over time e.g. a public finance system that works in one country can’t just be picked up and transferred to another country and work effectively.  It might also be more effective to look to other developing countries for process technologies that can be adapted rather than trying to import developed world approaches that might not easily fit in an environment with more limited resources as well as different traditions (and maybe trying to impose developed world models has been one of the failings of big aid especially in the areas of institution  and capacity building).

He also suggested that innovative technologies can sometimes substitute for more complex processes giving the example of how creating a modern water and sanitation system can be very complex and expensive, but use of simple water testing/purifying devices and use of soap can massively reduce the mortality and morbidity from waterborne disease at a fraction of the cost (While I take this point in terms of making aid affordable and practical – I’d still see this as a step along the way rather than a long term replacement for having good locally adapted infrastructure).

So in a sense he was saying that “knowledge” is the killer application for development. But one of the challenges is the dissemination of that knowledge and also creating a demand for new knowledge or technologies. Building this demand can be quite challenging (I’ve written about the difficulties of building demand for knowledge in an organizational context before), and especially so if the technologies  go against local culture and traditions and the value of adoption isn’t immediately apparent.

This last topic fascinates me since a lot of aid work around capacity building and communication for development is effectively about trying to transfer and adapt “process technologies” but these are the hardest type to explain and to transfer. There’s often a lot of buzz around innovative technologies (mobile phones, cookstoves, vaccines, solar panels), and these seem more susceptible to quantitative measurement through tools such as clinical trails, RCTs, cost benefit analyses. They might well be a focus because they offer comparatively quicker results. Process technologies are harder to take root since they not only involve identified good practice principles but also adaptation to local culture and politics – as well as behavioural change. In reality of course the two are not divorced since you also need processes to deliver innovative technologies. Yes, you can run externally funded immunization  campaigns  -but for them to be sustainable in the longer long term they need to be locally financed and managed through an effective local health system. So perhaps we are leaving some of the bigger challenges for later – which is fine as long as we don’t put them off altogether!

One ray of hope for the marketing of new technologies is that once people have accepted them, then they are unlikely to be willing to go back. Creating higher expectations can be a very powerful way of encouraging and sustaining change. Once you have a cellphone it becomes hard to manage without it. Once you are used to your teachers turning up regularly to class and know you can complain if they don’t then it’s hard to turn back. Ironically this might be part of why we don’t recognize the great progress that has been made – because we already expect it and demand it – so perhaps the best approach is to be both grateful and optimistic, but also critical and demanding at the same time.

Here is the Ustream recording if you want to see the whole talk for yourself: http://www.ustream.tv/recorded/15268574

Bonus: Liesbet Steer from ODI also attended the session and shared a project she has been working on called “Development Progress” which brings together practical examples of progress stories from developing countries which are backed up by evidence – and includes quite a few examples of process innovations. This is a nice complement to Kenny’s book in highlighting what can be done.

Our future knowledge workers

 

This morning I attended an event at my son’s school. The pupils – 4th graders i.e. 9-10 years old were each to make a short presentation on a topic that interests them and which they had researched.

A few things struck me as being quite different from my own school days at that age. In particular:

– All the presentations were done using Powerpoint and through a microphone

– They were asked NOT just to read what they had written but talk to and elaborate on the slides. For the most part they did this surprisingly well – something I wish we could teach more grownups to do.

– They were asked to cite references – almost all of them used wikipedia. All sources cited were from the web.

– about half chose “serious” topics including George VI, Pablo Picasso and Dinosaurs. About half were popular culture topics including Jennifer Lopez and a lacrosse player and a football palyer that I’d never heard of! All were incredibly detailed.

Maybe I shouldn’t be struck by any of this  – but I found it very interesting how relatively comfortable they all already were in public speaking, using powerpoint and using the internet for research. I also saw how popular culture, especially sports and music already fills up a lot of children’s passion and mind-space.

Don’t get me wrong – I wouldn’t put any of the presenters up in front of a business audience to present just yet – but they’ve certainly got a big head start over where my generation were in school or even in college and our early working lives. I can barely imagine what they will be able to do when they join the workplace and how they will challenge our exisiting ways of thinking and working.

Good and bad reasons to work on knowledge management

I recently heard about a conversation that took place during a planning meeting of one of our programme teams where a staff member suggested that they should start doing knowledge management work, to which the section chief reportedly responded “do we have to?”

My initial response was incredulity – and disappointment in how we were obviously not being effective at selling the importance of our work. But once I thought of it further I realized it’s not such a bad question to ask after all.

Much as I like to spread the word about the importance of using evidence and knowledge in development work, and the benefits of effective knowledge management – I think we all need to ask ourselves whether what we are doing is needed and whether it adds value (given competing demands on time and resources). In our efforts to get knowledge management more “institutionalized” as part of the way we do business we’ve tried to include it in our regular systems including getting it into our organization’s strategic plan, into our performance assessment system for country offices, into our guidance etc. While this is a good way to get KM on everyone’s radar and to get it systematized, it also has a downside – that people might not realize why we should be doing KM in the first place.

Here are some poor reasons why people work on knowledge management

• It’s trendy right now
• My competitors are doing it
• My peers are doing it
• Senior management talk about it a lot so it must be important
• We have to report on it
• My boss asked me to do it
• We can get some money to work on it  (although I haven’t seen this case that often!)

Why are these not good reasons? Because while these might make you start to work on KM, it’s unlikely that they will motivate you to do it well, since by following these “incentives” you are not focussing on why to do KM at all, or on which type of KM activities are going to be most useful to the work you are doing. You are more likely to be going through the motions just soyou can say you have done something.

Here are some better reasons to do knowledge management work:

• It saves money
• It saves time
• We don’t keep reinventing the wheel
• We reduce known mistakes and risks
• We invent and spread innovations and new approaches
• We improve the organization’s reputation
• We improve our own reputation
• We improve our own performance
• We improve the organization’s performance

Knowledge Management is not an end in itself – its aim is to make use better at doing whatever it is we are doing as our core business. If it isn’t then while it might be intellectually satisfying  – it’s not a good investment of our time and money.

(This blog post was inspired by this post by Nick Milton from Knoco Stories “Business focussed KM revisited”)

The development ideas hype cycle

I’ve been following closely the recent debates about the use of Randomized Controlled Trials (RCTs) in development and the emerging attention they are receiving in mainstream media, and the critiques and plaudits they are attracting from some academics and practitioners (Jennifer Lentfer pulled together a good compilation of postings here). But this discussion reminds me of something.

Those of you that have a technology background might well be familiar with Garner’s “Hype Cycle of emerging technologies“.

This is where a new technology seems close to being possible for commercial development, it is is discussed and hyped leading to inflated expectations of what it can do and its likely commercial success. When it fails to deliver on this promise the trough of disillusionment quickly comes along. And finally the real use and value of the technology slowly emerges from the gloom in a stage known as the slope of enlightenment, finaly reaching a stage of maturity and appropriate use known as the plateau of productivity.

Here’s a picture:

This cycle lo0ks very familiar to me when I think about new ideas in development and the latest “silver-bullet”  which gets touted as the end of poverty only to lead to disappointment and then eventually to quiet adoption as one tool among many in the development toolbox.

Right now it looks to me as if RCTs are heading rapidly up towards the peak of inflated expectations (despite the critiques) getting a lot of attention – but probably being oversold as a solution in the process. Chris Blattman sums this up nicely in his post “Go short on randomized trials“.  It’s probably fair to say that RCTs can be a very valuable tool in development, and a tool we should be using much more than we do now – but if I’m right about the applicability of this cycle to development ideas – then we are also headed for a nasty shock where suddenly everyone will be saying how RCTs aren’t really that helpful at all, are too expensive etc. and only after a suitable period of doom and gloom will the idea reach its true potential as a valuable tool in development, but one among many rather than being a silver bullet.

It would be interesting to look at a number of other current and past hot topics in development and map them against this cycle to see where they fall in the same way Gartner does with emerging technologies (which is by the way a fascinating read for techies).

For me use of mobile phones is clearly another approach that is currently reaching a peak of inflated expectations and enthusiasm whereas other aspects of ICT for Development (such as OLPC) are already in the trough of disillusionment. some approaches such as cash on delivery aid are probably heading up the path towards inflated expectations – but still have some way to go before they reach their peak. Others such as cookstoves are probably already there.

Sitting in the trough of disillusionment right now are probably the MDGs and “Aid Effectiveness” (in its Paris Declaration form). Emergency cluster response might possibly be here too. But eventually, one hopes, the best parts of these ideas will rise from the ashes and their effective use will be commonplace and routine but no longer as interesting to talk about (PRSPs and National Development plans anyone?).

I’d love to hear from you about your favourite/least favourite development trends and fads and where you think they fit on this ideas lifecycle. Social media? Public-Private partnerships? Microcredit? Debt relief? Social entrepreneurs?