KM on a dollar a day

Musing on knowledge management, aid and development with limited resources

The joy of polls

with 8 comments

I’ve always been a fan of opinion polls. And after Nate Silver’s triumphant predictions of the US election outcome you’d think everyone would be. But the aid/development sphere has still a way to go to catch up.

In a recent blog post I explained a little about some of the challenges in the post-2015 global consultation. A while ago  week I was a “discussant” at the recent Tech Salon organized on using technology for qualitative M&E and I talked a little more about this project. In preparing my thoughts and listening to the discussion one specific idea kept coming into my mind – that is the importance of opinions and perceptions as part of monitoring and evaluation. I’ve written previously about the need to “listen to beneficiaries”  mostly from the point of view of it being the right way to do participatory development that also has a chance of being sustainable – but it also happens to be good, if not traditional M&E.

In a traditional approach to project monitoring we often look very closely at the supply side of things  – how much of our budget have we spent, how much did we deliver, how many people did we train, was it done on time and according to plan. Or if we are able to measure impact we might look at things the number of kids completing school, or the death rates from preventable diseases. An advantage of this type of measure is that they are usually quantifiable and measurable, and there is a clear logic or change theory around how these measures relate to your project activities.

But what about looking at demand. Do people want what we are giving them, or do they want something else? Are they satisfied with what we are giving them and how it is provided?

Asking people about their opinions, perceptions and feelings is also a useful and complementary way of project monitoring that might tell you a different story that a supply/delivery focused measurement based on “hard” data. Imagine a school project where the school has been built, the teachers hired, the curriculum designed and the cash transfers for the poorest have been distributed, but school attendance and achievement are not progressing at the same pace. You’ve spent your budget, delivered all your project outputs, but something still isn’t right. If you want to find out what’s happening, then you will need to ask people why don’ t they send their kids to school  – is the curriculum wrong, are the school hours inconvenient, do they value education, are there household labour needs for the children, or is there just no better jobs to get if you get an education. Maybe people wish that the aid project was on something else entirely.

Participatory research is of course one way to dig into what beneficiaries want. But this is also costly, time-consuming and usually is only able to cover a small sample of people. It’s good for digging deep to understand an issue  – but sometimes you also want to get a broad and, if you are lucky, representative view of what people are thinking and feeling. This is where opinion polls and surveys come in. Surveys are of course widely used for political purposes – but they are also used a lot by companies seeking to understand the reach and appeal of their brand and their products (which do you like better “Coke” or “Pepsi”, Which three adjectives on this list best describe our product?).

So why not use polls to ask about how you are doing with your project. Or even further, how can we find out about the image and reputation of your organization? Do people think you are effective? Do they think you are easy or hard to work with? And your “products” – do they think your programmes are effective, are they timely? are they responsive to what people need and what they want?

Many large NGOs do use opinion research to measure the appeal of their organization and their work for fundraising purposes – and use this to carefully tailor their appeals and marketing, although they are not always too keen to publicize this.  But use of polls in developing countries is much less common – partly due to logistical challenges – but also possibly because the incentives to do so are less, as there is a greater incentive to increase funding than to satisfy beneficiaries, who don’t pay.

But some organizations are now starting to also take this on. UNDP country offices run surveys of their national partners (which are curiously not to be found online). The UN recently carried out a partnership survey for the “Quadriennial Comprehensive Policy Review of operational activities for development”  – essentially a review of how well programme country governments think the UN is supporting them with its development work. This gave some interesting feedback on how we are doing which provoked some internal discussion – but also (thankfully only) a few who questioned whether government’s opinions of  how we are doing  are as valid as our internal monitoring data.

But the incentives to use polling for monitoring are still limited, unless donors are prepared to ask for them and even to finance them. They can be very informative to  help understand whether a programme is reaching and is appreciated by beneficiaries – but ultimately this will only be given importance if this feedback has consequences – and one of the best  ways would be for donors to ask for this – or for then to finance this type of research to be carried out independently themselves as an additional way of evaluating programmes and organizations they fund.

And now back to the post-2015 agenda for a brief advertisement.  While the process to develop the post-2015 development agenda is commendable for its use of  online discussions, consultations with technical experts and technical analytic papers – there is also an important role for opinion polling here too. Polling can reach a wide audience  that don’t have the means or the time to participate in face to face or online discussions but who can express their views via an online and/or SMS based poll on what their priorities for the post-2015 might be.

But luckily someone has already thought of that! The UN, the World Wide Web foundation and ODI have teamed together to launch the “My World 2015” public poll. Go there and vote for the changes that you would most like to see included in the post-2015 agenda.!

myworld

Written by Ian Thorpe

January 2, 2013 at 3:56 pm

Posted in Uncategorized

8 Responses

Subscribe to comments with RSS.

  1. Ian, thanks for (as usual) a great post. I think the question you are asking is only becoming more important for global development- unless we, very quickly and very effectively, manage to bring outside discussions inside the offices of development organizations, turning them into initiatives that people who we work for/with want, then we’ll be slowly pushed to the margins of discussions. I’d like to make a counter-argument for polling (and surveys, and focus groups). Couple of points here:

    1. Time factor: From the time a decision is made to do a public opinion poll to the time the results are in, I’d guess at minimum 2-3 months go by. Considering the increasingly complexity of our societies, the results of the research would give us a snapshot in time that has already gone by without capturing subtle differences and levers in the society.
    2. Costs/delay/attribution: Related, if based on the research above a policy maker designs and implements an initiative (easily another 2-3 months, and I think I’m being very naïvely optimistic here), how long will it take him/her to check: (i) whether that initiative has had a desire impact or what impact it has had; and (ii) what other things could have happened that might have skewed the impact (attribution). Using opinion polls/surveys/focus groups for this closed loop of policy making (research the problem-design/implement initiative- monitor impact-redesign initiative) is not only costly but it has a built in time delay effect that puts a big question marks over their usefulness. In other words, these methods are easy to game to give you results you actually want to get.
    3. Biases: Three points here- When polled, people generally tell you what you want to hear and when so, their responses are heavily biased by their most recent experience (which may or may not be relevant, but even it is, you’re getting an aggregate perception of recent experiences of many people…talk about a snapshot in time). More critical in my opinion, people give answers to only those questions that they’re asked- in other words, researcher’s ability to scan for weak signals and emerging issues (those that typically end up blowing in your face) is limited. And final bias, the results are generally interpreted by ‘experts’- whether internal or external but humans nonetheless, which means that the interpretation of the data is layered with biases of those ‘experts.’ Often, they tell you what they’re were taught is important, or what they think is important or what they believe you wish to hear.

    What is an alternative? Reading the analysis of the Obama’s election campaign at MIT Technology Review (http://www.technologyreview.com/featuredstory/508836/how-obama-used-big-data-to-rally-voters-part-1/) , it’s as close to any effort I’ve seen at keeping real-time tabs on a large population of people’s preferences, needs, wants, and a host of variables that impact these, again, in real time.

    Until global development can pull off an effort such as this one, in every one of our agencies, in all of our Country Offices (now how cool would that be), another alternative is the use of narratives. The argument here is that people have codified knowledge from generation to generation through story-telling. At the soccer game, during a coffee break with a friend, at a check-out counter in a grocery store- this is where people tell stories about their daily frustrations and needs and wants. The question for us is how to capture these street narratives, aggregate millions of these that take place daily and turn them into policies and programs that will address those frustration and needs and wants? Well this is exactly what we will try to do in regard to several intractable development problems (when I say intractable I mean issues that we have a load of research and insight on but that has yielded very little progress): what is the cause of resentment of local populations in protected areas; what hinders or facilitates local level citizen engagement; how can we measure in real-time social inclusion of Roma. For each one of these questions, we within UNDP and our partners in the Gov’t and civil sector have commissioned and participated in many polls and surveys- where we got responses that, for the most part, we intuitively (or not so) knew we would get. Very little surprises… well surprises are exactly what we’re hoping to get with using narratives for M&E and research.

    I think this comment has stepped over the line of being a comment and into a commentary category and I apologize- you just struck a cord with your post!  Thanks again! Millie

    Millie

    January 7, 2013 at 4:04 pm

  2. Hi Millie – thanks for your thoughtful comment. Maybe you should write a blog about it🙂
    I don’t really disagree with anything you say on the limitations of polls except perhaps that I don’t think that “street narratives” are an equivalent alternative to polls but rather a complementary tool that might be used together or might depend on what you are trying to find out.

    Compared to traditional research methods polls can be relatively quick, and also have a much broader coverage and also despite their biases be more independent than traditional evaluation stakeholders interviews. The main benefit to me – which is shared by narratives is that they reflect the views of stakeholders rather than programme logic.

    Using narratives, and I presume you mean by collecting lots of them, you can dig into details in a way that a questionnaire can’t, and find (surprising) details or experiences you weren’t aware of. The challenges are scale, sense making and representativeness i.e. how do you get enough narratives, how do you interpret them when they are likely to be diverse and possibly contradictory, and the fact that there are probably important differences between those who tell their stories and those who don’t. I also wouldn’t be surprised if people tell you what they think you want to hear in personal narratives either. Similarly narratives can be time consuming to collect – although unlike in a typical survey or poll you can set up a system to collect them continuously so they can also be a means to track feedback over time rather than only at fixed intervals.

    It’s interesting though to see how these approaches can be complementary. For example narratives can be used to identify unseen issues at an individual level and then polls could be used to validate these in a larger population (since its much easier for a beneficiary to respond to a a quick poll than it is for them to tell you a story- and you can process more replies).

    The post 2015 discussions are an interesting case in point -the online discussions on http://www.worldwewant2015.org allow detailed, nuanced inputs from a wide variety of stakeholders including the identification of some potentially important but unanticipated ideas. At the same time if you want to get mass input from a broader audience to get indicative information on people’s priorities than you have the http://myworld2015.org poll which will help identify on mass what people care about the most. I expect that the poll will get a lot more responses, be more representative and be much clearer to interpret but at the same time we already know the questions so we are not going to come up with anything new.
    Both of these are extremely valuable despite their limitations – and they are complementary, not a replacement for each other.

    Ian Thorpe

    January 8, 2013 at 6:09 am

  3. […] KM on a dollar a dayThe joy of polls […]

  4. […] most recent question was about methods that can help us listen to our beneficiaries and better and capture their […]

  5. […] need to look at external validation too – whether though data collected by others – or though polls, or through collection of stories of impact from the perspective of the […]

  6. […] to do measurement have never been better what with big data, real-time monitoring, SMS reporting, opinion polls, really simple reporting, beneficiary storytelling, satellite imagery, RFID tracking etc. in […]

  7. […] those they are seeking to help ( a common topic on this blog, and one for which there are various approaches we should be using […]

  8. […] based on their regular interactions with us.  Ongoing dialogue and periodic feedback surveys or polls can be very informative as to how well our main clients think we are doing  and whether we are […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: