Archive for the ‘rants’ Category
Late last year I went to the South Asia region to participate in a discussion on knowledge management. I talked briefly about what we are doing globally, and then each country gave an update about what they are doing – and what their priorities and challenges were.
While I was trying to make the case for cross organizational networking and working out loud, as well as the potential for using social tools, what I heard from the ground was a bit of a wake up call as to where many offices still are in their thinking about knowledge management.
One of the biggest preoccupations of offices was being able to organize and then retrieve their everyday documents, and for many of them the way to do this was by reorganizing their “shared drive” i.e. a shared storage space on server in a local area network (I define this because some of you might be surprised that we are still working this way).
So what’s my advice about how to improve the management of the shared drive. Simple – GET RID OF IT.
What’s wrong with having a shared drive in your office?
- It’s only accessible when you are in your office – you can’t easily access it when you are at home or travelling.
- Access control is limited – usually at the level of folders directory and you can’t easily share things with people outside of your group on an ad-hoc basis (also because the probably don’t have access to your LAN)
- There are limited metadata and only a simple search to help you find things.
- Space is usually limited and you are most likely already close to your quota.
- Let’s face it – your shared drive is probably a mess. You most likely have years of accumulated documents in a directory structure you didn’t create and possibly don’t understand. No-one knows what half the materials are or whether they are important and whether or not they are current. There is no logical file or directory naming convention, no-one can tell who edited what and which is the latest version.
So what can you do instead? Use the cloud!
In UNICEF we are a Microsoft Office365 house so the tools I’m going to mention here are based on that but Google, IBM and other providers all have their own suite of tools that do the same thing.
If all you need is access to a set of official reference documents that people don’t need to frequently edit and update or that are confidential then the best place to put these files is on your intranet – don’t refer people to copies you keep elsewhere – it’s much easier to have a simple intranet page (or set of intranet pages) and keep them logically organized and up to date than it is to create a separate storage silo.
If you need a place to store shared documents that people need to collaboratively develop and edit, or a space for working documents e.g. for an office, a particular team, for a community of practice or network, you should use a SharePoint document library.
For individual files that you want to access remotely, or where you want to collaborate on these files with a more limited or ad-hoc number of colleagues or even e4xternal partners then a cloud based storage solution such as One Drive is your easiest option.
In a cloud based solution you can:
- Access files from anywhere with an internet connection, whether at work, at home. You can also usually configure it maintain a local copy of your computer which allows you to work offline and sync with the cloud version later.
- Your stuff is available 24/7 – it’s not dependent on the IT departments servers and whether they are being maintained or if they break down, or if they get destroyed by a fire, or if someone spills coffee on them.
- You can share with more easily with colleagues and even with external partners, deciding exactly who can access the document and whether they can view or edit. This can be controlled at the level of libraries, folders or even individual documents. You can even link it to conversations about the documents through Yammer.
- Cloud based storage is comparatively cheap.
- SharePoint libraries also allow you to much better tools to organize your data including being able to add metadata attributes that are either optional or mandatory to help you sort your documents (such as document type, organizational unit, subject classification and tags). And when your stuff is in the cloud it’s also much easier to search in particular the SharePoint Search and Delve are quite powerful tools to search not only the documents you entered, but anything else that you have access to. Delve is even smarter in that it uses the “Office Graph” i.e. the set of interactions you have with colleagues and the materials you work on to help predict which materials are most relevant to you.
- SharePoint has versioning control so you can see what was changed when by whom – very useful for ensuring you have the right version of a document.
- It takes a bit more work and technical expertise, but SharePoint also allows you to program workflow i.e. to automate common work processes for working on document (such as a chain of approvals) which means you can automatically make sure a document goes through a pre-defined work process, see where it is along the way and identify any bottlenecks in getting it completed.
But if you are to move to the cloud you probably are wondering how do you migrate all the years of accumulated files to your new platform. The simple answer is DON’T. Take a copy of your hard drive and put it on DVDs or a portable drive somewhere if you are really concerned you might need it again later – otherwise only take over those files that you are currently using or have referenced in the past 6 months – or for which you know exactly what it is and what you need it for. Forget about the rest – even if they seem potentially valuable, the truth is you will never use them. I’d recommend creating an entirely new system for filing and categorizing your documents – one that makes sense for how you organize your work now with the current document types, topics, work units etc. that you are currently using and forget about how it was done before.
The step of giving up on your old shared drive and creating something completely new in the cloud (i.e. not a migration of your existing mess) is itself one of the biggest benefits of making the change. Creating something that works for you right now that you understand, control and has what you need.
As you may have noticed. I’ve not been blogging much lately – It’s hard to find the time to blog in-depth on a topic, so I’m experimenting now with much shorter blog posts in an attempt to share more often.
Appropriately enough the first topic for a short blog is the death of blogging.
I’ve been trying to encourage colleagues to blog, particularly now as UNICEF has launched its own public blog and is planning on expanding this to allow offices and departments to run their own sub-blogs under the UNICEF umbrella. This led to a colleague forwarding me this article in the NYT asking whether blogging is dead (and people have been talking about the death of blogging since at least 2009).
I’d argue blogging isn’t dead – it’s just become one among many tools for communication. I started blogging inside UNICEF back in 2006 and publicly in 2010. Back in the early days blogging was an exciting new tool that was going to allow people to share their personal views unfiltered to the world. It was something anyone could do with little technical expertise and at little expense, and it was going to revolutionize and democratize communication. When I first started blogging it was something almost a bit counter-culture and also what the cool kids were doing.
Like any new technology or tool it suffered from the Hype Cycle, in particular it rode the peak of inflated expectations. But in the meantime a lot has happened. Lots of people tried blogging and gave up because it was too much effort, or the benefit wasn’t apparent enough, or they simply lost interest. Meanwhile a few star bloggers stood out and got famous, (including a few in the aid world), and then many absorbed by the mainstream (think Perez Hilton, Lifehacker, Guido Fawkes or FiveThirtyEight which became major properties in their own right). And mainstream media and corporates also got in on the act and started producing on-message, well written, slickly designed, well promoted blogs to burnish their image – to humanize it a little bit, but still while being firmly on brand and of course these overshadowed many of the original start-up blogs. New tools emerged – first Twitter, then Tumblr, Snapchat etc. and even inside the workplace with tools like Yammer and Slack that were even simpler and quicker, and of course more suited to diminishing attention spans – now even a blog is tl;dr.
So where does this leave blogs? Is it a good idea to start a blog or even to continue blogging? It can be. Even with the decline in blogging there are still a lot of them around, and many with interesting high quality content.
If you want to start a blog to be cool and ahead of the curve, to show your counter-culture street cred., or to become famous in your field of work blogging is unlikely to be a shortcut – that time is long past.
But if you want to have a medium were you can share your ideas and your work in a first person format; one that is shorter than a book, less formal than an academic article, more personal than a news story but with more substance than a tweet or a vine, then blogging is a great medium. Writing a blog can be a great way to refine your ideas, just through the act of writing, and engaging others to get feedback on it, even with a small following. For organizations it can be a great way to personalize your work and an alternative to slick corporate messaging which can often be a turn off. It can also be a great way to get your staff and your partners to talk about their work – their successes, but also their challenges – this can increase your outreach and make it more authentic, and also can give your staff a voice.
So it’s not so strange that UNICEF is launching a blogging platform after the hype and excitement of blogging is part – it’s actually a sign that blogging is now a mainstream tool for communication, one among many tools, but a valuable one nonetheless.
One thing organizations are not good about is figuring out how to retain or reuse knowledge of staff who leave the organization, whether through retirement or leaving for a job in another organization. Many organizations do very little at all to address this, but even for those that do, there are a few fundamental flaws in how organizations look at this problem:
1. They consider that whatever knowledge you generate or use while you are in a job belongs to the organization and so should be kept there
2. They consider knowledge to reside either in documents and products produced by staff – or in a slightly more enlightened state they consider that it resides in individuals and can be downloaded from the when they leave.
3. They create a firewall between current and former staff, including retirees making it hard to maintain personal networks and relationships that are often where the real source of knowledge exchange lies.
Probably the most effective strategy to retain knowledge is to be the kind of organization that retains and develops its best staff – and that could be a series of blog pots in its own right – but staff will leave whether through retirement, to seek new opportunities, or as a result of restructuring – so we need to think of ways to address this.
At KM World a couple of weeks ago (yes I’m still blogging about that) there were a few interesting presentations about how to tap into the knowledge and expertise of former staff – in particular a very interesting one by Lesley Schneier a retiree of the World Bank about a system they had set up to tap into the expertise of Bank retirees to support current projects on a pro-bono basis, building on the enthusiasm of many retirees to keep in touch and be helpful and the interest of current staff to get free expert advised from seasoned professionals. There were lots of good ideas on how to make us of the desire of retirees to stay involved to support current work but also a couple of important lessons on the challenges too.
One challenge at the Bank is the interaction between staff and retirees takes place on an externally facing platform which is separate from the platforms that staff use in their day-to-day work (because I presume IT security?) which means that it is much more of an effort for current staff to go there, and many don’t even know that it is an option.
I don’t have any magic solutions to the problem of “knowledge walking out the door” but it seems to me that there are a few practices we could try to change which could make an important difference:
Most IT systems use your current official email as a means of authenticating you into social platforms. This is great as it is a good shortcut to check whether you are a current employee and so should be permitted to be part of the internal intranet and all the secrets and confidences in contains – but when you leave you lose your official email and access to your internal resources, but also to whatever internal social networks you have.
However in reality the nature of a social network is that is social, that is it is based on personal and professional relationships that have developed over time. Once you have left an organization your social connections (and willingness to collaborate with them) do not disappear overnight with a change of job or of email address. Neither do your past contributions suddenly become useless – even if over time they might decline in relevance.
So a couple of practices to consider might be to allow some kind of continued access to communities or similar sites after you retire or leave the organization. Obviously there would be some things which are too internal or confidential to share, especially in commercial operations; but in my area of work, international development, any technical discussions on how to do our work better should not be confidential and so there is no reason to lock people out after they leave the organization – after all they may still be willing to provide free advice to support current projects.
Maybe an even better way to handle this would be to create communities that are by default open in the first place and have your staff use those for non-internal non-confidential collaboration – that way you can more easily engage with and learn from external experts even if they have never worked for your organization – and also your internal expertise can be of benefit to others without needing formal cooperation thus increasing the influence of your knowledge work.
One particular bad practice I have seen that should be avoided at all costs is to delete a persons contributions to online discussions after they leave an organization, or after they choose to leave a network, or to delete their user profile. It’s useful to mark on someone’s profile if they are currently working for the organization and if they are currently active in the network as this is useful information when reading the contributions – but don’t delete either the contribution or the profile of the person who created it.
Another challenge with retaining knowledge is the handover process. In some places we don’t have any consistent handover procedures at all and if there is a large gap in hiring (as is often the case in the UN) then a newcomer is often forced to try to recreate from scratch the knowledge base of their predecessor. A few thoughts on this:
1. It’s good to have some standard handover process in place with some standard elements of what should be included and a timing to do this that is before the last minute. Even better if these are in some widely accessible database. Better still if there are some “teeth” to make sure that this is complied with. This could be as simple as a standard template – but there are also some more creative techniques out there involving interviewing, video etc. that make it more alive and less procedural.
2. It is however unreasonable to expect that someone can download all relevant knowledge into a form or interview since a lot of the important tacit, just in time knowledge can’t be captured in a handover note – in fact you don’t always know what is most important until you need it. Having human contact between someone leaving a post and a new person coming can help a lot with this. Ideally having an overlap when two people are in post together so that the newcomer can see in practice how the job is done can be extremely valuable, and it would be good if we could change our hiring practices to allow for this – especially for critical office positions that require a lot of “feeling” for the job that can’t be easily captured in written form. If an overlap isn’t possible then a face to face handover meeting, or better still a meeting with some follow-up calls can greatly help.
3. Lastly since this is “World Out Loud Week” – one of the best ways to mitigate against knowledge loss is not to try to capture everything when the person leaves, and when memories of important events and learnings have often been retrospectively interpreted to fit your own world view – wouldn’t it be better instead to capture knowledge all the time as you work? That way it is much easier to see the individual steps on how a project was developed, how and when key decisions were made and what were the key drivers. Encouraging all staff to work out loud (inside the organization if they are not willing or able to do it publicly) and making all that available to the rest of the organization to learn from is one of the best ways of preserving living knowledge that comes from how we actually do our work (rather than how we talk about it afterwards).
I’m currently participating in a “UN Transformational Leadership course” which I’ve mentioned in past blogs. One of interesting self-discoveries I’ve made from this course is that sometimes we are the ones that create the barriers to change or to pursuing our big goals.
Through some introspection I realized that one of the biggest challenges I face is that I find it hard to say no when people ask me to help them. And the more I help people, the more people ask me to help, and the less focused I am on pursuing my own goals.
This desire to help others isn’t entirely altruistic. Like most people I want to feel valued, and like many I want recognition for what I do and for my expertise. I’m probably also a bit of a procrastinator. This has translated into me looking to help and advise others which has then translated into more and more external asks. Having a public blog and social media accounts and working in the field knowledge management also means that I get a lot of requests for help.
So what would be a good, healthy way of managing this, without stopping being helpful as at all? Here is a little framework I’m prototyping for myself. This time I’m asking for YOUR help to give me feedback and share your tips for dealing with “too many questions”.
To start I realized that my time is broadly divided into three major blocks 1. Work 2. Family time (as a husband and father of three) and 3. sleep. Those are all pretty important, and anything that doesn’t fit within those categories takes valuable time away from them, so I need a few good measures as to how to think about requests for my time. If I were a consultant, or worked for employer that allowed outside activities then an important category might be “do I get paid?” but I can’t get remunerated for what I do beyond my salary, and that doesn’t depend so much on how helpful I am.
So here are the questions I plan to use to screen requests for help:
1. Is it directly related to my job? Is it explicitly part of my workplan, my job description, or is it at least related to the broad purpose of my job. If not, does it at least contribute to the priorities of and mission of the UN?
2. Do I know you? How do I know you? – are we friends, colleagues or past collaborators? Is our current or future professional relationship likely to be mutually beneficial (even if not equal), have you helped me in the past?
3. Is your question interesting? Are you asking me something I’m curious about myself, or something that I’m passionate about either professionally or personally?
4. Is it something that I can easily answer? Is it related to my expertise? Easier questions are more likely to get answered than complex involved ones, but at the same time, very broad generic questions (tell me everything about KM) are less likely to be answered. It’s also not good to ask a question for something you could have easily researched yourself.
5. Will my input be useful and used?, and will I get any feedback on it? I can’t tell you the number of times I’ve been asked for inputs, never to hear what happened to them, or even what happened to the project they were requested for.
6. I hate to finish with this but — Is there something in it for me? Not money since I can’t take it. But will it help me develop my skills and knowledge? Will it help me in my current work? Will it help me make new connections? Will it help me develop my career or find my next job? Or at least do I get more than a thank you e-mail?
I’m going to try to run future queries through this filter and only take on those that score well on the above criteria. I’ll let you know how it goes. If I don’t respond to your e-mail don’t be offended, I want to help, I really do – but I also have to get stuff done.
P.S. Here is a sample of the types of questions I regularly get:
Can you help me get a job? Can you review my CV? Can you comment on my KM strategy?, Can you comment on my publication?, Can you tell me who in the UN (or elsewhere) is working on X? Can you tell me what software tool I should use for Y? Can you answer my (10 page) questionnaire? Can you tell me how to do Z knowledge management related task? Could you give me feedback/input on my project? Can you tell me where to find all the research on A? Could you help publicize my publication/project? Can you speak at our conference (and pay for your own travel)? [addendum since I wrote this bog: Can you meet me to check out/help me market my new must have KM related software]
In a future blog I’ll give my generic answers to some of these questions, so you can read those before asking me
I started writing this blog post in frustration at hearing once again the mantra in the aid world, and particularly in the UN of need for us to be doing “Evidence based policy and programming” and discussions and laments about why we don’t yet have this and what to do about it. But then this morning I saw two excellent blog posts by Kevin Watkins of ODI and Owen Barder of CGD on Jim Kim’s “Science of delivery” which also seem to touch on some of the same frustrations.
It seems to me that at least part of why this discussion remains frustrating is that we don’t agree on what evidence is, and what it’s role should be in policy or programming.
At first glance it might seem obvious that our policies and programmes should be “firmly based on the best available evidence” and that wherever the ‘best available evidence” isn’t good enough we should be doing more research to fill the gaps. And it can be quick to jump to the conclusions that we need to invest more in research, carefully designed experiments (such as RCTs) and independent evaluations and on the other to put in place incentives or rules than encourage or even force policy makers and programme managers to use the evidence that is available.
While all this might be useful, I don’t think it will be the silver bullet that some think it will. If this is so self-evident then it begs the question as to why we don’t already practice what we preach. Why isn’t aid work primarily if not exclusively evidence-based?
Firstly, however much we invest in research and evaluation there are limitations to what we will know (see my earlier post the truth is out there for a more detailed discussion of the limits of what we can know). Some interventions lend themselves to rigorous experimental design or data analysis (e.g. cash transfers, medicines) while others are harder to do as experiments (e.g. policy reforms) or to research cost effectively.
Secondly incentives to use, or not to use evidence also matter. It’s not enough to improve the packaging and dissemination of new knowledge, or to make using knowledge part of procedure – there are many other factors that govern whether or not knowledge is used both individual and institutional e.g. focusing on implementation rates reduces the incentives to take the time to learn from research or other experiences. (See my earlier post “creating a demand for knowledge?” for more ion the challenges of getting people to use what is already available)
Thirdly, context matters. In most cases it’s simply not enough to exactly copy a “best science based practice” from one location and directly replicate it in another. At best the approach will need to be tailored or adapted to a new context, at worst the approach may not work at all. Kevin Watkins mentions the importance of politics and how different political structures and power relations can have an important impact on how effective a “scientifically sound” approach might be in practice, and you can’t factor this into an experimental design. Owen Barder also highlight how the traditional “evidence-based approach” also fails to deal with complexity i.e. that there are complex and dynamic relationships between the context and the programme which evolve together in unpredictable ways for which a static approach to using evidence and best practice is ineffective.
But I’d add an additional dimension to the issue of context and complexity. Programmes are implemented by individuals, not just by institutions, and individual people bring their own layer of complexity to a problem which is hard to measure or control. Different project managers performing the same role can have different levels of technical skills, but also different personal motivations, and different personal relationships with other key players in the project – and these can interact with the project in unpredictable ways. What’s more we tend to overlook the fact that someone may perform well in one context with one team yet perform poorly in another situation – so it’s impossible to standardize how exactly a project team will work on the project unless we reduce all elements of judgment and unpredictability – transforming programme management to a production line function.
Yet I don’t think all is lost. There are some additional things we can do to improve how we learn and implement apart from investing more in research. Here are a few of them;
1. Take the broadest possible view of what “evidence means”. Evidence might be rigorous research, but it might also be case studies, stories, ethnographic studies, analysis of “big data”. Take a broad view but recognize what each type of “evidence “ us useful for and what its limitations are. don’t overlook key areas such as political context and power analysis.
2. Do more pilots and dynamic experiments – encourage more experimentation – not only in the sense of control-design experiments but also in the sense of coming up with lots of possible ideas, then trying them out, adapting them as you go and building on those that are yielding results.
3. Start from, but don’t blindly copy past practices – encourage use of existing knowledge and experience – but as a starting point that will be built upon and deliberately and continually adapted.
4. Encourage ongoing collaboration and sharing of tacit knowledge between practitioners as well as sharing of explicit research or evaluation reports and results.
5. Work with and learn from beneficiaries – they will often have insights into why something does or doesn’t work, or what might be done to address a problem that an outsider cannot see.
6. Keep a diary with detailed ongoing records of what is happening on the ground both internal and external factors to help generate enough material to get a useful insight into what is working and why. Encourage self-reflection by those involved in the project based on this information. But you can also use this as a key source for more rigorous independent analysis.
7. Work on incentives – make sure that incentives are there to generate and to use evidence but also personal insights and experience.
8. Adapt – rather than looking for an idealized approach to solving a problem once and for all – keep searching for improved ways to solve it that work in the context where they are applied and keep modifying and varying your approach based on what you are learning.
I’m not sure if delivery can be entirely evidence-based – but evidence informed and learning-based would be a good start.
Way back in the mists of time when I first started my professional career in DFID we had but a few places from where we would get instructions and requests or where we could find the information we needed to get our work done. Basically we had a full set of procedure manuals on the shelf, an inbox for inter-office memos and the circulation file for general “might be of interest” announcements and interesting articles as well as two month old copies of The Economist and Paris Match (because we were all supposed to be practicing our French).
Now when I come in to the office (or more likely shortly after I wake up) I scan through the following: work e-mail, private e-mail accounts, Twitter, Facebook, Teamworks (UNDP’s internal social network), the Coordination Practice Network, our office Intranet, Feedly (for my RSS feeds both news and blogs I follow). Less regularly I also check LinkedIn, my old UNICEF e-mail (yes I still get messages there), KM4Dev, Zunia and a whole host of other networks and communities I signed up for at one time or another. And that’s not to mention text messages, instant messages (on at least three different platforms), Skype and telephone. The only thing I don’t check is my office mail slot – because I never get anything there.
Overall the greater access to information and diversity of channels is a very positive thing. I’m able to find out about things and respond to them much more quickly and get access to a much wider range information and expertise to help inform my work than I could have even imagined when I first started working. And the fact that this is almost real-time allows discussions, collaboration and feedback loops that were all but impossible before.
But one big peeve with this Brave New World is the fragmentation of all this information, and the challenge of keeping up to date with it all, knowing which things to follow and managing my time to check so many different inboxes and networks. Just how many social networks do you need to follow?, and how often do you need to check them to keep in touch with colleagues and keep up to date on the latest thinking? What are the chances that you are NOT checking precisely that network where the contact or information you need is most easily accessible? And what advice do you give to your late adopting tech-wary colleagues about where they should put their efforts and expend their limited attention (and why should they bother to be part of your community when they are already struggling with so much stuff)?
Wouldn’t it be great if you could go to one place to get all your development related updates and messages? (or whatever topic it is that interests you). Can’t we all our contacts use the same set of tools? But if this is such a simple idea why don’t we have it already?
One attempt to solve this problem has been through the development of knowledge portals. Many organizations have tried and failed to produce knowledge portals that have “everything you need to know about X”. Curating “everything” about a topic is a monumental effort that is unlikely to find everything that is out there, and also such a portal will necessarily reflect the views and information architecture preferences of the curator which will not fully meet the needs of a diverse group of users. Competition between different portals can also confuse things and sometimes even means that people are not willing to share their content with each other forcing users to follow multiple portals defeating the original intent.
Another approach at the organizational level is to pick a software platform that “does everything”. Many software companies are keen to sell corporate platforms that include social collaboration, messaging, content feeds often laid over work process software – common examples being done through SharePoint. These are usually expensive custom creations within any organization and they don’t always play nicely with external software tools, especially social ones.
The other fix that many of us use is to get e-mail notifications from our various systems and funnel them to a single inbox. The challenges with this are that your inbox quickly becomes overloaded with diverse updates, some systems don’t provide e-mail updates or at least not useful ones and for me at least I still find the need to split these systems somewhat artificially into my work and private e-mail accounts (e.g. twitter – is it work or private?) also because of limitations on the size of our work inboxes and concern about how “private” my work e-mail really is.
Some advocate for moving away from e-mail altogether. Something I’d like to see in, in theory, but it only works if everyone you collaborate with also moves off e-mail and agrees on a common platform (or small set of platforms) to use – something that seems highly unlikely in my work environment where every little thing is done by e-mail and where social platforms are still used by a relatively small “enlightened” few – and many of those are internal platforms that don’t easily allow collaboration across organizations.
What I’d really like to see, but haven’t yet, is a software tool that would allow you to have a personal “mega-inbox” or dashboard that would combine together all the various streams that I’m interested in, business and personal, my e-mails, social networks, RSS feeds and monitored intranet pages. Ideally such a tool would be portable i.e. accessible by the web and not tied to my employer so I could take it with me if I switch jobs (although my access to feeds from internal sources wouldn’t continue if I leave an organization). It would allow us each to use the tools and networks we feel most comfortable with but be able to add new ones without learning new tools and new logins. What this would need is a common system for providing and authenticating feeds and for processing user postings that could work across all of the most common platforms.
This could be an enormous productivity boost for all of us, and a great opportunity for some enterprising software developer out there – so what are you waiting for!
I’ve been reading with interest Bill Gates “Annual letter” on why measurement matters in aid and development – and finding myself both nodding and shaking my head at the same time.
Of course there is a lot to be said for better measurement in development work. A few of the important reasons why we should care more about measurement include:
1. Accountability – donors, (and beneficiaries) are increasingly demanding to know where the aid money is going and what is being achieved with it – and greater transparency and accountability makes it easier to see how aid agencies are doing, to compare them and to invest wisely. It also creates powerful disincentives against poor management and corruption (hey you! – you know we can see what you are doing – right?)
2. Knowing how you are doing – If you have a clear plan in place with identified outcomes and a good monitoring/ measurement system to support it then you can keep tabs on your programmes and see how well they are progressing, quickly identify problems and take steps to get back on course. You know where you are going – and how far along the way you are. While it seems obvious that every programme should have this in practice too many programmes have unclear objectives and poorly formulated monitoring indicators – often because there isn’t sufficient thought and resources put into ensuring the indicators can be collected and analyzed regularly and in a timely manner.
3. Testing hypotheses – if you want to know whether a particular intervention will work, or how impactful it will be under what circumstances, or if you just want to understand how different aspects of development are interrelated – then to do this “scientifically” you will most likely need to use research methodologies that are heavy on data collection (such as randomized control trials).
4. Learning – and if you are running a project you can also use measurement to learn from your programme as you go. Not only to identify small course corrections on your plan – but also to adjust your plans based on actual experience and to identify positive and negative spillovers from your project which you might not have anticipated.
5. Bottleneck analysis – this involves looking at each element of a system and monitoring to see which elements of the chain are most responsible for failing to reach development goals (kind of a six-sigma or quality circle approach to aid) and thus help decide where to focus your attention to get the best results. An example would be for primary education you might look at i) education budgets ii) school construction iii) teacher training iv) curriculum development v) cost barriers and incentives such as school fees, uniforms etc. vi) culture and attitudes (e.g. on girls’ rights to education). Looking at each aspect of the system you can see which of these are the most critical barriers (or bottlenecks) to achieving greater coverage and focus on addressing those first.
Most of these issues are not new, but the possibilities to do measurement have never been better what with big data, real-time monitoring, SMS reporting, opinion polls, really simple reporting, beneficiary storytelling, satellite imagery, RFID tracking etc. in addition to better traditional data as statistical capacity and evaluation methodologies continue to improve.
So why the head shakes? For me the Gates letter while advocating measurement and science seems a little “faith based” in its level of belief in what better measurement can achieve – as if better measurement alone would be enough to change the world. But there are a few pretty strong reasons why that’s not the case:
1. Not everything worth doing can be easily measured. In fact there is a danger of focusing on what can be relatively easily measured (e.g. vaccination rates) while not focusing on things which are important but hard to measure (like human rights). There are already some calls in the post-2015 discussions to stick with the MDGs because we already more or less know how to measure them. (For more on what can be measured and what can’t see one of my first and most popular blog posts “The truth is out there”)
2. Complexity – Measurement is often used in too much of a linear or reductive way looking at each step in a programme as an independent issue to model, manage and benchmark. But in reality different elements of an environment are interrelated in unseen ways and these also might vary between different contexts that on the surface appear similar. Taking the issue of education – increasing the number of schools and teachers might for example also decrease the quality of teaching and thus have a negative impact on enrollment and even outcomes.
3. Having the data is not a guarantee of acting on it – failure – a strong focus on measurement and accountability in a difficult funding environment means that there is little willingness to admit and learn from failure in the fear that this will mean that organizations and programmes that are unsuccessful – but are learning from those mistakes and providing valuable lessons for others will be defunded. Ed Carr has an excellent analysis of this issue in his blog on the Gates letter.
4. Political will – knowing is a first step, but without a will to address the issues identified it won’t take you far. Too much monitoring, evaluation and research goes unread since it isn’t in a language or format decision makers can understand and use. Communication of results is one part of this. Mobilization is another. Sometimes more anecdotal evidence is better for persuading decision makers and the public – or at least hard data – but backed up by real life examples. Sometimes despite the data being clear, there just isn’t the political will to address the difficult issues raised or the leverage to put pressure on decision makers to take the hard choices they need to. Looking at evidence without looking at power and politics and who is influenced by what and why won’t take you very far. Unfortunately just because you can create or access knowledge isn’t any guarantee it will be used (See creating a demand for knowledge for more on this)
5. Evidence isn’t neutral – a follow on point to the last one is that the use of data and knowledge or even its creation is very political. Deciding what is important to collect, who will use it and how – or deciding as a funder or senior aid agency manager which data “matter” or what evaluations really mean about the success of a project are highly subjective and often depend on other information than what appears to be being discussed objectively (such as who is involved with the project as funder, manager, evaluator and beneficiary and the relationships between those actors).
So measurement is important and its good that Gates raises it and promotes it. But while it may be necessary to improve measurement to improve aid, and while we may have unprecedented opportunities, it’s far from sufficient – and without thought to the limitations of measurement – and the necessary complementary actions in the area of people, politics and power it may not achieve very much.