Methodologies

Ask the Survey Doctor: Non-Economic Evaluation — Measuring Outcomes and Impact of Governance, Social, or Security Programs

By Craig Charney | Insights | Series II | No. 3 | February 2014

If you are struggling with surveys, evaluations, assessments, or market research in developing countries, email the Survey Doctor with your question. 

Q: I’m stumped by the lack of standard methods to measure and evaluate the non-economic outcomes and impact of development projects, particularly in the social, governance, or security spheres. How can we measure the results?

A: It’s tough enough to estimate the return on investment for economic development projects — so it’s understandable that development professionals tear their hair about showing results from other types of programs. Indeed, the scarcity of hard data on the outcomes of democracy, health, or security sector reform work is one of the pretexts for claims that development aid is a waste of funds.

But the right kind of evaluation can solve this problem. 

The first requisite for effective evaluation — of any project, economic or non-economic — is a clear theory of change. What is your program ultimately attempting to accomplish? What are the levers it will pull or the things it will do to try to achieve them? How can you tell if it actually has done those things? What is the process by which they will work through to the ultimate result you seek? What are the intermediate steps or signs you can observe that indicate that the change process is underway?

If you can answer these questions you are well on the way to measuring the effects of your program. The key factors you are trying to measure are the results you seek, the initial conditions, and the intervening elements. In social science terms, we have the key variable (what you’re trying to change, measured both at the end of the project and the beginning) and the intervening or intermediate variables (the chain of things that is changed to finally alter the key variable).

What you’re doing can be diagrammed this way:

When you have a sound theory of change, you can take the next step: deciding how to observe change in these factors. This means looking for observable signs of what is happening to your key variable and the intervening variables. In evaluating projects whose final results, in terms of the key variable, will take a long time to be observed, measuring the intervening or intermediate variables is particularly important to determine whether the project is on track.

How can this be done? For economic projects the result is easy — you can measure the inputs and the outputs (or savings) in monetary terms. 

For most democracy promotion, social and security sector evaluations, there are intervening factors and final results that can be observed and quantified, usually on a type-of-project basis.

For instance, in governance, consider voter education projects, intended to help explain election processes and safeguards and deter electoral malpractices (the intervening variables), in order to make elections themselves more legitimate (the key variable). Surveys can measure the initial states of election knowledge, attitudes to malpractice, and confidence in the elections. Then follow-up work can look at exposure to the program, changes in knowledge and attitudes among those exposed, along with the ultimate legitimacy of the elections — immediately afterwards and sometime later.

How about evaluation of a health project that aims to promote better health (the key variable) through better water service (the intervening factor)? First step is to measure water service and use and target health conditions (like frequency of colds or water-borne diseases) before the project. Then you need to find ways to measure how much water service has changed after (color, smell, how many stories it goes up in high rises). You also should measure changes in water usage — for instance, number of daily hand-washes or showers — and change in the rate of the health conditions that interest you.

What about evaluating security sector reform projects? Take, for instance, a program aimed at beefing up police numbers and improving their behavior (the intermediate scene from Egyptian voter education TV campaign, 2011 variables) in order to improve public safety and security. Once more, start with the initial states: how often police interact with the public, how those interactions play out, how the public interacts with police, and the public’s experience of crime, feelings of security. During and after the project, it’s possible to measure changes in how often people interact with police, how they behave, how crime rates change, and how perceived security and resulting behavior change.

One thing is common to all these scenarios: reliable evaluation implies baseline and follow-up studies, which are all too rare. The basic principle of evaluation is a constant: find out the changes produced by your intervention — which means you need to know the states before and after. (There are some ways to fake it if you don’t have a baseline, but none are as reliable as a good initial study.)

Yet most development projects and proposals don’t include baseline and final evaluations, our 2013 Development Evaluation Survey found. More than half of all development projects and proposals (52%) lacked a final evaluation altogether — eliminating the possibility of measuring results. Only 25% of evaluations included both baseline and final evaluations.

In short, well thought-out projects and good evaluation practices can shed light on the impacts of both economic and non-economic development projects. But they do mean thinking about the change process and evaluation from the project start — not adding on a study as an afterthought at the end.