USAID Evaluations at Five: Known Unknowns and Unknown Knowns

What happens to a U.S. Agency for International Development (USAID) evaluation after it has been completed? Does it sit on a shelf, disappear into an online netherworld, or actually get used to shape programs and inform decisionmaking?

Five years into its landmark evaluation policy, USAID has conducted a fascinating study that looks at precisely these questions. Clocking in at 226 pages (including annexes), Evaluation Utilization at USAID is a tough slog. But it contains riches of information about what we know, what we don’t know, and what we know we don’t know about the impact of our evaluations and the programs they examine.

1. We know that evaluations are being used. The survey found that 93 percent of evaluations sampled had stimulated learning, and 71 percent had been used to design or modify a USAID activity or project. Not surprisingly, better quality evaluations were used more, as were those that were timely for decisionmaking.

Given that most evaluations were conducted late in the program cycle, however, results were most often used for new project and activity design rather than for midcourse corrections.

2. We wouldn’t have known as much without the evaluations. While most interviewees said that evaluations confirmed things they already knew, 52 percent of evaluations contained at least some information that was previously unknown to USAID staff.

3. We know more about the performance of our activities than about the performance of our policies. In the overwhelming majority of cases, evaluations were being conducted at the individual activity and project level, where impact tends to be limited, rather than at the sector or program level, which aim at higher-level results. More disturbingly, the study did not find a single relevant evaluation at the policy level, concluding, “Evaluations of USAID policies are almost nonexistent, and evaluations are rarely used to inform the formulation of new policies.”

4. We don’t know what impact our programs are having. While there are many acceptable types of evaluations, and impact evaluations are not appropriate in all instances, USAID has relied excessively on performance evaluations, which composed 97 percent of the sample. Out of 609 discrete evaluations conducted between 2011 and 2014, only 8 were impact evaluations, and only 13 were ex-post evaluations. Impact evaluations are of special interest because they measure the extent to which observed outcomes are directly attributable to specific interventions, and ex-post evaluations are important because they measure the extent to which outcomes are sustained after the interventions are completed.

5. Our partners don’t know what we know—and we don’t know what our partners know. One of the study’s more depressing findings was that USAID evaluations generally are not being planned with or shared with local partners.

While USAID almost always disseminates evaluation results to its own staff, it does so to implementing partners only 76 percent of the time and to country partners only 43 percent of the time. Just 24 percent of the evaluations were planned with the involvement of the country partner.

This omission is counterproductive not just because country partner awareness of evaluation findings can help to increase the effectiveness and sustainability of USAID activities, but also because it can stimulate learning and inform decisionmaking for the country’s own development efforts. Moreover, the analysis found that “learning is higher for USAID when country partners participate in the evaluation processes.”

Why are we not collaborating with local partners and sharing results, beyond posting evaluations online? The excuses that the study’s authors reported hearing—that government ministries weren’t involved in the activities being evaluated, and that USAID staff were “too busy with other obligations”—are almost too pathetic to bear repeating.

6. We don’t even know what we know. USAID’s online library of evaluations, known as the Development Experience Clearinghouse (DEC), contains a wealth of information, but posting evaluation reports online does not ensure that knowledge is shared or lessons are learned. In fact, delivering evaluation reports to the DEC “was not statistically linked to utilization by USAID or its partners.” Helpfully, the study includes numerous recommendations for making evaluation findings more accessible and digestible.

7. We sometimes don’t even know the right questions to ask. Too often, evaluation questions are poorly crafted, limiting the quality and utility of the findings. The study recommends issuing guidance on how to write good evaluation questions and how to construct actionable and practical evaluation recommendations. In addition, there appears to be a shortage of highly qualified evaluators among the contractors hired to provide evaluation services, as well as a dearth of well-qualified local team members in many countries. As a result, nearly half the units surveyed reported inconsistent or low-quality evaluations as an obstacle to their use.

8. We know we need to plan ahead. Almost half of those surveyed reported that evaluations came too late to influence key decisions or were no longer relevant to a rapidly changing situation on the ground. Part of this problem is due to the excessive length of time it takes to complete procurements at USAID, but more importantly, it is due to a failure to build evaluations into the program cycle. In order to be certain there is a clear program logic, accurately measured baselines, meaningful indicators, and appropriate program design, evaluations need to be planned from the very beginning of a program, project, or activity.

9. We neglect to act on what we know. While post-evaluation meetings are fairly routine, there is a big drop-off when it comes to accepting or rejecting recommendations, deciding on actions to be taken, preparing an action plan, designating responsibilities, setting a time frame, and tracking implementation. The report provides helpful templates for summarizing evaluation recommendations and tracking action status, but USAID will need to endorse, adopt, or adapt these templates and require their use. The agency should also establish clear and meaningful targets for utilization of evaluations, thus helping to instill and shape a culture of learning.

10. We know what needs to be fixed. In its five-year review of evaluation practice at the agency, USAID recognizes most of the shortcomings and impediments highlighted by the evaluation utilization study and describes its efforts to rectify them. For instance, USAID has:

• issued program guidance that includes evaluation planning from the very beginning;
• trained more than 1,600 staff in evaluation;
• conducted a meta-review of evaluation quality and coverage;
• begun recruiting Monitoring and Evaluation (M&E) fellows;
• established several evaluation communities of practice;
• designated an M&E point of contact for every bureau and independent office;
• joined international initiatives to promote state-of-the-art evaluation methods; and
• required each Washington bureau to develop an evaluation action plan for 2016.

Looking to the future, USAID plans to:

• host impact evaluation learning clinics in key sectors;
• offer new training in performance monitoring and evaluation and strengthen existing training modules on evaluation;
• encourage project-level evaluations for higher-level results;
• develop gap maps, synthesize evidence in particular sectors, create learning agendas, and include learning plans in Country Development Cooperation Strategies; and
• commission another meta-evaluation to determine whether and how evaluation quality has improved from 2012 to 2016.

These steps are impressive, and welcome. They are part of the reason why the study concludes that “USAID evaluation utilization practices are already strong and compare well to those of other USG agencies.” But they overlook what is perhaps the most important gap of all: local participation. USAID should issue tools and guidance for incorporating country partners not just in gathering feedback during evaluations, but in designing, implementing, and utilizing evaluations. This will almost certainly need to include partner training and capacity building. But without bringing local partners into the picture, we miss the entire point of the exercise, which is not just to make aid better, but to make development more sustainable.

Diana Ohlbaum is a senior associate with the Project on Prosperity and Development at the Center for Strategic and International Studies (CSIS) in Washington, D.C.

Commentary is produced by the Center for Strategic and International Studies (CSIS), a private, tax-exempt institution focusing on international public policy issues. Its research is nonpartisan and nonproprietary. CSIS does not take specific policy positions. Accordingly, all views, positions, and conclusions expressed in this publication should be understood to be solely those of the author(s).

© 2016 by the Center for Strategic and International Studies. All rights reserved.