This article is the second part of our 2-part series on impact evaluation. In the first article, “ Impact evaluation: overview, benefits, types and planning tips,” we introduced impact evaluation and some helpful steps for planning and incorporating it into your M&E plan.
In this blog, we will walk you through the next steps in the process – from understanding the core elements of an impact evaluation work plan to designing your own impact evaluation to identify the real difference your interventions are making on the ground . Elements in the work plan include but are not limited to – the purpose, scope and objectives of the evaluation, key evaluation questions, designs and methodologies and more. Stay with us as we deep dive into each element of the impact evaluation work plan!
Developing an appropriate evaluation design and work plan is critically important in impact evaluation. Evaluation work plans are also called terms of reference (ToR) in some organisations. While the format of an evaluation design may vary on a case by case basis, it must always include some essential elements, including:
This section provides information on the background of the intervention to be evaluated. The description should be concise and kept under one page and focus only on the issues pertinent for the evaluation – the intended objectives of the intervention, the timeframe and the progress achieved at the moment of the evaluation, key stakeholders involved in the intervention, organisational, social, political and economic factors which may have an influence on the intervention’s implementation etc.
Consultation with the key stakeholders is vital to determine the purpose, objectives and scope of the evaluation and identify some of its other important parameters.
The evaluation purpose refers to the rationale for conducting an impact evaluation. Evaluations that are being undertaken to support learning should be clear about who is intended to learn from it, how they will be engaged in the evaluation process to ensure it is seen as relevant and credible, and whether there are specific decision points around where this learning is expected to be applied. Evaluations that are being undertaken to support accountability should be clear about who is being held accountable, to whom and for what.
The objective of impact evaluation reflects what the evaluation aims to find out. It can be to measure impact and to analyse the mechanisms producing the impact. It is best to have no more than 2-3 objectives, that way the team can explore few issues in depth rather than examine a broader set superficially.
The scope of the evaluation includes the time period, the geographical and thematic coverage of the evaluation, the target groups and the issues to be considered. The scope of the evaluation must be realistic given the time and resources available. Specifying the evaluation scope enables clear identification of the implementing organisation’s expectations and of the priorities that the evaluation team must focus on in order to avoid wasting its resources on areas of secondary interest. The central scope is usually specified in the work plan or the terms of reference (ToR) and the extended scope in the inception report.
Theory of change (ToC) or project framework is a vital building block for any evaluation work and every evaluation should begin with one. A ToC may also be represented in the form of a logic model or a results framework. It illustrates project goals, objectives, outcomes and assumptions underlying the theory and explains how project activities are expected to produce a series of results that contribute to achieving the intended or observed project objectives and impacts.
A ToC also identifies which aspects of the interventions should be examined, what contextual factors should be addressed, what the likely intermediate outcomes will be and how the validity of the assumptions will be tested. Plus, a ToC explains what data should be gathered and how it will be synthesized to reach justifiable conclusions about the effectiveness of the intervention. Alternative causal paths and major external factors influencing outcomes may also be identified in a project theory.
A ToC also helps to identify gaps in logic or evidence that the evaluation should focus on, and provides the structure for a narrative about the value and impact of an intervention. All in all, a ToC helps the project team to determine the best impact evaluation methods for their intervention. ToCs should be reviewed and revised on a regular basis and kept up to date at all stages of the project lifecycle – be this at project design, implementation, delivery, or close.
Impact evaluations should be focused on key evaluation questions that reflect the intended use of the evaluation. Impact evaluation will generally answer three types of questions: descriptive, causal or evaluative. Each type of question can be answered through a combination of different research designs and data collection and analysis mechanisms.
Examples of key evaluation questions for impact evaluation based on the OECD-DAC evaluation criteria.
Key Impact Evaluation Questions based on the OECD-DAC evaluation criteria
Measuring direct causes and effects can be quite difficult, therefore, the choice of methods and designs for impact evaluation of interventions is not straightforward, and comes with a unique set of challenges. There is no one right way to undertake an impact evaluation, discussing all the potential options and using a combination of different methods and designs that suit a particular situation must be considered.
Generally, the evaluation methodology is designed on the basis of how the key descriptive, causal and evaluative evaluation questions will be answered, how data will be collected and analysed, the nature of the intervention being evaluated, the available resources and constraints and the intended use of the evaluation.
The choice of the methods and designs also depend on causal attribution, including whether there is a need to form comparison groups and how it will be constructed. In some cases, quantifying the impacts of interventions requires estimating the counterfactual – meaning, estimating what would have happened to the beneficiaries in the absence of the intervention? But in most cases, mixed-method approaches are recommended as they build on qualitative and quantitative data and make use of several methodologies for analysis.
In all types of evaluations, it is important to dedicate sufficient time to develop a sound evaluation design before any data collection or analysis begins. The proposed design must be reviewed at the beginning of the evaluation and it must be updated on a regular basis – this helps to manage the quality of evaluation throughout the entire project cycle. Plus, engaging with a broad range of stakeholders and following established ethical standards and using the evaluation reference group to review evaluation design and draft reports all contribute to ensuring the quality of evaluation.