There are four important roles to distinguish within the evaluation process: (a) the evaluation commissioner; (b) the evaluation manager; (c) the evaluator; and (d) the evaluation user.

	The evaluation commissioner is the party or stakeholder who decides that an evaluation should take place. This could be the IOM programme manager, relevant IOM chief of mission (CoM), a thematic specialist or unit(s) from Headquarters and/or from a regional/country office, the donor or any combination of these stakeholders.
	The evaluation manager is the person who is in charge of managing the evaluation. It is possible that the evaluation manager is from the same entity or office that commissioned the evaluation. Most often in IOM, the evaluation manager is the programme or project manager. It is important to note that, at times, several stakeholders may be part of an evaluation management committee, overseeing the evaluation process together.
	An evaluator is charged with conducting the evaluation. Evaluators can be external consultants, IOM staff or evaluators recruited by IOM, donors, partner organizations and governments.
	The evaluation users are key players for guaranteeing the full utilization and benefits of evaluation. They can be direct users, who are, for instance, directly concerned with the implementation of the recommendations and accountability purposes, as well as indirect users that can be more interested with the learning dimension of the evaluation.
In addition to these roles, there are other stakeholder engagement and reference groups that play an important role, for instance in terms of quality assurance. For further information on reference groups, see Information box.
	The United Nations Evaluation Group’s (UNEG) Norms and Standards for Evaluation (2016) further elaborates on stakeholder engagement and reference groups.8  Specifically, the document states that “inclusive and diverse stakeholder engagement in the planning, design, conduct and follow-up of evaluations is critical to ensure ownership, relevance, credibility and the use of evaluation. Reference groups and other stakeholder engagement mechanisms should be designed for such purpose.”9 Stakeholder engagement and reference groups are recommended for complex evaluations, multi-country and multiprogramme with a wide range of stakeholders. In these cases, such groups may be particularly useful and can ensure a more participatory approach throughout the evaluation.

• 8UNEG, 2016.
• 9Ibid., p. 24.

Module 6 of the IOM Project Handbook, published in 2017, outlined three phases for the evaluation process: (a) planning evaluations; (b) managing evaluations; and (c) using evaluations.11 The IOM Monitoring and Evaluation Guidelines, however, proposes a three-stage process including the following stages: (a) planning for evaluation; (b) undertaking evaluation; and (c) follow-up and using evaluation. These three stages of evaluation are as follows:12

• 11Module 6 of IOM Project Handbook, p. 422 (Internal link only).
• 12Adapted from World Bank, 2015

The first step in planning for an evaluation is defining the purpose and evaluability of an evaluation. The evaluation purpose describes the overall reason why the evaluation is being conducted and its expected results. Agencies and organizations may use different terminology, and IOM is open to accept such terminology when preparing the evaluation ToR.

Some guiding questions that can be used to frame the purpose of an evaluation are as follows:

Assessing the evaluability – in other words feasibility – of an evaluation is an essential part of the evaluation planning process, increasing the likelihood that the evaluation will be able to produce credible information in a timely manner or by limiting its scope.18 It encourages evaluation managers to set realistic expectations of an evaluation on the basis of the contextual realities on the ground, including financial realities and timing, as well as on the monitoring and data collection mechanisms already in place.

It is important to review relevant project documents or strategies to identify what has already been agreed upon with the donor or at the institutional and governmental levels. As some time may have passed since the start of the planning for the intervention to be evaluated, it is also important to review the choices made so far in the intervention to ensure that earlier decisions taken still hold when the evaluation takes place. The programme manager may need to discuss any planned changes with the donor.

The process of planning an evaluation involves trade-off decisions, as the evaluation manager will have to weigh the cost and feasibility of various evaluation designs, as well as the benefits of the evaluation (operational, institutional and strategic).

To define the purpose and assess the evaluability of an evaluation, managers must be aware of the common types of evaluations, methodologies, and evaluation criteria. Understanding these concepts and technical requirements and specificities can also help evaluation managers to manage their evaluations more effectively.

Types of evaluation

Evaluation types can be defined according to the following elements, and evaluations can be a combination of the different categories:

Evaluation type according to timing

One distinction is made on the basis of the timing of the evaluation exercise; in other words, when in the intervention life cycle, the evaluation is conducted.

Evaluation types according to purpose

Evaluations defined by their purpose can be formative or summative. Formative evaluation is conducted during implementation for the purposes of improving performance. It is intended to assist managers adjust and improve project, programme and strategy implementation based on findings, as well as stakeholders’ suggestions and needs. A summative evaluation is conducted at the end of an intervention time frame and also for the benefit of stakeholders not directly involved in the management of the implementation such as donors. It provides insights about the effectiveness of the intervention and gives then the opportunity to use best practices identified during the evaluation. A summative evaluation can inform higher-level decision-making, for instance to scale up an intervention, consolidate it or continue funding follow-up phases.

Based on the purpose of the evaluation
Formative evaluation		Summative evaluation
	Conducted during implementation Intended for managers and direct actors Redresses and improves the project or programme		Conducted at the end of a project or programme Intended for those not directly involved in management Provides insights about the effectiveness of the project Gives the opportunity to use best practices identified during the evaluation Informs higher-level decision-making for follow-up actions

Evaluation types according to who conducts it

A third distinction is made according to the person(s) who conduct(s) the evaluation exercise. There are three types of evaluation based on who conducts the evaluation: (a) internal; (b) external; and (c) mixed.

• 18UNEG, 2016.
• 19Cosgrave et al., 2009.
• 20Some define self-evaluations as being all evaluations conducted in an organization, including those conducted by external consultants, that are not falling under the responsibility and management of independent central evaluation offices, funded by independent mechanisms and budget.
• 21Ibid.

The cost and logistical implications of each type of evaluation will also vary based on who will conduct it. If an external evaluator (or an evaluation firm) or evaluators are contracted to conduct the evaluation, they will charge fees for this service. The fees for evaluators will vary depending on their experience, qualifications and location (locally recruited evaluators with the same level of experience may often be less expensive than internationally recruited evaluators), and evaluators may charge different fees depending on the complexity and difficulty of the assignment. Additional fees may also be charged for travel to insecure locations. The amount to budget for evaluator fees also depends on whether the evaluation is to be conducted by a single evaluator or an evaluation team.

For further information on the decision to select a single evaluator or evaluators, see the subsection, Select evaluator(s) of this chapter.

If an IOM staff member from a different IOM office, which is not involved in the project, will conduct an internal evaluation, the cost for the time spent in-country to conduct the evaluation needs to be considered as travel duty (TDY). For an internal self-evaluation, or evaluations conducted by a staff member from the implementing office, there are normally no fees associated, except for those that would relate to data collection and analysis (for example, in the case of surveys with enumerators or for field visits).

🢂 For further information on budgeting for evaluation within an IOM intervention, see  Annex 5.1. Budgeting for evaluation. 

Evaluation types according to technical specificities and scope

The fourth group of evaluation types is defined according to technical specificities and scope. This group is the most diversified, and the most common types of evaluation are presented here, with additional references provided in the Resources box and within the annexes. The scope of an evaluation allows the understanding of what will be covered and what type of evaluation may be conducted.

IOM usually conducts programme and project evaluations that examine respectively a set of activities brought together to attain specific global, regional, country or sector assistance objectives, and an individual activity designed to achieve specific objectives within a given budget and time period. IOM may also conduct evaluations of a strategy or policy. These may use similar approaches as for programme or project evaluations. In addition, IOM conducts thematic evaluations that examine selected aspects or cross-cutting issues in different types of assistance (such as poverty, environment and gender).

The following evaluation types are relatively common within the IOM context, as well as in international cooperation activities and deserve to be mentioned. A process evaluation examines the internal dynamics of implementing organizations, their policy instruments, their service delivery mechanisms, their management practices and the linkages among these. A country-programme or country-assistance evaluation is more common in United Nations agencies and bilateral assistance that use country programming approaches and defined as an evaluation of one or more donor or agency’s portfolio of development.

Furthermore, IOM also conducts meta-evaluations, which aim to judge the quality, merit, worth and significance of an evaluation or several evaluations.24 Synthesis evaluations are also encouraged as they provide the opportunity to identify patterns and define commonalities.25

Evaluations may also be defined by their technical specificity and the approach that will be used during the evaluation, for instance, a participatory evaluation, which may be defined as an evaluation method in which representatives of agencies and stakeholders (including beneficiaries) work together in designing, carrying out and interpreting an evaluation. The collaborative effort deserves to be underlined, but it also brings organizational constraints that render the exercise relatively complex. A distinction should be made here as well between participatory evaluation and participatory techniques. The latter consist, for instance, of focus group discussions or preparatory meetings and can be included as an evaluation approach irrespective of other types of evaluation selected.

• For further information regarding types of evaluations based on scope and technical specifies, as well as additional evaluation types within this category, see Annex 5.2. Expanded list of evaluation types by specificities and scope.

• 24A meta-evaluation is an instrument used to aggregate findings from a series of evaluations. It also involves an evaluation of the quality of this series of evaluations and its adherence to established good practice in evaluation. See Ministry of Foreigner Affairs (Denmark), 2004.
• 25A synthesis evaluation is “a systematic procedure for organizing findings from several disparate evaluation studies, which enables evaluators to gather results from different evaluation reports and to ask questions about the group of reports”. See General Accounting Office, 1992 (name was changed to Government Accountability Office in 2004), The Evaluation Synthesis.

An impact evaluation attempts to determine the entire range of long-term change deriving from the intervention, including the positive and negative, primary and secondary, long-term change produced by the intervention, whether directly or indirectly, intended or unintended.

Evaluation criteria

Evaluation uses a set of criteria for the assessment of an intervention. Evaluation criteria are standards by which an intervention can be addressed. While several criteria exist, IOM primarily uses two established references for evaluation criteria: (a) the OECD/DAC criteria, which had originally been developed for development-orientated interventions and were adjusted in December 2019 to also be relevant for humanitarian interventions; and (b) the Active Learning Network for Accountability and Performance (ALNAP) criteria, which were developed for humanitarian interventions.

• 29 Humanitarian Coalition, n.d.; Module 6 of IOM Project Handbook, p. 434 (Internal link only).
• 30Ibid.
• 31The OECD/DAC adopted revised criteria on 10 December 2019, incorporating one new criterion, “coherence”. The revised criteria and an explanatory brochure can be found at OECD, n.d.

The OECD/DAC criteria are commonly used in the evaluation community and were updated and adjusted in 2019, including the addition of a new criterion, “Coherence”. A table reflecting those changes is provided.

In addition to the updated definitions, sample evaluation questions related to each criterion are also included.

OECD/DAC and ALNAP evaluation criteria

The focus on given criteria may change at different stages of the intervention life cycle. In an ex-ante evaluation, the focus could be on relevance, while for a midterm evaluation, it could shift towards effectiveness and efficiency so that recommendations for improvement can be made during implementation. By the end of the life cycle, final and ex-post evaluations are better able to assess the overall performance, sustainability and impact of the intervention. However, the evaluation criteria must always take account of the specific requirements of the evaluation and the interest of end users of the evaluation and of other stakeholders.

The evaluation commissioner and/or manager, in consultation with relevant stakeholders, select the evaluation criteria to be used and the questions to be answered. The criteria selected must clearly be spelled out in the ToR and properly reflect the purpose and scope of the evaluation.

• 33Ibid., p. 7.
• 34Buchanan-Smith et al., 2016, p. 113.
• 35OECD, 2019, p. 8.
• 36Buchanan-Smith et al., 2016, p. 114.
• 37OECD, 2019, p. 9.
• 38Buchanan-Smith et al., 2016, p. 113.
• 39 Ibid., p. 114.
• 40 Ibid.
• 41OECD, 2019, p. 10.
• 42 Buchanan-Smith et al., 2016, p. 113.
• 43OECD, 2019, p. 11.
• 44 Buchanan-Smith et al., 2016, p. 113.
• 45OECD, 2019, p. 12.
• 46 Buchanan-Smith et al., 2016, p. 114.

The evaluation ToR are a key framing and planning tool for managing an evaluation, as this provides clear and detailed specifications on the objectives and scope of the evaluation, as well as the roles and responsibilities of the parties involved, such as the evaluation manager, the evaluator(s) the evaluation users and/or possible partners. They also provide information on the timing, methodology and budget of the evaluation. Poorly developed ToR can cause confusion and result in expectations, and focus, that may differ between the involved parties. Having a clear understanding of the different evaluation types and criteria outlined in the previous sections will help formulate the evaluation ToR. The ToR are part of the contractual agreement between IOM and contacted evaluators, as they outline evaluator obligations at all stages of the process, as well as the evaluation commissioner and/or manager expectations.

As part of the planning phase, once the purpose and evaluability of a planned evaluation have been defined and the ToR for the evaluation has been elaborated, a selection process for evaluator(s) must take place, in line with what has already been agreed in the project proposal and/or ToR.

Internal versus external evaluators

The following table provides some tips on the benefits of using internal or external evaluator(s):

Note: Adapted from Module 6 of IOM Project Handbook, p. 440 (Internal link ony).

A mixed team of internal and external evaluators can be used, with the roles and responsibilities defined by their strengths highlighted above. An external evaluator could, for instance, benefit from the knowledge of the organization of the internal evaluator to prepare the inception report and focus on methodology given her/his evaluation expertise.

The expected duration, timing and complexity of the evaluation may also impact the choice. An evaluation may require significant time and travel to various locations; this may have implications for using an internal evaluator, who would still need to perform the regular tasks when conducting the evaluation and may only be available for a shorter field visit. The supervisor of the internal evaluator may also object to release the staff for a longer duration and absence. To complete their evaluation, internal evaluators may need to work on an evaluation report over the course of an average period of three months, while fulfilling the responsibilities of their full-time position.

The question of timing and constraints related to the selection of an evaluator, therefore, needs to be considered and the recruitment process initiated well in advance of the planned start date of the evaluation exercise by the evaluator(s). Recruiting an external evaluator requires IOM to issue a call for evaluator(s) to organize a selection process and have the contract approved and signed according to relevant procedures, which may also take some time. Such procedures are not needed for an internal evaluator, but the availability of an internal evaluator needs to be negotiated with the potential evaluator’s supervisor within IOM, and more time may be required to complete the exercise given the internal staff’s ongoing tasks, as specified above.

Selecting evaluator(s)

In parallel to the decision to select an internal or external evaluator, the use of multiple evaluators instead of a single evaluator can also be considered. For instance, a team may be required if specific expertise is required to analyse the performance of an intervention (such as an engineer to review a construction- related programme component or a health expert to review the response to a specific disease within a programme) or if additional national evaluators can bring an added value to the exercise (such as in case of complex interventions that require good knowledge of national context, stakeholders or politics). The evaluation commissioner, the evaluation manager or management committee can determine which option is best suited to the task, include it in the ToR and adjust the selection process accordingly.

The main points for consideration in the choice between a single evaluator or a team can be summarized as follows:

    (a)  Complexity of the evaluation: Should the evaluation require significant data collection, in- depth and multiple field visits, have a multi-country scope, a combination of specific evaluation methods, involve multiple languages or require a national evaluator perspective, evaluation team may be best suited to conduct the evaluation.

    (b)  Duration of the evaluation: If the evaluation time frame is short, it may be better to consider a team, where the members can work together and complete the evaluation within a shorter time frame.

    (c)  Multiple areas of expertise: If an evaluation requires different areas of very specific expertise that may not be found within one evaluator, it may be necessary to consider selecting an evaluation team that can meet the requirements through its various members.

Selection process

The following section discusses the selection process for hiring an external evaluator or evaluators; this can be done mainly by applying the following: (a) recruitment of an individual consultant; or (b) engagement of a consulting firm or service provider. This process applies to external evaluator(s) only, as internal evaluators in IOM have been pre-identified at the global and regional levels and may be engaged through direct negotiation with the identified evaluator(s) supervisor.

Selecting individual consultants

For the recruitment of a single evaluator, a call for evaluator(s) is issued, including the evaluation ToR (see Annex 5.4. Evaluation terms of reference template) and the following additional elements:

• Requirements: This is the list of competencies required for the individual
• Instructions for the submission of the application: This should include what additional documents are expected to be submitted as part of the application, such as previous evaluation reports. It should also include the deadline for the submission of the application and the contact details for the person to whom the application should be sent.

Once the applications are received, the evaluation manager/committee assesses them and shortlists applicants. IOM has developed a scorecard for the assessment of applications (see Annex 5.6) for evaluations, which is a helpful tool in the selection process. Once the selection is completed, the evaluation manager and/or programme manager prepare a contract, in accordance with IOM instructions on hiring consultant(s) (IN/84 Guidelines for Selection and Employment of Consultants (Internal link only).

Selecting a consulting firm

For the selection of an evaluation team, a request for proposal is issued in accordance with IOM procurement instructions as per the IOM Procurement Manual (Internal link only). A template for the Request for Proposal (RFP) for evaluations is available here in the event that a consulting firm is needed (Internal link only).

The process of overseeing the implementation of the evaluation implies not only supervising the evaluator(s), but also managing and organizing the collection of documents and other materials for the evaluation, organizing the field visits, interviews and written surveys, as well as maintaining communication with key stakeholders.

At the outset of this phase, the evaluation manager, evaluation commissioner, evaluation management committee (if present) and selected evaluator(s) should jointly review the ToR to ensure that there are no comments, questions or key points that need to be renegotiated. It is also standard practice to have a management meeting at the beginning of the evaluation process to ensure that the evaluation manager, evaluator(s) and stakeholders (if relevant) all share a common understanding of the evaluation process and various roles and responsibilities. Furthermore, evaluators should be requested to develop an inception report. This will provide insight into their understanding of the evaluation ToR, as well as useful information on the way they will conduct the evaluation (for further information on the inception report, see the section, Evaluation deliverables). Any changes that result from reviewing the inception report should be well documented and reflected in the relevant documents and/or ToR. At this stage, the evaluation manager should have already provided the evaluator(s) with the key documents to start the evaluation, and additional resources can be shared when the final agreement on the work to complete is reached.

The clear definition of the roles and responsibilities of all parties directly involved in the evaluation is also essential for a sound implementation, with each individual having tasks to complete and deadlines to respect in order to ensure quality.

Evaluators are expected to provide several key deliverables, which should be clearly stated in the ToR. Each of these deliverables are outlined below, with key information concerning their content and potential structure.

• 48IOM, 2022.

Reviewing and providing feedback to the draft evaluation report is a critical step in the evaluation process. Involving the evaluation commissioner, manager (or management committee), as well as other key stakeholders in the process also ensures that all the intended evaluation users will receive the information that they need. If this is not undertaken properly, there is a risk that the evaluation may be discredited by its users once it is published. This step allows for a transparent and open process to review the evaluation prior to finalization.

When the draft report is provided by the evaluator(s), the evaluation manager should coordinate the comments and responses and consolidate all feedback to present it back to the evaluator(s) without delay. Feedback should focus on the technical aspects of the evaluation and factual evidence. Bear in mind that the evaluator is required to make factual corrections but is not required (and should not be requested) to revise findings, conclusions or recommendations in a manner not consistent with presented evidence, as this contravenes evaluation ethics.

In case significant issues surface in the final stage of reporting, the evaluator and manager should reassess the process and develop a plan to address those identified issues. The challenges should be thoroughly assessed to determine if mistakes have been made and whether they can be corrected. All parties can also ensure that the recommendations in the report are acceptable and actionable.

If the evaluation manager and evaluator(s) do not reach an agreement on the interpretation of data and/ or on the conclusions and recommendations that flow from that interpretation, the evaluation manager can prepare a management opinion, highlighting the disagreements with justifications.49

In general, the final report review process should not be another opportunity to provide new information for the evaluation, as relevant information should have been provided during the data collection and analysis phases. However, if new relevant information has just become available, or a recent or concurrent event has had an impact on the analysis or recommendations (as it has happened with the COVID-19 unexpected crisis), the evaluation manager should discuss it with the evaluator, and additional time can be allocated to incorporate the new data and information into the report or into an addendum (for instance, examining how COVID-19 affects the recommendations already made).

• 49For further reference, see IOM, 2019, p. 5.

After the evaluator receives the consolidated feedback, she/he should revise the report as necessary, and submit the finalized version.

Communication on the progress of the evaluation is key for guaranteeing quality and relevant reporting, and each party has a role to play, in particular at the level of the evaluation management and the evaluator(s). Maintaining quality standards for an evaluation is particularly important, as it also enhances the credibility and objectivity of the exercise.

IOM has developed a Guidance on Quality Management of IOM Evaluations, to provide a common understanding of the quality and assurance standards for IOM evaluations. This guidance aims to establish consistent mechanisms to ensure the quality of outcomes, streamline the evaluation processes, and promote a culture of evidence-based learning. It also addresses the need for a standardized approach, incorporating recommendations from various reviews and assessments conducted over the years.

Quality standards ensure that evaluations are conducted in line with the procedural and technical requirements, as well as with the evaluation norms and standards, applied in the organization. They also contribute to the provision of accurate and useful information and to regularly monitor the quality of the evaluations.

Each evaluation actor can contribute to achieving quality standards by providing relevant inputs. Quality control is the primary responsibility of the evaluation manager, who should ensure that an evaluation is conducted in line with the IOM Evaluation Policy and Guidance, as well as any requirements and standards agreed upon with other stakeholders, for instance the intervention donor.51  The evaluation manager and evaluator(s) have the responsibility to guarantee conformity with established quality standards in the carrying out of activities at all stages of the evaluation process.

In addition to the Guidance on Quality Management of IOM Evaluations, IOM has also developed three mandatory quality control tools. Both the evaluation manager and the Evaluator are required to utilize these mandatory quality control Tools to assess the completeness and quality of key documents during the different stages of evaluation:

• Checklist for Terms of Reference
• Quality Control Tool for Inception Reports
• Quality Control Tool for Evaluation Reports

• 51Quality control is defined as “part of quality management focused on fulfilling quality requirements”. It is one activity related to quality assurance, which is “part of quality management focused on providing confidence that quality requirements will be fulfilled”. Quality control efforts should be done at the level of evaluation management, and quality assurance is the responsibility of the centralized evaluation function within an organization. See definitions from ISO 9000:2015: Quality management systems on ASQ, n.d.
• 52Adapted from World Bank, 2015.

After the final report is approved, the evaluation commissioner or manager should work on the follow- up to the evaluation recommendations, in coordination with senior management and the project stakeholders, as appropriate. The evaluation commissioner and manager should consider and discuss with relevant entities how the findings of the report will be communicated to a broader audience as well. The evaluation manager will then finalize the management response matrix drafted by the evaluator, in line with the instructions provided in the IOM publication, Management Response and Follow-Up on IOM Evaluation Recommendations.

It is a monitoring tool that must be referred to on a regular basis until all the follow-up actions have been implemented or are no longer applicable. The relevant use of evaluations as an accountability tool should be done in a timely manner; therefore, it is recommended to complete follow-up actions and the review process within 18 months of the evaluation’s final submission, even when not all follow-up actions have been finalized. The monitoring of the implementation of the management response matrix can be assigned to specific staff within the office. Progress on the follow-up actions included in the matrix should be shared with relevant entities, as well as with the regional M&E officers and the Central Evaluation Unit for their records.

Sharing and publicizing evaluation reports are important steps for guaranteeing the relevant use of evaluation. Evaluation managers and/or commissioners may want to discuss and prepare a communication and dissemination strategy, which will require deliberate action and analysis to reach the right audience. The following points may be considered:

• How will the evaluation be used and disseminated?
• How will the findings in the evaluation report be shared with various groups of stakeholders who may have diverging points of view?
• When is the best time to disseminate the evaluation to ensure its optimal use?

Disseminating evaluations contributes to fulfilling their purpose of learning, by ensuring that the knowledge gained through evaluation can be widely used to improve the quality of new interventions, as well as implementation methods. It is recommended to think about how evaluations will be shared, and with whom, early in the planning phase. These decisions should also take into consideration the specific needs when deciding to share evaluations internally within IOM or externally.

The IOM Evaluation Policy specifies that all evaluation reports are to be made public, but the “sharing strategy” can vary. In the case of an externally shared evaluation – for example, an evaluation of an IOM strategy (corporate, regional or country) or policy (usually corporate) – it could be of interest to all IOM Member States and possibly some donors, while for project evaluations, external interest may be limited to the local government(s) and the specific donor(s) who funded the project. However, in the case of projects, external distribution can also include implementing partners, collaborating non-governmental organizations (NGOs) and/or beneficiaries, which may not be the case for an evaluation of an IOM strategy.

For reports shared internally, a similar distinction applies, as with external reports. While the evaluation of a policy may be shared more often at the departmental level, evaluations of projects, programmes, as well as local or regional strategies, are more valuable for the field offices concerned and relevant thematic specialists at the regional and departmental levels. If the policy evaluation’s dissemination is mainly at the department level for the purpose of organizing the follow-up or lessons learning, it can also be shared more broadly, including to all IOM offices worldwide, given their possible interest on a corporate policy. Some cases are also very specific; for instance, the regional M&E officers and the Central Evaluation Unit need to be kept informed of the publication of evaluations to add them to its central repository of evaluation reports and/or on the IOM Evaluation web page.

Generally, it is recommended to have just one version of a report that can be shared both externally and internally and that serves all stakeholders with varied points of view. It has happened, in a limited number of cases, that two versions of an evaluation report – one for limited distribution and internal use and the other for external consumption – were produced; for instance, when the report contains some sections covering confidential or sensitive issues related to demobilization activities. If uncertain about the dissemination of an evaluation report, the evaluation manager should consult with the CoM for country-level interventions, regional directors for regional or cross-regional interventions and/or the regional M&E officer or the Central Evaluation Unit.

Knowledge and learning derived from evaluation can feed back into the organizational learning and planning processes through regular reflection, accessibility to the evaluation reports and regular exchange of information through learning sessions. This can be visualized as follows:

In addition to evaluation, other processes can enhance learning from interventions. The following are three examples of evaluative approaches that can also incorporate learning in addition to accountability, and also used for monitoring purposes:

    (a) Lessons learned;

    (b) Project performance review (PPR); and

    (c) After-action review (AAR).

Lessons learned and lessons learning

Lessons learned can be understood as generalizations based on evaluation experiences with projects, programmes, strategies or policies that abstract from the specific circumstances to broader situations. Frequently, lessons highlight strengths or weaknesses in preparation, design and implementation that affect performance, outcome and impact. In other words, they are intended to describe the knowledge gained from experiences in well-defined situations. Documenting lessons learned and incorporating them into other interventions can lead to improving the quality of service delivery. In particular, they can help to avoid practices that may regularly fail to produce results or other common mistakes.

The following graphic provides an overview of the process of how lessons learned are identified (through implementation and evaluating implementation), how they are developed and, finally, incorporated and used to improve implementation.

While lessons learned are generally surfaced through conducting evaluation, lessons learned can also be captured through specific lessons-learning workshops, which bring together various stakeholders to brainstorm on performance and identify the lessons learned from an intervention. This approach can be used as well for interventions at the policy or strategic levels, where stakeholders may be asked to reflect on their approaches to a particular topic or thematic area over time.

Another similar concept, in terms of generating knowledge from an evaluation, is the notion of good practices, which can be seen as the identification of a procedure that has proven to produce results in a satisfactory way and that is proposed as a “standard” practice suitable for widespread adoption. A lesson learned with an identified practice that produces such satisfactory results is identified to be worthy of replicating and possibly upscaling, may, over time, become an “emerging good practice”.55

• 55ILO, 2014, p. 2.

In general, disseminating lessons learned can take place as a part of, or in a similar manner to, disseminating a final evaluation report. In particular, they can be incorporated into any evaluation summary documents made available to relevant users. The Evaluation brief template and guidance (Annex 5.10) contains a specific section for the presentation of lessons learned, when required.

In some cases, lessons learned may be of particular interest to relevant thematic specialists for their further dissemination and applicability to other similar interventions. The “use” of lessons learned is particularly critical in the development of new interventions, at the project or programme level, as well as in the development of strong strategic and policy guidance to a particular area of IOM’s work. Therefore, evaluation managers should carefully consider with whom to share lessons learned and identified good practices, in order to best incorporate them into future and planned IOM interventions.

Project performance review

IOM has developed a PPR tool, which is an assessment that focuses primarily on the performance of a project or programme using OECD/DAC criteria, with a focus on effectiveness and efficiency. The objective of a PPR is to support field offices in assessing the performance of their interventions, using a constructive, participatory and coordinated approach. The exercise usually takes place during implementation, so that corrective measures can be taken if necessary. The criteria of relevance, impact and sustainability are briefly analysed through the PPR, and it may also look at the extent to which the outcomes of an intervention are being achieved or may be achieved due to the activities, as well as outputs completed.

A PPR also looks at cross-cutting issues, analysing the level of accountability to beneficiaries and affected populations, particularly in emergency context, as well as assessing the intervention’s link to global, regional or country strategies.

It is important to note that a PPR is not an evaluation, as it is less comprehensive than an evaluation. An evaluation takes more time and preparation, covers issues in greater detail and is able to produce more evidence-based analysis and findings. Further differences between a review and an evaluation can be summarized as follows:

After-action review

An AAR is a structural discussion about an intervention that enables a team to consider and reflect on what happened, why it has happened and how to sustain strengths and improve weaknesses.56  It is a facilitated process involving key actors, in which the general principle is to be neutral and objective to ensure that the discussions stay focused on challenges, remain positive and do not evolve into self- justification. Essentially, the review should focus on questions, such as the following: “What was expected versus what actually happen(ed)?”, “What went well and why?” and What could have gone better and why?”.

An AAR involves the following steps:

Source: Adapted from Buchanan-Smith et al., 2016 and USAID, 2006.

As a first step, participants brainstorm on their understanding of the objective(s) or intent of the action and then develop a timeline of what actually happened and has changed over time. The next step is more focused on an analytical approach, as they identify what went well and why, and what could have gone better and why. At the end of the process, conclusions of what could be done better next time are summarized into lessons learned. Participants may be asked to vote for what they regard as the three most important lessons in case of multiple considerations. An AAR discussion is a facilitated process and may not last more than half a day or a day. Depending on the resources and time available, it can either be formal, with additional preparatory work, or informal as detailed in the box below.

Source: Adapted from USAID, 2006.

• 56Adapted from Buchanan-Smith et al., 2016.

 Other examples of evaluative approaches and tools are summarized as follows:

• 57Adapted from Wilson-Grau, 2015.

Adapted from Module 6 of IOM Project Handbook, pp. 423–431 (Internal link only).

Click HERE for more info

Adapted from OIG/Evaluation, IOM Evaluation Guidelines (January 2006), Annex 2.

Cluster evaluation: An evaluation that analyses a set of related activities, projects or programmes to identify common threads and themes.

Country-programme/Country-assistance evaluation: An evaluation of one more or more donor or agency’s portfolios of development.

Cross-section evaluation: A systematic evaluation of various evaluation reports on a specific project type, on projects involving one particular sector, or on one particular instrument or theme, designed to review and possibly update existing development policy directives.

Democratic evaluation: An evaluation approach that addresses critical evaluation issues, such as dealing with power relations among stakeholders, including stakeholders’ perspectives, and providing useful information to programmes. Power redistribution is accomplished by “democratizing knowledge” and holding all groups, including the client, mutually accountable.

Empowerment evaluation: An evaluation promoting close involvement between the evaluator and the project/programme participants to produce more meaningful and useful evaluation results. Empowerment evaluation is necessarily a collaborative group activity, not an individual pursuit.

In-depth evaluation: An approach that consists of focusing evaluation or a part of an evaluation precisely on a category of outputs, or on a group or category of impacts.

Incorporated/built-in evaluation: An approach to implementation that involves fairly continuous self-evaluation by principal actors and participants, according to pre-established criteria related to the purpose and goal of the assistance.

Meta-evaluation: An evaluation that aims to judge the quality, merit, work and significances of an evaluation or several evaluations.

Partial system evaluation: An evaluation also used in emergency situations, which covers only a part of the system. It can be related to thematic or sector evaluations.

Participatory evaluation: An evaluation method in which representatives of agencies and stakeholders (including beneficiaries) work together in designing, carrying out and interpreting an evaluation.

Process evaluation: An evaluation that examines the internal dynamics of implementing organizations, their policy instructions, their service delivery mechanisms, their management practices and the linkages among these.

Quasi-experimental impact evaluation: An evaluation that compares different groups before and after programme implementation to assess the programme impact and value added of further investments. It uses rapid and economical studies that combine exploitation of existing data sets with rapid sample surveys, tracer studies, interviews and others.

Real-time evaluation: An evaluation implemented in emergency situations that aims to provide a rapid feedback on humanitarian operations and be an immediate catalyst for improvements in organizational and operational performance. The methodology cannot be rigid, and flexibility and adaptability are required, although it must guarantee quality.

Sector evaluation: An evaluation of a variety of aid actions, all of which are located in the same sector, either in one country or cross-country. A sector covers a specific area of activities, such as health, industry, education, transport or agriculture.

Single-agency response evaluation: Also in emergency situations, an evaluation that covers the overall response by a particular agency.

Single-agency/Single-project evaluation: An evaluation that covers a single project undertaken by a single agency in an emergency situation.

Stakeholder evaluation: An evaluation that involves agencies, organizations, groups or individuals who have a direct or indirect interest in the development assistance, or who affect or are positively or negatively affected by the implementation and outcome of it. Stakeholders work together to develop and finalize instruments and procedures, produce recommendations, and make decisions throughout the evaluation process (related term: Participatory evaluation, which focuses on methodology).

Strategic evaluation: An evaluation of a particular issue aiming to advance a deeper understanding of the issue, reduce the range of uncertainties associated with the different options for addressing it and help to reach an acceptable working agreement among the parties concerned. It is usually adapted when urgency of the issue poses high risks to stakeholders and has generated conflicting views.

Synthesis evaluation: “[A] systematic procedure for organizing findings from several disparate evaluation studies, which enables evaluators to gather results from different evaluation reports and to ask questions about the group of reports.”58

System-wide evaluation: An evaluation used in emergency situations that covers the response by the whole system to a particular disaster or emergency.

Theory-based evaluation: An evaluation that focuses on an in-depth understanding of the workings of a programme or activity, the programme theory or logic. It needs not assume simple linear cause-and- effect relationships, but maps out the determining or causal factors judged important for success and how they might interact.

• 58GAO, 1992.

Cross-cutting themes can be defined as additional considerations or areas that intersect with an intervention, or that can be easily integrated into it, without losing focus on the main goals of the intervention. Mainstreaming a cross-cutting theme is generally understood as a strategy to make the specific theme, given its importance, an integral dimension of the organization’s design, implementation and M&E of policies and interventions. The inclusion of themes can evolve over time and new themes can be added; they are not necessarily the same for all organizations and not all may be relevant to be considered in an intervention.

This section will cover the following themes: (a) rights-based approach (RBA); (b) protection mainstreaming; (c) disability inclusion; (d) gender mainstreaming; (e) environmental sensitivity and sustainability; and (f) accountability to affected populations (AAP). It is important to note that this annex treats the M&E of cross-cutting issues only. In the event that these thematic areas become the main focus of an intervention, it is no longer to be considered as a cross-cutting theme.

    🢂 Evaluation terms of reference (ToR) should ensure that questions pertaining to the in-tegration of relevant cross-cutting themes are reflected inside a specific section or un-der relevant criteria, specifying that it will be examined as a cross-cutting theme.

The following are some questions that can be asked during both monitoring and evaluation processes to ensure that an RBA perspective is covered:

Participation

• Have the various stakeholders (including both rights holders and duty bearers) been involved in planning and designing the M&E of the project and determining the type of data to collect?
• Are other individuals or groups, such as local civil society groups or NGOs, involved?
• Are key groups, particularly the most marginalized groups of rights holders, included and/or involved in the M&E process?

Equality and non-discrimination

• Is the M&E process explicitly designed to detect or measure discrimination against particular groups throughout its objectives and outcomes?
• Is the data collected appropriately disaggregated, such as by age, disability, ethnicity, sex, nationality and migration status, to track any gaps in considering equality and discrimination throughout intervention outputs and outcomes?

Accountability, transparency and rule of law

• Are the M&E processes directly linked to any rights such as measuring the realization of specific rights?
• Do the M&E processes account for any form of complaint mechanisms and how are received complaints dealt with?
• Are the findings from the M&E shared publicly in a transparent manner?
• Are the findings from the M&E used to promote changes in law or policy of the State?

During the evaluation, the evaluator should also consider the following tips for ensuring that RBA is integrated in the evaluation process:

    (a) Include mechanisms to ensure that the most marginalized groups of rights holders are/were involved in the evaluation.

    (b) As an evaluator, ask yourself: Were all stakeholders included and how will the evaluation explicitly detect or measure discrimination against particular groups? For example, the evaluation may be designed to detect any form of discriminatory practices that may have occurred during the implementation of the project or as a result of the project.

    (c) Identify channels to field any form of complaints that may be received during the evaluation.

• 60Please note that this section is based primarily on the guidance from 2016 for protection mainstreaming within MCOF. This will be further updated upon availability of new guidance on protection (IOM, n.d.f, p. 4).
• 61IASC, 2013.
• 62IOM MCOF specifies the following 15 sectors of assistance: (a) camp management and displacement tracking; (b) shelter and non- food items; (c) transport assistance for affected populations; (d) health support; (e) psychosocial support; (f) (re)integration assistance; (g) activities to support community stabilization and transition; (h) disaster risk reduction and resilience building; (i) land and property support; (j) counter-trafficking and protection of vulnerable migrants; (k) technical assistance for humanitarian border management; (l) emergency consular assistance; (m) diaspora and human resource mobilization; (n) migration policy and legislation support; and (o) humanitarian communications (IOM, 2012).
• 63IOM, 2018c, p. 16; see also IASC, 2016.
• 64Ibid.