Clearly articulating the expected results or desired change of an intervention is the foundation for M&E. It is necessary to identify what requires change, what expected change looks like and, finally, how such change can be achieved through IOM interventions. This is where ToC comes in handy. While there are many different definitions of ToC, this section focuses on approaches relevant to the IOM context.See examples of different shapes of ToCs at Center for Theory of Change, n.d.a

A ToC may be viewed as a tool or methodology to map out the logical sequence of an intervention from activities to results, showing multiple pathways that may lead to change, including pathways not related to the planned intervention. It may also be viewed as a deeper reflective process and dialogue among staff and stakeholders, reflecting the values and philosophy of change that make more explicit the underlying assumptions of how and why change may occur as a result of an intervention. At its best, a ToC is a combination of these two views.

It is most often defined as illustrating a link between activities, outputs, outcomes and objectives, creating a chain of results, referred to as the pathway of change or the causal pathway. A causal pathway is the process of step-by-step mapping through which all the required preconditions necessary to reach a desired change are determined. It is essentially a comprehensive articulation of how and why desired change will occur within a specific context.Center for Theory of Change, n.d.b. Some basic components of a ToC often include a big picture analysis of how change is expected to occur in relation to a specific thematic area, an articulation of a specific pathway in relation to this and an assessment framework that is designed to test both the pathway and the assumptions made about how change happens.Stein and Valters, 2012.

What is a Theory of Change?This definition is adapted by IOM from Center for Theory of Change, n.d.c. Theory of Change is a comprehensive description and explanation of how and why a desired change is expected to happen in a particular context. Specifically, it focuses on mapping out what an intervention or change initiative does (its activities) and how these lead to achievement of the objective through results (outputs, outcomes, objectives). In this way, a ToC articulates hypotheses about how change happens by explaining the connection between an intervention and its effect. It does so by surfacing the logic and rationale for an intervention and articulating the assumptions inherent in the approach (multiple pathways).

A ToC can be viewed as a product of a series of critical thinking exercises that provide a comprehensive picture of the different levels of change expected to occur due to an intervention, at the stage of its development, during its implementation, as well as following its completion.

The use of a ToC is more and more common and developed for all types of interventions. It can be applied to design, monitor, as well as evaluate different types of interventions and is best used to measure the complexity of transformation and change. Because a ToC acknowledges that change is not linear, but dynamic and complex, it often seeks to articulate social, political and community-based change(s) or empowerment initiatives. A ToC is a process-oriented approach that can be used to analyse the interrelations and/or interactions in complex systems in which IOM, partners and allies work. Such a process helps navigate in unpredictable and complex environments and helps track and assess change in the system to which an intervention may contribute.

It is important to note that different terminologies may be applicable when defining the multiple pathways of change, such as: (a) objectives, outcomes and outputs; (b) long-term, intermediate and short-term outcomes; or (c) outcomes and pre-conditions. This section uses the terms objectives, outcomes, outputs and activities to align with IOM’s M&E terminology and either develop or supplement IOM results matrices.

ToC is a guiding framework for all stages of thinking, action and sense-making for interventions involved with social and/or political change processes.

Graphic depiction

When graphically depicting a ToC, diagrams can be generally flexible in format and may be simple or complex. They can be vertical, horizontal or circular.

The graphic depiction of a ToC can help with mapping out multiple causal pathways to identify the most feasible one for a given intervention. Another advantage of graphically depicting a ToC is that it makes possible causal links more understandable and immediately visible. It enables comparisons between different pathways and can help identify implicit assumptions.See examples of different shapes of ToCs in Figure 3.1, as well as at Center for Theory of Change, n.d.a.

Participatory approach

The process of developing a ToC should be participatory and collaborative, and include key stakeholders, as well as the beneficiaries, or people that the Organization seeks to assist, and/or affected populations, who can offer their different perspectives to define what an expected change within a specific thematic field may look like.For the purpose of the IOM Monitoring and Evaluation Guidelines, IOM uses the OECD/DAC definition of beneficiary/ies or people that the Organization seeks to assist as “the individuals, groups, or organisations, whether targeted or not, that benefit directly or indirectly, from the development intervention. Other terms, such as rights holders or affected people, may also be used.” See OECD, 2019, p. 7. The term beneficiary/ies or people that IOM seeks to assist, will intermittently be used throughout the IOM Monitoring and Evaluation Guidelines, and refers to the definition given above, including when discussing humanitarian context. Their participation can also help identify underlying assumptions that are inherent to explaining why a particular change is expected to occur.

Multiple pathways of change

A ToC acknowledges that change is dynamic and complex and can show different possible pathways that might lead to change (see Figure 3.2 where each colour of the arrows represent a different pathway). The process of developing a ToC helps discover these multiple pathways of change.

A common challenge when using the ToC is the lack of a “theory” and/or using a weak theory. For instance, weak theories do not explain how change is expected to occur or do not state/establish assumptions clearly. It is important to ensure that the ToC actually articulates a logical theory, which makes the implicit causal mechanisms explicit and supplements the graphical representation.

By developing a valid and relevant ToC, implementers can ensure that their interventions will be delivering the appropriate activities for desired and realistic results. It ensures that interventions are easier to monitor and evaluate, bring to scale, as well as sustain, as each step – from the ideas and assumptions behind it, to the results it hopes to achieve and resources required – are clearly articulated within the theory. A well-articulated ToC can also promote a common understanding of the intervention for all actors involved in implementation, thereby facilitating a cohesive and common approach.

The process of developing a ToC can help identify whether, and at which stage or level, assumptions, logical jumps or missing key steps in the change process are taking place. Developing a ToC is a good way to raise further questions such as the following:

   a) Why is a particular change expected to happen?

   b) What evidence is available to support that expected change will/has occur/red?

   c) What logical jumps are made?

   d) What assumptions are made?

When developing a ToC, it is important to understand its purpose. ToCs can be applied at different levels, ranging from world views, strategies and policies, to the project or programme level and all the way down to activity level. For instance, world views can help clarify social and political theories that inform one’s thinking. Organizational ToCs can help inform the vision, mission and values that the organization requires to contribute to social change. For policy ToCs, it can help identify how an organization expects change to evolve in a specific sector and how it contributes to it.

Scholars have not reached agreement on an overall definition of, and methodology for, developing a ToC, and donors may follow different approaches to drafting a ToC. For instance, the approach by the United Kingdom Foreign, Commonwealth and Development Office (FCDO) to drafting a ToC largely differs from that of the United States Agency for International Development (USAID) or that of the European Union.FCDO is formerly the Department for International Development (DFID). While FCDO’s approach still contains strong elements of a logical-framework approach (looking at inputs, outputs, outcomes and impact), USAID emphasizes the possibility of multiple pathways that may lead to change, while highlighting the importance of underlying assumptions throughout the process. The following depict these two different approaches:

Source: Vogel and Stephenson, 2012

Note: FCDO is formerly the Department for International Development (DFID).

Source: Kedzia, 2018

Irrespective of how different stakeholders approach a ToC, they all have one commonality: they enable the articulation of how, why and under what conditions a change is expected to occur within a specific context. While there is no one standard approach to developing a ToC, the following section illustrates a formula that can be applicable in most contexts and is commonly used by USAID to measure social and behavioural change.

While ToCs can be illustrated in different ways, the logic of the chain of results, or causal pathway, can be tested using if-then-because statements. In other words, it helps reveal assumptions that are “tested” through actions/activities, while assumptions play a central role in developing the ToC.

Every step taken, from the overall objective of the intervention to each of its activities, has a ToC behind it that can explain and articulate the logical connections (or the different pathways) between the lowerlevel results, such as between outputs and outcomes, as well as between the higher-level results, such as the outcomes and objectives.

A common challenge when designing an intervention are logical leaps and gaps. There may be a disconnect between strong problem analysis and seemingly unrelated activities, with weak links and/or assumptions between objectives, outcomes, outputs and activities. Through surfacing underlying assumptions, the ToC may provide a bridge between analysis and programming.

Generally, a ToC can be articulated using the “if X, then Y, because of Z” formula. That is, “if X action/ activity occurs, then Y result will occur, because of Z assumption(s)”. The process of surfacing such underlying assumptions can help identify where logical jumps are made or helps identify missing key steps in the change process.

The following section will focus on one of the many possible pathways illustrating the application of the if-then-because formula, noting that this exercise can be repeated for many different pathways for different levels.

When reviewing a ToC, one may want to focus on the links a ToC is trying to make. The following checklist consists of five simple questions that may help with the review process:

1. Are both the action (X) and the result (Y) clearly formulated? Is it clear what approach is being utilized? When considering the result, ensure that terms being used are clear. Ask whether the results are measurable, and if so, what would be a good source of verification for this. How would this be monitored and how would can sources of verification for this result be attained?
2. Is the result realistic and achievable given the scale of action, resources and time frame? Try to assess whether the results are proportional to the scale and scope of action. Is the result observable? Is it immediate, intermediate or long term? How will the result be measured? Are there any logical jumps and/or gaps between a modest action and an ambitious result?
3. Do the assumptions explain why the approach is suitable? A strong assumption is able to articulate why the chosen approach should produce the expected change/result. This reflects the intervention’s problem analysis.
4. Are the relevant preconditions that are necessary for success included in the assumptions? Assumptions targeting necessary preconditions mostly reflect conditions beyond the intervention’s control that are believed to be necessary to attain results, but do not explain why/how change will occur. This may also be viewed as being linked to thinking about risk. To identify such assumptions, it is helpful to ask one’s self that if the intervention were to fail, what may have gone wrong.
5.  Does the ToC make sense if it is reversed?

Earlier, it was shown that ToCs can be applied to an activity, project, programme, strategy or policy. The following shows how to review a ToC at all the levels by applying the “if X, then Y, because Z” formula to each one. In cases where the “if X, then Y, because Z” formula is applied, the higher-level ToC’s “if” statement becomes the lower-level ToC’s “then” statement, meaning that for each statement, the action/ intervention of higher-level ToC should correspond with the result/desired change of the lower-level ToC.

Separating the ToC from the logical framework is challenging, because they both stem from the same family of approaches related to programme theory. A ToC, as described in the previous section, starts from the premise that the process of social change is complex, taking into account different perspectives and analysing the underlying assumptions of an intervention’s design. In contrast, a logical framework offers a more simplified picture of intervention logic, not considering all the underlying assumptions and related causal links, but rather only those related to the particular intervention and selected during the project’s initial conceptualization. A logical framework can be viewed as a more rigid and linear way of thinking about change.

Originally, logical frameworks were intended to summarize complex stakeholder discussions about the objectives and results that an intervention would reach and contribute to, as is also the case with the ToC rationale. The intention was to analyse internal and external dependencies that would influence the intervention’s effectiveness, including direct assumptions that require to be taken into consideration in the analysis.Vogel, 2012.However, the logical framework insufficiently reveals all the pathways that can influence or lead to change. ToC thinking helps bridge this gap, by showing multiple inter-connected causal pathways and by making explicit the assumptions that explain how and why change is expected to occur. Table 3.1 summarizes some of the aspects of a logical framework as compared to a ToC.

Table 3.1. Comparing and contrasting Theory of Change and logical framework

Source: Adapted from Hivos, 2015, p. 15.

Despite these differences, a ToC and logical framework remain complementary, can be used together, and ToC thinking can be applied to the process of drafting the IOM Results Matrix. This encourages intervention developers to make explicit the ToC assumptions about how desired change will occur, highlighting the change process and related causal pathways between expected results. These also can then be articulated when addressing the “why” and “how” within the rationale section of the Project Proposal Template, more specifically, providing the strategic thinking that informs the “assumptions and hypotheses underlying the causal relationships between activities, outputs and outcomes”.The IOM Project Proposal template is available in Module 2 of IOM Project Handbook, pp. 182–183 (Internal link only).

Developing an IOM Results Matrix is one of the foundations for monitoring an intervention at IOM. It provides an overview of the parameters for the measurement of intervention results. The Results Matrix, mandatory for all IOM project proposals and available in the IOM Project Handbook, can also be used for the implementation of a strategy or policy and is drafted during the initial development stage.Module 2 of IOM Project Handbook, p. 119 (Internal link only).

Understanding how a results matrix is developed can contribute to improving general understanding of how to monitor intervention progress and results effectively. The following sections will illustrate the various steps involved in developing a Results Matrix.

Note: Visualization of an IOM Results Matrix located in Module 2 of IOM Project Handbook, p. 121.

Vertical logic

The term vertical logic refers to the “means-end relationship between activities and the results”, as well as the relationship between “the results and their contribution to the broader objective”.The means-end relationship is defined in IOM Project Handbook, p. 25 (Internal link only).

The IOM Results Matrix uses the terms “objective”, “outcome”, “output” and “activity” to demonstrate vertical logic. The diagram below provides a visual representation of the vertical logic (as well as horizontal logic) within a results matrix.Module 2 of IOM Project Handbook, p. 122 (Internal link only).

Source: Adapted from Module 2 of IOM Project Handbook, p. 122 (Internal link only).

Vertical logic focuses on the results at each level. It is the process of taking the logical steps from the objectives down to the activities, with the aim of linking and demonstrating how the results at each level contribute to the next. Results that are properly linked demonstrate the causal connection from one result level to the next, forming a causal pathway or results chain.

Horizontal logic

Horizontal logic “defines how each of the levels in the vertical logic will be measured and the assumptions that are required for the means-end relationship to hold true”.Module 2 of IOM Project Handbook, p. 123 (Internal link only).

Source: Adapted from Module 2 of IOM Project Handbook, p. 122 (Internal link only).

Horizontal logic completes the Results Matrix by identifying what assumptions are required for the results to occur and how progress on each of the results will be measured.

When it comes to expressing results within a results matrix, there is diversity of terminology used by different organizations and agencies. However, the underlying logic for the development of the Results Matrix is similar, allowing for alignment of concepts and related reporting, as well as M&E processes. The following chart provides examples of the Result Matrix terminology IOM has used in the past, as well as the terminology IOM currently uses and compares these with the terminology used by several other key development entities. Careful consideration should be given to ensuring that the results at each level of the vertical logic line-up when transferring interventions between donor/partner formats and IOM templates.Additional information on the use of different terminologies can be found in Church and Rogers, 2006.

Source: IOM Project Development Training, 2018.

Note: FCDO is formerly the Department for International Development (DFID).

Applying vertical logic

IOM currently uses the terms objective, outcome, output and activity to demonstrate vertical logic, while indicator, assumption, data source and collection method, target and baseline help to elaborate the horizontal logic. The following is a summary of those definitions:For more details on each component, see Module 2 of IOM Project Handbook (Internal link only).

Applying horizontal logic

The following sections provide more detail on how horizontal logic is applied to the Results Matrix. The components of horizontal logic are assumptions, indicators, baseline, targets and data source and collection methods. The horizontal logic connects the measurement of results (through indicators) with the assumptions behind how the results are expected to occur.

Assumptions

Please refer to the visualization of the IOM Results Matrix above to identify the Assumptions column in the Results Matrix.

“Assumptions are the necessary and positive conditions that allow for a successful means–end relationship between the different levels of results.”Module 2 of IOM Project Handbook, p. 137 (Internal link only).

Assumptions help complete the intervention logic by placing the intervention in the specific context in which it will be implemented. Assumptions also help identify important conditions on which the success of the intervention depends and that lie outside IOM’s line of control. In this sense, assumptions identify the required preconditions for results to occur.An assumption checklist and examples of writing assumptions can be found in IOM Project Handbook, pp. 137–140 (Internal link only).

Indicators

Please refer the visualization of the IOM Results Matrix above to identify the Indicators column in the Results Matrix.

After having set up achievable and well-defined results and having identified related assumptions, the next step in developing a Results Matrix is to select indicators to monitor progress towards achieving those results. Indicators consist of information that signals change.

Indicators can be defined as “the quantitative or qualitative factors or variables to measure achievement or to reflect expected changes”.Ibid.; see also OECD, 2010.Indicators can either be qualitative, quantitative, binary or proxy indicators. Indicators help indicate change towards an intended result and demonstrate whether an intervention is on or off track.

Indicators are not intended to demonstrate why the intervention has made a difference; this is inter alia covered by evaluation. Similarly, they do not demonstrate how change occurs, which is, inter alia, covered by the ToC or by evaluation. They help understand whether change has occurred.

There is no specific rule on the ideal number of indicators for any one result, but they should be able to measure whether the result has been achieved.

Table 3.2 summarizes the use of indicators at the different levels of results.

Table 3.2. Levels of intervention control

Although indicators need to be tailored to results of an individual intervention, it can be useful to look at the successful indicators of other interventions and modify them as needed. In some cases, standard indicators have been developed within specific thematic areas and may act as a guide to help measure results vis-à-vis certain international norms and standards; hence, they may require further adaptation.

It is also recommended to align indicators with existing country-level, regional, thematic and global strategies and policies. Examples of this can be to cross-check indicators to ensure their alignment with indicators in a country-level United Nations Sustainable Development Cooperation Framework (UNSDCF), a government strategy or national action plan to address a particular thematic area or topic, indicators established as a part of the Sustainable Development Goals (SDGs).For additional information, please refer to the UNSD, 2020.

It is also important to keep in mind the specific, measurable, achievable, relevant and time-bound (SMART) criteria used to develop indicators:

Source: Module 2 of IOM Project Handbook, p. 147 (Internal link only).

An indicator must also be both relevant (directly related to the result) and achievable (requires a reasonable amount of time and resources to gather and analyse data). Assessing whether an indicator is achievable requires assessing the data source/means of verification (MoV).

Data sources and data collection method

Please refer to the visualization of the IOM Results Matrix above to identify the data sources and data collection methods in the Results Matrix.

In order to monitor indicators, a source of information to verify each indicator is required. Within IOM, these sources of information are referred to data sources, defined as “identify[ing] where and how the information will be gathered for the purposes of measurement of specific indicators.”Module 2 of IOM Project Handbook, p. 143 (Internal link only).In addition to data sources, other commonly used terms include MoV or source of verification (SoV).

Data sources should identify what information to collect, how to collect that information and in what form (collection method) and with what frequency. Data sources can include documents (such as reports, government publications and records), data sets (such as national census data and project monitoring data sets), records (such as training attendance sheets and beneficiary case files) and people (such as beneficiaries and/or affected populations, stakeholders, project staff and government officials).

Data sources can be primary or secondary. Primary data is collected by the implementing organization and may include personnel, budget, administrative data, surveys, interviews and direct observations. Secondary data is collected by others outside the implementing organization.

It is required that each known data source collection method be specified in the initial Results Matrix, as time, capacity and budget must be built into the intervention in order to successfully implement them. Available resources and frequency of collection should also be identified during the development phase of an intervention. This is particularly important for M&E purposes, as it can also have specific budgetary implications.

It is equally important to collect only the data required for measurement of the indicators and data that is intended for further use. Collecting other additional information may result in added cost and time. Chapter 4 will provide further information on data collection methods and tools.

Baseline and targets

Please refer to the visualization of the IOM Results Matrix above to identify the baseline and target columns in the Results Matrix.

Baseline data and targets can be defined as follows: “Baseline data provides a foundation against which to measure change over time, while targets establish precisely the mark the project intends to hit.”Module 2 of IOM Project Handbook, p. 145 (Internal link only).

Baseline data can be considered as the starting point, depicting the initial conditions before the implementation of an intervention. The baseline provides the first measurement of an indicator. It sets the current condition against which future change will be measured.

Establishing targets sets the threshold for a particular indicator; it establishes what the intervention hopes to achieve, measured against a baseline. If targets are met for all of a result’s indicators, the intervention can be considered as having successfully achieved its result. The target set for a particular indicator should be appropriate within a given context or location, keeping in mind that an appropriate target in one scenario, location or context, may not be appropriate for another context. Key considerations to establishing a target may include budget considerations (what is the maximum that can be accomplished with the available resources), donor or key counterpart priorities (what is the intervention being asked to achieve) and contextual limitations (what is feasible given the environment in which the intervention is implemented). Setting targets for some results may be straightforward, while for others, it can be quite complex.Examples of baseline data and targets set at different result levels can be found in Module 2 of IOM Project Handbook, pp. 145, 146, 148–151 (Internal link only).

Once the Results Matrix is developed and finalized, it can be converted into a monitoring tool that can be used during the implementation of an intervention: the Results Monitoring Framework (RMF). The RMF is developed at the start of the implementation, after a project has been funded and activated. The RMF is the primary tool to monitor the results of any intervention. It enables all members of the implementing team, as well as stakeholders, to track the progress being made towards achieving intended results. By specifying the data collection method, the RMF also highlights the requirements to obtain high-quality data. The RMF can be used alongside the detailed workplan for monitoring activities, financial reporting tools for monitoring budget compliance and the risk management plan for monitoring risks, to ensure a holistic monitoring approach.

Results Monitoring Framework

Source: Module 4 of IOM Project Handbook, p. 262 (Internal link only).

The RMF reflects much of the same information contained in the Result Matrix, but it contains five additional columns: data analysis, frequency, responsible person, achieved and progress. The RMF additionally removes the assumptions column from the Results Matrix.See the proposed template in Module 4 of IOM Project Handbook, p. 263 (Internal link only). The following shows the distinction between the two tools.

Figure 3.13. Results Matrix and Results Monitoring Framework comparison

Results Matrix

Results Monitoring Framework

The five new columns are completed as follows:

The following table looks closer at the different types of monitoring and outlines the tools most useful in conducting each type of monitoring.

Table 3.3. IOM’s four types of monitoringAdapted from IFRC, 2011.

Monitoring a policy or strategy differs from monitoring a project or programme in that it looks at the bigger picture or macrolevel of what the organization is trying to achieve. Despite this difference, similar approaches can be used when monitoring a strategy. It is useful for monitoring purposes to differentiate between strategies developed at the country, regional and global levels. As strategies aim to attain results, usually through higher-level results than those found in projects and programmes, it is also possible to apply a results matrix to a strategy. The following table provides key considerations for the development of strategies to facilitate monitoring, as well as guidance on how to universally monitor strategies developed at different levels. Resources for relevant strategy development tools are provided at the end of this section.

Table 3.4. Monitoring a strategy

Remote management approaches have substantial implications for monitoring and accountability practices, as well as the ability of the implementing organization to provide assurance of reaching project/ programme results. Where situations may restrict staff members from meeting with beneficiaries or monitoring activities directly, they must rely on other staffing approaches or external partners. Remote management approaches are required in circumstances where management may not have physical presence and hence limited technical oversight, monitoring and accountability, as well as in situations with an increased risk of fraud and corruption occurring. The following are some of the common situations in which remote management approaches can be adopted.

In general, remote management, including remote monitoring and some aspects of TPM, can be a temporary or a more permanent response to security and logistical challenges in the direct implementation of programmes.This section is based on IOM's TRD and DPP/Evaluation feedback collected on TPM in 2018. Depending on the degree of remoteness, different strategies may need to be applied. The office managing the intervention must identify a set of general parameters and thresholds that define when direct implementation is no longer possible or advisable (such as what are the risks the organization is willing to take). If required, mission staff can seek assistance to set parameters and thresholds from their respective regional office and/or the Central Evaluation Unit (DPP/Evaluation) at Headquarters.

Considerations include the following:

Remote management may have significant implications for the organizational set-up, accountability, monitoring and assurance of the quality of interventions. When a situation calls for remote management, the set-up and use of monitoring approaches require more attention. Due to logistical difficulties in conducting the monitoring in complex and remote environments, the need for additional training and contractual arrangements with additional reporting lines may arise. Strong remote monitoring approaches become key to supporting and contributing to remote management.

A common challenge of remote monitoring is the allocation of sufficient resources for planning and budgeting the set-up of rigorous and effective monitoring systems. It is therefore important to identify operational constraints and budgeting limitations encountered in those fragile and complex environments. This ultimately may also prevent abusive use of no-cost extensions. Considering such constraints and limitations in the monitoring section of each project proposal may also reassure the donor that attention is paid to monitoring in order to guarantee the overall quality of an intervention.

Through the use of specific remote management approaches, monitoring of implementation can still continue. The following table outlines key challenges in the context of remote monitoring and possible solutions to address them.

Table 3.5. Remote monitoring: Key challenges and possible solutions

TPM is the system of contracting third parties to conduct the monitoring, and/or cover monitoring functions, such as collecting (and at times verifying) data, on behalf of the implementing instance or donor. However, the definition can further be contextualized, and TPM modalities can take various shapes, especially in non-permissive environments. Therefore, when speaking of TPM, it is important to clarify which of the following possible situations is being discussed.

• IOM uses TPM – IOM, as the managing entity, uses TPM to monitor its interventions: IOM hires an external monitoring team to conduct the monitoring of its own and/or of its implementing partners’ performance.
• IOM is the third-party monitor – Another managing entity uses IOM to monitor its interventions: IOM is the TPM entity and conducts monitoring of non-IOM interventions.
• IOM is the subject of TPM by another entity – An entity (usually a donor agency) uses external TPM entities to monitor IOM’s performance: An IOM-managed intervention is being monitored by a TPM entity, working directly for the intervention donor.

TPM is not only used in emergency situations; it can also be used in large-scale programmes, for instance within the United States Refugee Admissions Program, which uses TPM to monitor its activities implemented around the world by different agencies, or the European Union Trust Fund programme with IOM’s performance being monitored by another entity.

Challenges and risks of third-party monitoringThe following sections are drawn from information compiled by TRD, developed in consultation with DPP/Evaluation in 2018.

While TPM allows for the indirect tracking, verification, validation, real-time collection of data and course correction of IOM interventions, it also has some limitations. Such limitations can include poor-quality reporting of monitoring visits conducted, unexpected costs (inter alia, related to improving the quality), lack of transparency and difficulty using TPM to collect outcome-level data. Many of the risks inherent in remote management are also true for TPM.

Reporting is a critical component of M&E, as it enables the presentation of collected data and findings generated for key stakeholder use and overall performance. Reporting on results provides evidence and feedback to decision makers to inform their decision-making processes. Internal monitoring reports and external donor reports can serve several purposes, depending on an intervention’s information needs, as well as the information derived from M&E, which can be put to different uses. M&E information can serve to:Adapted from Kusek and Rist, 2004.

Collecting data for a results-based report

The data necessary to report on progress in achieving the targets set for indicators should be collected throughout the implementation of an intervention and compiled and analysed prior to reporting. This includes data needed from project stakeholders and implementing partners. Therefore, it is important to establish why, when and how data needed for reporting will be collected.

The Results Matrix as well as the RMF collects key information and can therefore be used as resource for results-based reporting.For more detailed information regarding the RMF, please refer to the Results Monitoring Framework section of this chapter.

Irrespective of how well data is collected and analysed, data must also be well presented, with the target audience in mind. If poorly presented, reporting becomes less useful, which wastes resources and time. When identifying the purpose of data collection, it is important to plan for strong reporting structures from the very beginning. The chart below provides a picture of the cycle of data collection and analysis, reporting and use:

Writing a results-based report

When looking at reporting from an M&E perspective, some additional practices can be used to ensure results-based reporting.The IOM Project Handbook provides information about the content of reporting, including general report writing tips in Module 5 at pp. 367, 387–393 (Internal link only).It is important to emphasize active language that captures the process of change, by explaining measurable, visible and concrete change in a given context. Language used in the report should focus on what is different as a result of the intervention, rather than what is implemented to create that change.

In the past, reporting often focused too much on the activities implemented and not on results at the output and outcome level, nor on the impact of the intervention towards achieving an intervention objective. Reporting should:

• Tell the story of implementation, moving from the activities to the results and demonstrating how resources were used to achieve those results;
• This story should provide an overview of the quantitative and qualitative data that demonstrates achievement towards results;
• Underscore the challenges encountered by the project or programme and where improvements might be needed.Chapter 6 of UNICEF, 2017, pp. 146–148.

In addition, there are other recommended tips for writing results-based reports that adequately capture change.

A good balance needs also to be kept between descriptive writing and analytical writing throughout. Descriptive writing helps to describe what has occurred, while analytical writing supplements this by adding analysis that may help explain why these things happened, what they have achieved (beyond the activities and the immediate outputs) and what implications this may have.

In addition to being results-based and incorporating both descriptive and analytical writing, there are other considerations that strengthen the quality of reporting. Effective reporting is relevant, timely, complete, accurate, concise and user-friendly, consistent and cost effective. It also presents good or high-quality data. The following chart provides a list of questions to help identify whether reporting meets these criteria:Adapted from IFRC, 2011.

Strong reporting can satisfy IOM’s relationships with donors, improve donor support and facilitate the monitoring of a project. Weak reporting can be an indication of a lack of accountability and transparency mechanisms, with possible consequences for the relationship with a donor or other stakeholders.For examples of these consequences, see Module 5 of IOM Project Handbook, p. 356 (Internal link only).

Planning for effective reporting can be used to identify specific reporting needs and information needs of the target audience early in the process. Information can also be for a particular purpose, such as strategic planning, ongoing project implementation, compliance with donor requirements, evaluation or organizational learning. Utilization-focused reporting, that is reporting information according to information needs that will be utilized, helps optimize the use of report findings and avoids unnecessary reporting.

The project manager is responsible for drafting, preparing and coordinating the donor reports in a timely manner.Module 5 of IOM Project Handbook, p. 358 (Internal link only).Within IOM, there are two main types of donor reports according to frequency, namely interim and final reports. Module 5 of the IOM Project Handbook lays out, in detail, the steps required to draft and prepare a narrative donor report.See Module 5 of IOM Project Handbook, pp. 260–361 (Internal link only). In summary, these steps are as follows:

• Review project documents (originals and any subsequent revisions) to ensure that the most up-to-date information is presented in the report; specifically, the most recent agreed-upon version of the Results Matrix, reporting requirements and deadlines.
• Review previous interim reports to ensure consistency of information and provide updates on any previous challenges raised in prior reports.
• Review donor agreement and amendments to confirm key reporting requirements stipulated in the agreement, in particular, the frequency of reporting, format and reporting language. If these are not stipulated in the agreement, the IOM Project Handbook provides additional instructions on how to proceed at p. 261.

IOM reporting template section II (Progress made towards realizing outcomes and outputs)Ibid., p. 364.

Section two of the reporting template describes results (outcomes and outputs) and related activities of the implementation during the reporting period. If the intervention has conducted frequent monitoring, information pertaining to results, such as outcomes and outputs, can be derived from its monitoring reports, its Results Matrix and/or its RMF. Progress made towards incorporating the cross-cutting themes should also be mentioned in this section of the report.Guiding questions for incorporating cross-cutting themes into donor reporting can be found in the annex of chapter 5 of the IOM Monitoring and Evaluation Guidelines.

IOM reporting template section III (Progress achieved compared with the indicators in the Results Matrix)Module 5 of IOM Project Handbook, pp. 365–366 (Internal link only).

The third section of the reporting template focuses on reporting on progress against the indicators of the Results Matrix, with detailed information on how to complete each cell.

IOM Results Matrix in the reporting template

Progress towards the realization of the objective, outcomes and outputs is recorded concisely in the column “Progress made during the period” while the progress made towards the results for the duration of the intervention is included in the column “Cumulative progress”.

A well-designed Results Matrix, and related RMF, with detailed indicators and data sources, can outline how data can be monitored at each results level – output, outcome and objective. Following the above, the data from monitoring findings is then recorded by being entered into the column “Achieved” of the RMF and, when reporting, into the column “Progress made during reporting period” for each level (output, outcome or objective – see above). As implementation progresses, monitoring reports and progress tracked over time in the RMF will show how progress on the indicators slowly lead to overall results at different levels that will inform reporting.

IOM reporting template section IV (Challenges encountered and actions taken)Module 5 of IOM Project Handbook, p. 367 (Internal link only).

This section of the reporting template is an important part of reporting on results, as it provides a space to explain how results were affected by unintended consequences during implementation. The section describes and analyses significant difficulties or delays faced during project implementation and summarizes the corrective measures that have been taken or are being planned to address and rectify the situation. An analysis of the impact of any assumption in the Results Matrix, which did not hold true, or any risk realized during the reporting period must also be included in this section. The effect of the unrealized assumption or realized risk on the delivery of specific results and the impact of the overall project implementation should be regularly monitored to diminish any negative effects. Monitoring can also contribute to the assessment of whether the issue was outside of IOM’s control, such as in the case of a political event or a natural disaster. If the problem was due to a flaw or oversight in project design or due to insufficient mitigation measures in the risk management plan, this should also be mentioned. Finally, it is important to describe the measures or treatment plans that have either been planned or have been taken to address the situation and how it will be monitored. A table is provided within the reporting template to guide users to match each challenge with actions taken.Ibid.Reporting on those issues is also an important element for future evaluations. This is also be a good time to update the risk management plan, if needed.

IOM reporting template section V (Conclusion)Ibid.

In this section, a brief summary of the key achievements realized during the reporting period should be provided, as well as the next steps in the project’s implementation outlined. In the case of an interim report, briefly reiterate if there are any significant or persistent challenges anticipated for the upcoming period. It can be helpful to show how these future key activities are envisioned to lead to further results. For final reports, good practices and/or lessons learned during implementation should also be highlighted. To capture lessons learned, it is important to identify the exact challenges and remedial actions that were taken that lead to achieving positive results in subsequent activities. If they exist, lessons learned logs can be useful for this section of the report. Evaluations can also contribute to the lessons learned process and complement the final report.

As stated above, reporting based on M&E findings and data can contribute to organizational learning and help improve the quality of interventions overall. While chapter 5 of the IOM Monitoring and Evaluation Guidelines provides more detail on learning through evaluation, this section highlights the link between reporting and learning with a focus on reporting based on findings and data from M&E activities.

Results-based reporting is crucial for decision-making and planning during the implementation of an intervention, as it can provide the information necessary for evidence-based decision-making. The findings and data generated through M&E activities provide crucial input for project management and informs decision-making. Furthermore, reflecting on findings from M&E allows for learning from day-today activities. In turn, the lessons learned from daily activities help to improve the quality of both ongoing and future activities, as well as for future programmes.

Reporting, reflecting and learning should occur throughout the intervention life cycle. While at the conceptualization stage, findings and lessons learned can be incorporated from previous evaluations; at the implementation stage, monitoring and/or a midterm evaluation can provide information to help decision-making to improve performance and impact and contribute to learning. Finally, at the completion stage, project/programme staff can reflect on the intervention to prepare for donor reporting and an evaluation facilitates the collection of higher-level lessons learned.