MONITORING, EVALUATION, ACCOUNTABILITY AND LEARNING (MEAL)
In line with Save the Children’s global strategy, the quality framework is rooted in an integrated approach to MEAL and provides RB and Save the Children with a basis for continuous improvement of the programme’s delivery model and generates robust evidence of the programme’s effectiveness and impact. One of the key objectives of this project is to test a robust model for child survival at scale and use the evidence to advocate for national and global scale-up / replication. Our MEAL approach therefore rests on an appropriate combination of internal monitoring, ongoing operational research and independent evaluations of the programme. Figure 1 below provides a summary of our monitoring, evaluation and research approach as well as indicative timelines for each of the main components of MEAL.
A comprehensive monitoring system has been designed and is being implemented to collect/collate data against key input, activity, output, outcome and impact indicators at regular intervals throughout the programme’s lifetime.
- Programme staff submits regular (monthly and quarterly) programme implementation plans (PIP) detailing target populations, target areas and action plans. The monitoring system also enables a review of the progress achieved in addressing the key bottlenecks identified during the inception phase. Each state updates and presents the PIPs on a monthly and quarterly basis. Project staff conducts regular site visits to verify/corroborate PIP findings.
- Key Performance Indicators (KPIs) have been developed for this programme, which are largely informed by the WHO/UNICEF 7 point plan. During the inception phase, Save the Children has conducted a detailed baseline study, which has provided baseline data against a number of output, outcome and impact indicators the programme is measured against (see Annex 2)
- A number of process indicators have beendeveloped to monitor the programme’s delivery model, the programme expenditure and provide a high level assessment of the programme’s efficiency.
An evaluator has been contracted to conduct an independent evaluation of the programme. The evaluation includes a baseline (building on the baseline already conducted by Save the Children and filling gaps as appropriate), a midline evaluation and a final evaluation.
The evaluation team have developedan approach and methodology that is derived systematically from the programme’s intervention logic, as well as being compliant with the OECD-DAC criteria for programme evaluation. To ensure this double compliance, the following checklist is used:
Table4– OECD DAC ‘Checklist’
In particular, this independent evaluation looks into the following areas of our Signature Programme:
- Impact on beneficiaries: This will involve identifying the effectiveness and impact on children under five across the intervention areas. The programme’s impact will be assessed through a quasi-experimental design: the intervention areas will be matched with comparison areas through a range of matching criteria. Tracking the performance of both the intervention and comparison groups will allow us to measure the impact of the programme on diarrhoea-led mortality in intervention areas.
- Value for Money: The evaluation will include an assessment of the programme’s Value for Money (VfM) during the midline phase of the evaluation, which should be framed by the 4 E’s framework (economy, efficiency, effectiveness, equity). It will assess how the programme’s input of financial resources links to programme outcomes through an impact chain, and will measure the extent to which the programme is doing the right things, in the right ways and at the right cost.
- Advocacy: Using the advocacy framework adopted by the GoI and Save the Children’s advocacy toolkit, the evaluation will measureto what extent advocacy activities enhanced the impacts and effectiveness of the country programmes, and how the advocacy efforts have benefitted from being part of the Signature Programmes.
- Innovation: By definition, the innovative aspects have less prior robust evidence about their effectiveness and therefore the more innovative aspects of the causal chain should receive particular attention. This involves identifying the intended causal pathway and collecting evidence to understand impact. The operational research also contributes to assessing the effectiveness and impact of innovative components of the programme (See “3 – Additional research”)
- Scalability: Scalability is achieved when a Signature Programme identifies a model or way of working that could be expanded with no loss of effectiveness, efficiency, economy or equity, in ways that are acceptable to key stakeholders, and which are sustainable. The evaluation assesses the extent to which the programme is scalable, and identifies opportunities for replication in other contexts.
- Partnership and Coordination: Save the Children promotes partnership and coordination among key government departments and stakeholder effective implementation of the 7 point plan. Therefore, the evaluation also looks into how partnership and coordination with key stakeholders and between different government programmes brought additional value to the Signature Programme. This is assessed in line with improvements in economy, efficiency or effectiveness.
In partnership with technical agencies Save the Children is conducting an evaluation at three point of time (base-line, mid-line and end-line) to ascertain the effectiveness of the 7 point plan in the target areas. Baseline has provided the outcome and output indicators at the time of initiation of project and has provided a base from which impact can be measured at midline and endline.
The evaluation team has used a quasi-experimental approach to evaluate the impact of the programme. To do this they have measure the impact of the components on target populations and a control group. Tracking the performance of both the intervention and control groups will allow us to measure the programme’s impact. All surveys have acontrol group selected by matching key social indicators at the unit to be selected. These units are in a block in rural setting from the same district and ward/or cluster of slums in urban setting from the same district.
The measurement of the impact will use a Difference-in-Difference (DD) approach. We willcalculate the mean difference between the Baseline and Endline values of outcome and impact indicators for both intervention and control areas. Then, we will calculate the difference between these two mean differences. The second difference (the difference-in-differences) is the estimate of the impact of the programme. Figure 3 summarizes the approach, where D represents the estimated effect of PDP (the Difference-in-Differences).
Figure 2 – Difference-in-Difference approach
The evaluation has used a cross-sectional design. Additionally, a longitudinal study (cohort study) has been conducted, which has allowed us to assess the effectiveness of the 7 point plan in reducing the incidence of diarrhoea and diarrhoea episodes per child by tracking diarrhoea outcomes across a cohort of 600 children
3. Additional Research
Save the Children has designed a longitudinal cohort study to assess the impact of specific interventions on reducing the incidence of diarrhoea and diarrhoea episodes per child. During the inception phase of the project, the first phase of this cohort study was conducted to understand the seasonality in incidence of diarrhoea, months when the incidence reaches a peak or low point, etc. However, the inception phase was short (only about 6 months) and is not likely to generate adequate data on seasonality. Hence, the longitudinal cohort study continued with a larger cohort for the entire period of the study. Based on the trend data, the programme managers devised strategies to address seasonal spurts in diarrhoea. For example, they advised government on the management of both treatment package (supply of ORS/ Zinc) and prevention measures (e.g. sanitation drives before the peak season). This can be combined with the reporting of ORS and Zinc stock position by facility-level staff through a mobile app.
Save the Children will also carry out a research study on the status of water quality and health implications for communities that have recorded very high levels of arsenic and fluoride in their drinking water, as well as a research study in the area of sanitation to assess the impact of construction of latrines (at HH level, at community level) on high water table areas. This is because poor water quality and very poor environmental sanitation are the parameters that have important bearings on children developing diarrhoea in the homes/communities.
While country programmes move to scale up interventions, there was a need for research on delivery of strategies, on overcoming barriers to interventions and on better ways for implementation. To ensure success, the programme monitoring framework included an operational research component. This seeks to answer a range of questions including ones to assess progress on the effectiveness of the BCC models under use, models to enhance community capacity to hold governments to account for health services, the different models of low-cost water purification and sanitation methods etc. These questions ensure we are continually improving the models used to secure maximum impact and value for money.
The list of research questions has beenfinalized in consultation with the TAG.
4. Dissemination of findings
The following dissemination activities will be conducted in the coming years of the project:
- We will ensure that all the findings coming from various M and E activities in the project are widely disseminated through regular meetings with the government officials and project partners. We will organize and/or participate in cross learning events, information sharing platforms and other national and international events to disseminate the key findings and lessons learnt. We will also publish the information through our own channels of communication and advocacy as well as in the peer reviewed journals.
- The programme team will also aim to produce local evidence on the cost-effectiveness of strategies for the delivery of the 7 point plan and to document the impact on child mortality associated with increased access to regular immunization, quality treatment, and exclusive breastfeeding, access to safe drinking water, sanitation and hand washing facilities.
- We will identify champions in the communities, policymaking circles and the academic community in order to influence policy. Partnerships with key research and academic institutions may include NICED, All India Institute of Hygiene and Public health and IIT Delhi.
- Save the Children will design and implement participatory approaches in M and E. A Community Based Monitoring System (CBMS) will be designed and tested for its relevance. Save the Children will also test the use of technology (e.g. mobile phones) for monitoring progress against outputs. The effectiveness, feasibility and scalability of such methods will be tested during the course of the project.
MEAL Plan for year 2