While governments, development organizations, researchers and thinkers grapple with the new realities ushered in by COVID-19 (coronavirus), effective delivery and implementation of development programmes have become increasingly crucial amid lockdowns and travel restrictions.
Without appropriate levels of decentralized program delivery and accountability mechanisms that reach the last mile, program issues are likely to magnify and become binding constraints to the effective achievement of development goals.
In Zambia, the Supporting Women’s Livelihood (SWL) initiative of the Girls’ Education and Women’s Empowerment and Livelihood (GEWEL) Project recently piloted an innovative phone-based monitoring and evaluation system (PME). This is an application of the Iterative Beneficiary Monitoring (IBM) system developed by the Poverty and Equity Global Practice at the World Bank, and uses the principles of the Problem Driven Iterative Adaptation approach.
The SWL initiative provides extremely poor women aged 19 to 64 years with a comprehensive economic inclusion package including savings group facilitation, a productivity grant and phone/SIM card, life and business skills training, and mentorship.
The pilot was undertaken at about the same time as the outbreak of COVID-19 to address challenges associated with monitoring and evaluation (M&E) activities in a complex program implemented in a large and sparsely populated country.
It aimed to develop a cost-effective risk-based monitoring and evaluation system that could provide quick feedback on an ongoing basis to keep track of and improve implementation of the programme.
Moreover, the risk-based approach is intended to identify stages of program implementation as well as geographies where program implementation are deemed higher risk.
The pilot was carried out in four of the 51 project districts in partnership between the World Bank, the government’s Project Implementation Unit and a team from the University of Zambia.
This three-way partnership was crucial as it brought together technical skills, local ownership and programmatic knowledge. One of the early challenges was the low cellphone network coverage in rural areas, which meant that though all beneficiaries had phones, many could not be reached.
To maximize the likelihood of reaching beneficiaries on their project phones, district government staff were mobilized to raise awareness of the upcoming phone survey.
To the extent possible, phone surveys were also timed with productive grant payments as previous evaluations showed that beneficiaries were most likely to be actively using their phones within 24 hours of being paid.
The pilot provided the project team with a clear picture of implementation progress and quality, and also quickly highlighted potential areas of concern for further investigation and remedial action. For example, a comparison of pilot payments data with administrative data in the SWL payments system (SPS) shows that 82% of respondents reported a payment status that corresponded with the SPS.
We identified districts where the issue of mismatch between SPS and payment monitoring and evaluation (PME) interview results was highest (see Chart 1 below). Further investigation showed that the mismatch in reporting was because beneficiaries were unaware of the money being credited to their account.
This could be due to multiple reasons, such as the payment being credited to the beneficiary account just a couple of days prior to the PME interview or that the related PSP did not send SMS alerts or that the SMS alerts did not go through.
These led to discussions within the program around how to improve beneficiary awareness of their payments and related processes, but also on how to strengthen the engagement of district officers in this process as they often have direct contact with beneficiaries.
The pilot also proved to be cost effective, costing about $10k across four districts, which is far lower than the expenses (roughly three times) and per diems associated with field travel for M&E activities.
Chart 1: Payment status comparison – Administrative and Phone-based M&E records
In the current COVID-19 context, further institutionalizing remote M&E systems is becoming increasingly important. By triangulating and comparing data from multiple systems (including administrative or survey data) in real time, the projects can improve their monitoring capacity at a time when field visits and spot checks are largely off the table or restricted.
It must be noted, though, that such a phone-based M&E system is intended as a complement rather than a complete replacement of the traditional M&E system.
Nevertheless, in the COVID-19 context, as the situation evolves and development projects worldwide adapt to the ‘new normal’, innovative solutions such as this can help projects continue to assess implementation quality and troubleshoot efficiently.
