After an immunization session, Musa*, a health worker in Kano State, Nigeria, typically emptied contents of his portable vaccine carrier into the solar direct-drive (SDD) freezer in his clinic without much thought about sensitivities of the vaccines. In some cases, he stored diluents outside the freezer. Unknown to Musa, this action can cause vaccines to lose potency, potentially rendering them useless. Although Musa has previously received in-person trainings on immunization, they were not targeted to his specific skill gaps on vaccine management. To worsen things, refresher trainings occur infrequently, with either bulky or sometimes unavailable reference materials.
A health worker using the solar direct-drive for vaccine storage
One year after Musa was introduced to the Immunization Academy (IA)—a CHAI-supported e-learning platform that uses a combination of demonstrational videos and skills-based online assessments to teach and reinforce essential immunization skills—his knowledge on vaccine storage and handling has improved; he is now able to differentiate between heat and freeze sensitive vaccines, including the arrangement of the vaccines and diluents in the solar direct-drive. Adebisi*, Musa’s supervisor, also learned how to effectively conduct supportive supervision visits including planning, on-the-job mentoring to health workers, and effectively identifying and bridging skill gaps.
Many health workers and managers like Musa and Adebisi face limitations in effectively managing the day-to-day responsibilities of the immunization program and thus, need a system for continuous learning and performance improvement.
Health workers in Nigeria using the Immunization Academy app
Introduction of IA in Nigeria
Over the last decade, efforts by the National Primary Healthcare Development Agency (NPHCDA) and its partners to strengthen health worker capacity have yielded results. Between 2016 and 2018, health workers were evaluated across four immunization skill areas: vaccine handling, vaccine administration, communication and advocacy, and data management. The results found an average improvement of 21 and 12 percentage points in Kano and Lagos states respectively.[1]
Despite these improvements, skills and knowledge gaps persist across the country, often due to high costs associated with conventional in-person trainings, which are often lengthy, didactic, and disruptive to primary healthcare (PHC) services. The COVID-19 pandemic further highlighted the need for an alternative approach to health worker training.
Responding to observed challenges, CHAI began a partnership with the IA team in 2020 to enable continuous improvement of immunization knowledge and skills for managers and health workers in Nigeria through e-learning and assessment tools. Over an 18-month period, a series of engagements and activities were conducted by CHAI to aid the successful rollout of the IA e-learning suite in Nigeria.
Evaluating the effectiveness of e-learning
CHAI and the IA conducted a baseline assessment[2] of the effectiveness of e-learning. A total of 180 health facilities were selected for the assessment across 24 local government agencies (LGAs) in Kano and Lagos states. The states were selected based on their high population and diversity reflective of many states in the country. The assessment involved observation of health worker performance[3] across all selected facilities; 537 structured interviews with health workers and 24 focus group discussions were conducted.
The health worker performance observations showed significant improvement from 2018[4] in skills across all four domains, especially vaccine handling and administration. However, a deep dive revealed persistent capacity gaps in critical areas such as temperature monitoring and vaccine management, remembering the six key messages[5], as well as effective data use.
Fig 1: Measurement of health worker practice across key immunization skill areas
The results also revealed that nearly 40 percent of health workers had taken an online course, 99 percent were willing to use e-learning for trainings, 94 percent had smartphones, and 46 percent consumed 1GB – 5GB of data monthly. The focus group discussions highlighted key motivators (government endorsed certificates, recognitions, and promotions at work), and barriers (data costs, network issues, electricity, and workload) to the adoption of e-learning by health workers.
Fig 2: Health worker accessibility and experience with eLearning
Fig 3: Challenges with accessing eLearning platform
Contextualization of the IA suite for Nigeria
CHAI worked with the government and Expanded Program on Immunization (EPI) partners at the state and national levels to co-design and guide the rollout of the platform to over 2,200 health workers and managers based on findings from the needs assessment. These inputs were also used by IA to create a custom interactive tool for Nigeria, called IA Score Nigeria (more details about the platform here), which provided a personalized training approach to target and remediate individual learning gaps.
The program also offered certification endorsed by NPHCDA upon achieving a passing score of 70 percent to motivate healthcare professionals to participate and complete the program. Additionally, a dynamic dashboard was created to allow managers to review state, LGA, and individual learner data. With this practical dashboard, managers could easily see where skill and knowledge strengths and gaps existed. A leaderboard was also created to highlight immunization professionals in each LGA who had high scores and those who showed great improvement in their post-test scores.
Fig 4: Process Flow for the use of the IA Score Nigeria[6]
Results and impact
Following 15 months of expansion and use of the IA platform by different health workers, there was a 230 percent increase in enrolled users from nearly 10,000 before implementation to over 37,000 after across the country. Between November 2020 and July 2021, IA videos had been viewed approximately20,000 times in Nigeria; and over 550 NPHCDA-endorsed certificates have been awarded to health workers and managers through the IA score platform with average score improvements of 13 percent.
The use of the platform also showed tangible improvements in immunization practices among health workers.
To assess the impact of IA on immunization service delivery, CHAI worked with the states of Kano and Lagos to conduct an end-line assessment.[7] In both states, health workers in facilities with high uptake[8] of the IA platform had twice the improvement in immunization practices compared to those in low uptake facilities.[9] Health workers in high uptake facilities had a 27 percent improvement in health worker performance across four immunization areas compared to the baseline and those in low uptake facilities, who had a 13 percent improvement.
Fig 5: High Uptake Facilities’ Baseline vs Endline Immunization Service Delivery Performance[10]
Fig 6: Low Uptake Facilities’ Baseline vs Endline Immunization Service Delivery Performance [11]
A positive relationship (measured using R2) [12] was also observed between health workers’ use of IA and improvement in their overall immunization practices. The highest overall scores were observed among health workers using IA resources daily.
Fig 7: Comparison between the frequency of the use of IA reported by interviewees versus their overall RI practice performance when graded using the OSCE Tool
Institutionalization and scale-up
Some key success factors for the introduction, institutionalization, and eventual scale-up of IA in Nigeria was its integration into existing capacity building processes of the government including:
- Appointment of government focal persons to drive uptake among health workers.
- Planned health worker trainings such as new vaccines’ introduction.
- Peer-led learning.[13]
- Review meetings and supportive supervision using results from the performance dashboard.
- Training curriculum of the Community Health Practitioners’ Registration Board (for pre-service institutions).
Conclusion
Innovative low-cost e-learning approaches are critical for continuous improvement in health worker practice. The successful rollout of IA in Nigeria highlights opportunities to expand the scope of digital learning to include the full suite of primary healthcare (PHC) services to improve health system performance and ultimately save millions of lives.
This work was supported by funding from the Bill & Melinda Gates Foundation. We also acknowledge the successful partnership with Immunization academy, which owns the e-learning tools, and the National and State agencies without whom, we would not have achieved the recorded results.
Contributing authors:
Muhammad Sheriff, Assistant Program Officer, Vaccines Program; Isa Adamu, Senior Analyst, Vaccines Program; Dr. Owens Wiwa, EVP Global Resources for Health, West and Central Africa and Country Director Nigeria
