Operability, Acceptability, and Usefulness of a Mobile App to Track Routine Immunization Performance in Rural Pakistan: Interview Study Among Vaccinators and Key Informants

May combine two claims.

The TMK district had poor vaccination performance, with Penta coverage of 23% and PCV coverage of 11% at the start of the study.

“This district had poor vaccination performance, with a Penta coverage rate of only 23% and PCV coverage of only 11% at the start of the study.”

Probably not a substantial claim: may not pass S3 (load-bearing).

They could develop a monthly visit plan that could particularly target defaulters.

“They could develop a monthly visit plan that could particularly target defaulters.”

May combine two claims.

Migration to digital records from manual recording was preferred by 23 of the 26 vaccinators, while 3 vaccinators wanted both manual and digital record keeping.

“Migration to digital records from manual recording was preferred by 23 of the 26 vaccinators, while 3 vaccinators wanted both manual and digital record keeping.”

May combine two claims.

The district managers were interested in the app and expressed ownership of it.

“The district managers were interested in the app and expressed ownership of it.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some vaccinators used only the videos for immunization awareness education.

“Some vaccinators used only the videos for immunization awareness education.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The post-vaccination photograph was considered key to ensuring reliable data.

“The post-vaccination photograph was considered key to ensuring reliable data.”

Classified as probably not a substantial claim. May combine two claims.

The key informants requested to continue using the app and expand its use to other districts.

“The key informants requested to continue using the app and expand its use to other districts.”

May combine two claims.

The main uses of the app data were improving the number of routine immunization encounters and locating defaulting children.

“The key informants commonly reported that the main uses of the app data were improving the number of routine immunization encounters and locating defaulting children.”

May combine two claims.

Of the 26 vaccinators, 25 reported that the app was easy to use and they could operate it without difficulty.

“Of the 26 vaccinators, 25 reported that the app was easy to use and they could operate it without difficulty (Table 3).”

Probably not a substantial claim: may not pass S3 (load-bearing).

There were no reports of phone damage or loss during the study.

“However, there were no reports of phone damage or loss during the study.”

May combine two claims.

Data-driven accountability resulted in a demand for work recognition as well as fear of exposure.

“Data-driven accountability resulted in a demand for work recognition as well as fear of exposure.”

May combine two claims.

Internet connectivity issues often occurred in the more remote villages, but the app allowed offline data entry.

“Internet connectivity issues often occurred in the more remote villages, but the app allowed offline data entry.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The vaccinators wanted a replacement phone at least every 2 years to continue using the app.

“The vaccinators wanted a replacement phone at least every 2 years to continue using the app.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Their performance could be tracked accurately by their supervisors using GPS tracking.

“Their performance could be tracked accurately by their supervisors using GPS tracking.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The vaccinators mentioned the conditional fuel support provided by the district health office based on the number of monthly visits reported by the app.

“The vaccinators also mentioned the conditional fuel support provided by the district health office based on the number of monthly visits reported by the app.”

Classified as probably not a substantial claim.

They perceived that instant data entry through a mobile app generated a more accurate defaulter list.

“They perceived that instant data entry through a mobile app generated a more accurate defaulter list.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Interviewees recounted poor vaccinator performance prior to using the app and attributed this to poor work attitudes, insufficient supervision from supervisors, and a lack of fuel support and functional motorbikes for supervisory visits.

“The interviewees recounted poor vaccinator performance prior to using the app and attributed this to poor work attitudes, insufficient supervision from the supervisors, and a lack of fuel support and functional motorbikes for supervisory visits.”

May combine two claims.

GPS location of vaccinators provided reliable information, while previously checks were onerous.

“GPS location of vaccinators provided reliable information, while previously checks were onerous.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Issues included hanging of the phones after 2 years.

“Issues included hanging of the phones after 2 years.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Stakeholders requested that the project be expanded to other sub-divisions within the district.

“The stakeholders also requested that the project be expanded to the other sub-divisions within the district.”

May combine two claims.

Reporting of individual performance resulted in supervisory recognition and personal gratification in reaching targets.

“Reporting of individual performance resulted in supervisory recognition and personal gratification in reaching targets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Additional effort to increase awareness of routine immunization was needed in high-defaulter areas.

“Additional effort to increase awareness of routine immunization was needed in high-defaulter areas.”

May combine two claims.

The data encouraged competition among the vaccinators as well as sharing of lessons learned.

“The data encouraged competition among the vaccinators as well as sharing of lessons learned.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Their reported concern was they might forget to upload pending data or be unable to upload pending data if the phone was damaged or stolen.

“Their reported concern was they might forget to upload pending data or be unable to upload pending data if the phone was damaged or stolen.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The vaccinators reported that the lack of fuel support made it difficult to conduct outreach sessions.

“The vaccinators reported that the lack of fuel support made it difficult to conduct outreach sessions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Health officials expressed eagerness to continue the tracking initiative and anxiety that routine immunization coverage would decline again after closure of the project.

“The health officials expressed an eagerness to continue the tracking initiative and anxiety that routine immunization coverage would decline again after closure of the project.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Photos of the immunized child were uploaded more frequently than photos of the vaccination card.

“Photos of the immunized child were uploaded more frequently than photos of the vaccination card.”

May combine two claims.

Often, the phone had to be restarted, and there was a concern that unsaved data would be lost.

“Often, the phone had to be restarted, and there was a concern that unsaved data would be lost.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Political patronage is still an issue but better managed due to transparency of accountability and monitoring of the vaccinators in the target setting.

“Political patronage is still an issue but better managed due to transparency of accountability and monitoring of the vaccinators in the target setting.”

Classified as probably not a substantial claim. May combine two claims.

In the past, it was challenging to hold the vaccinator cadre accountable for routine immunization performance, making it difficult to improve vaccination volume.

“In the past, it was challenging to hold the vaccinator cadre accountable for routine immunization performance, making it difficult to improve vaccination volume.”

May combine two claims.

Only two vaccinators reported that parents would not allow a photo under any circumstances, in which case a photo of the vaccination card was uploaded instead.

“Only two vaccinators reported that parents would not allow a photo under any circumstances, in which case a photo of the vaccination card was uploaded instead.”

May combine two claims.

Vaccination volume and child details were very accurate and could be quickly verified.

“Vaccination volume and child details were very accurate and could be quickly verified.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The district was able to act as an example for other districts.

“The district was able to act as an example for other districts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

EPI motorbikes were provided by the district health office to the better performing vaccinators; these were co-financed by the project and the vaccinator.

“Additionally, EPI motorbikes were provided by the district health office to the better performing vaccinators; these were co-financed by the project and the vaccinator.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Public recognition was instrumental for increasing motivation.

“The stakeholders reported that the public recognition was instrumental for increasing motivation.”

Classified as probably not a substantial claim.

Vaccinators felt that the ability to record immunization entries during the visits was better than using recall to manually record data later at the health facilities.

“while vaccinators felt that the ability to record immunization entries during the visits was better than using recall to manually record data later at the health facilities.”

May combine two claims.

The app-generated maps helped to finalize microplans and identify missed locations.

“In addition, the app-generated maps helped to finalize microplans and identify missed locations.”

May combine two claims.

Real-time immunization data were mainly used to monitor vaccination volume, track children with incomplete vaccinations, develop outreach visit plans, alter existing microplans, and provide a results-based fuel allowance.

“Real-time immunization data were mainly used to monitor vaccination volume, track children with incomplete vaccinations, develop outreach visit plans, alter existing microplans, and provide a results-based fuel allowance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

During these meetings, vaccinator performance was reviewed, including that of poor performers.

“During these meetings, vaccinator performance was reviewed, including that of poor performers.”

Classified as probably not a substantial claim.

This perception was attributed to photo verification of the encounters.

“this perception was attributed to photo verification of the encounters.”

May combine two claims.

For district management, key enablers included a sense of empowerment for reviewing and planning EPI delivery and enhanced transparency to counteract political patronage of vaccinators.

“For district management, key enablers included a sense of empowerment for reviewing and planning EPI delivery and enhanced transparency to counteract political patronage of vaccinators.”

May combine two claims.

Regular EPI review meetings were convened at the district health office, and the meetings used the real-time app data.

“The key informants mentioned that regular EPI review meetings were convened at the district health office, and the meetings used the real-time app data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

EPI supervisors attributed improved data to the transparency of vaccinator movements.

“The EPI supervisors attributed it to the transparency of vaccinator movements”

May combine two claims.

District supervisors found the app data easy to access for quick verification of immunization activity and to provide supervisory direction.

“District supervisors found the app data easy to access for quick verification of immunization activity and to provide supervisory direction.”

Classified as probably not a substantial claim.

Disciplinary actions such as notice of explanation, show cause letter, and termination of irregular, ghosting, or nonperforming vaccinators were not effective due to political patronage of vaccinators by local legislators.

“Furthermore, disciplinary actions such as a “notice of explanation,” for which employees must provide an explanation for an allegation of violation of company policies, rules, and procedures; “show cause letter,” which asks the employee to provide a reason why they should not face a disciplinary action for a conduct or capacity issue in the workplace; and termination of irregular, ghosting, or non”

Probably not a substantial claim: may not pass S3 (load-bearing).

Vaccinators with high performance were recognized separately in quarterly public ceremonies attended by district legislators.

“Vaccinators with high performance were also recognized separately in quarterly public ceremonies attended by district legislators.”

Classified as probably not a substantial claim.

The validity of the app data was considered superior to that of data from manual records by both the vaccinators and district supervisors.

“The validity of the app data was considered superior to that of data from manual records by both the vaccinators and district supervisors”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Review meetings provided a platform for vaccinators to learn from each other's experiences and promote healthy competition.

“They added that the review meetings provided a platform for vaccinators to learn from each other’s experiences and promote healthy competition.”

May combine two claims.

This helped to not only accurately track the vaccinator but also identify vaccination coverage according to the geographical location.

“This helped to not only accurately track the vaccinator but also identify vaccination coverage according to the geographical location.”

Probably not a substantial claim: may not pass S2 (checkable).

Findings are relevant for informing the use of digital technology for results-based delivery by frontline health workers.

“Findings are relevant not only for wide-scale implementation of immunization tracking apps in Pakistan but also for informing the use of digital technology for results-based delivery by frontline health workers.”

• AssumptionRests on an assumption: The experiences and perceptions reported by vaccinators and district managers in the TMK district are representative enough to inform wide-scale implementation in Pakistan and other settings.

Probably not a substantial claim: may not pass S3 (load-bearing).

The study team randomly checked the photo entries and uploaded data for errors.

“The study team randomly checked the photo entries and uploaded data for errors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The country target is 90% set by the Global Vaccine Action Plan.

“In Pakistan, 66% of children are completely immunized; this is much lower than the country target of 90% set by the Global Vaccine Action Plan [23].”

• cites [23] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In Pakistan, 66% of children are completely immunized.

“In Pakistan, 66% of children are completely immunized; this is much lower than the country target of 90% set by the Global Vaccine Action Plan [23].”

• cites [23] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Interviews were conducted with all 26 vaccinators working in the 11 UCs in which the digital app was implemented.

“Interviews were conducted with all 26 vaccinators working in the 11 UCs in which the digital app was implemented.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Experience from high-income settings emphasizes a user-centric approach to the development of digital technology.

“Experience from high-income settings emphasizes a user-centric approach to the development of digital technology [21].”

• cites [21] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Technological factors such as internet access, data volume, and app usability can affect implementation.

“Technological factors such as internet access, data volume, and app usability can affect implementation [22].”

• cites [22] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

All the vaccinators used the immunization app for at least 24 months.

“All the vaccinators used the immunization app for at least 24 months.”

Classified as probably not a substantial claim.

Not all digital health programs perform as intended.

“Despite the proliferation of mHealth apps, not all digital health programs perform as intended, and assessments of the technology, health systems, and behavioral factors need careful consideration [18].”

• cites [18] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Findings were triangulated in line with the three broad research areas.

“Findings were triangulated in line with the three broad research areas.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Improved coordination as a result of easy, transparent data access can be an important by-product of digitalization.

“whereas improved coordination as a result of easy, transparent data access can be an important by-product of digitalization.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The interview information was transcribed in Sindhi and translated back into English, checked for errors, and manually analyzed.

“The interview information was transcribed in Sindhi and translated back into English. These were then checked for errors and manually analyzed.”

Classified as probably not a substantial claim. May combine two claims.

Content analysis of the transcripts was undertaken, developing main codes in line with the a priori themes and further detailed coding based on grounded findings emerging from the narrative.

“Content analysis of the transcripts was undertaken, developing main codes in line with the a priori themes and further detailed coding based on grounded findings emerging from the narrative.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In LMICS, mHealth apps are increasingly used for immunizations and other public health concerns.

“In LMICS, mHealth apps are increasingly used for immunizations and other public health concerns”

• cites [6] sourceNot checked yet

  Not checked in the free run.

• cites [7] sourceNot checked yet

  Not checked in the free run.

• cites [10] Kazi, A, 2016, J Vaccines VaccinNot checked yet

  Not checked in the free run.

• cites [12] sourceNot checked yet

  Not checked in the free run.

• cites [21] sourceNot checked yet

  Not checked in the free run.

• cites [22] sourceNot checked yet

  Not checked in the free run.

• cites [23] sourceNot checked yet

  Not checked in the free run.

• cites [24] JSIAfrica Routine Immunization System EssentialImproving the performance of routine immunization systems in Africa ? A case of the drivers of routine immunization system performance in Cameroonhttps://3qvw4sd8qlr2o7003493odw1-wpengine.netdna-ssl.com/wp-content/uploads/2012/11/Arise_CameroonReport_final5081.pdfNot checked yet

  Not checked in the free run.

• cites [25] JSI20182020-01-10Strengthening Immunization for the Urban Poor: Experiences and Lessons Learned from Five Country Studieshttps://www.jsi.com/JSIInternet/Resources/publication/display.cfm?txtGeoArea=INTL&id=22079&thisSection=ResourcesNot checked yet

  Not checked in the free run.

May combine two claims.

The evidence indicates that instant data availability for monitoring purposes, as observed with neglected tropical diseases, and ease of use and digital literacy of end users, as observed with oral cholera vaccinations, are important parameters for the successful trialing of digital apps.

“The evidence indicates that instant data availability for monitoring purposes, as observed with neglected tropical diseases [28], and ease of use and digital literacy of end users, as observed with oral cholera vaccinations [29], are important parameters for the successful trialing of digital apps.”

• cites [28] sourceNot checked yet

  Not checked in the free run.

• cites [29] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Nigeria study reported improvements in microplanning, monitoring, and immunization coverage, which were attributed to technology-based support for performance monitoring that reduced erroneous entries and data fabrication by vaccinators.

“They reported improvements in microplanning, monitoring, and immunization coverage, which were attributed to the technology-based support for performance monitoring that reduced erroneous entries and data fabrication by vaccinators.”

• cites [26] sourceNot checked yet

  Not checked in the free run.

• cites [27] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A household vaccination coverage survey conducted as part of the larger health system strengthening immunization study found a significant difference in vaccination completion rates between the baseline and end timepoints across study control and intervention areas.

“A household vaccination coverage survey conducted as part of the larger health system strengthening immunization study found a significant difference in vaccination completion rates between the baseline and end timepoints across study control and intervention areas [35].”

• cites [35] Gavi-The Vaccine AllianceAnnual Progress Report 2018https://www.gavi.org/results/gavi-progress-reports/Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

This study used iterative qualitative research methods for in-depth exploration of the digital technology interface with end users and decision makers within the health system.

“This study used iterative qualitative research methods for in-depth exploration of the digital technology interface with end users and decision makers within the health system.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Poorly designed devices and inadequate cellular infrastructure have been noted as barriers for integrating apps into health systems.

“Poorly designed devices and inadequate cellular infrastructure have been noted as barriers for integrating apps into health systems [30].”

• cites [30] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Ease of operability, satisfaction with reliable data, personal recognition, results-based field support, and empowerment are powerful enablers.

“Ease of operability, satisfaction with reliable data, personal recognition, results-based field support, and empowerment are powerful enablers.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A study on an mHealth app for midwives in Ghana found low levels of user acceptance and a software design that did not match the end‐user needs or work environment.

“A study on an mHealth app for midwives in Ghana found low levels of user acceptance and a software design that did not match the end‐user needs or work environment [32].”

• cites [32] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A study of a client data app for community health nurses in Ghana resulted in reasonably good acceptability due to the app's capacity to facilitate client follow-up and data reporting; however, the feasibility and usability of the app were hindered by high client volumes, staff shortages, and software and device challenges.

“A study of a client data app for community health nurses in Ghana resulted in reasonably good acceptability due to the app’s capacity to facilitate client follow-up and data reporting; however, the feasibility and usability of the app were hindered by high client volumes, staff shortages, and software and device challenges [31].”

• cites [31] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Triangulation of the findings across vaccinators, supervisors, and district managers strengthened the analysis and helped draw out common convincing narratives.

“Triangulation of the findings across vaccinators, supervisors, and district managers strengthened the analysis and helped draw out common convincing narratives.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

An important by-product of digitalization can be improved coordination as a result of transparent, easy data access.

“An important by-product of digitalization can be improved coordination as a result of transparent, easy data access.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

While several of the findings are specific to the study context, the study provides the key parameters for consideration during the current rollout of digital immunization interventions in Pakistan as well as lessons for the introduction of a health provider app in LMICS.

“While several of the findings are specific to the study context, the study provides the key parameters for consideration during the current rollout of digital immunization interventions in Pakistan as well as lessons for the introduction of a health provider app in LMICS.”

Probably not a substantial claim: may not pass S2 (checkable).

It also provides lessons for LMICS on technological and non-technological factors that require attention to contextualize health provider apps within the local health system.

“It also provides lessons for LMICS on technological and non-technological factors that require attention to contextualize health provider apps within the local health system.”

Probably not a substantial claim: may not pass S2 (checkable).

Our study provides timely evidence on the successful integration of an immunization tracking app to be used by vaccinators as Pakistan prepares to upscale digitalized immunization tracking.

“Our study provides timely evidence on the successful integration of an immunization tracking app to be used by vaccinators as Pakistan prepares to upscale digitalized immunization tracking.”

Probably not a substantial claim: may not pass S4 (based).

Embedding digital technology into mainstream health systems relies on acceptance by both end users and district stakeholders.

“Embedding digital technology into mainstream health systems relies on acceptance by both end users and district stakeholders.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

This iterative approach helped identify themes in an unexplored area and identified details that would have been missed in a more prescriptive quantitative study.

“This iterative approach helped identify themes in an unexplored area and identified details that would have been missed in a more prescriptive quantitative study.”

Classified as probably not a substantial claim. May combine two claims.

Technological ease of operability, personal recognition, results-oriented mobility support, and empowerment to improvise microplanning are powerful individual-level factors, whereas data transparency and empowerment for district planning are key organizational-level enablers to embed digital tracking in vaccine delivery systems.

“Technological ease of operability, personal recognition, results-oriented mobility support, and empowerment to improvise microplanning are powerful individual-level factors, whereas data transparency and empowerment for district planning are key organizational-level enablers to embed digital tracking in vaccine delivery systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

To limit bias, researchers who were not part of the intervention team undertook tool development and data collection.

“To limit bias, researchers who were not part of the intervention team undertook tool development and data collection.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Immunization performance tracking through digital technology needs essential acceptance by frontline health force, and mere vertical enforcement is not enough.

“We contend that immunization performance tracking through digital technology needs essential acceptance by frontline health force, and mere vertical enforcement is not enough.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The findings are relevant to not only the current upscaling of digital technology to track immunization in Pakistan but also the application of digital technology for results-based delivery by frontline health workers.

“The findings are relevant to not only the current upscaling of digital technology to track immunization in Pakistan but also the application of digital technology for results-based delivery by frontline health workers.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Another study that introduced an app for community health volunteers in Kenya found that acceptability improved after a period of initial anxiety; however, there were feasibility challenges related to battery drainage and difficulty keeping the phone charged.

“Another study that introduced an app for community health volunteers in Kenya found that acceptability improved after a period of initial anxiety; however, there were feasibility challenges related to battery drainage and difficulty keeping the phone charged [33].”

• cites [33] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Yet another study in South Africa with an app for mHealth workers to electronically track patients with multi-drug resistant TB found a high intent for use but low actual usage resulting from forgetfulness and low levels of responsibility for such work.

“Yet another study in South Africa with an app for mHealth workers to electronically track patients with multi-drug resistant TB found a high intent for use but low actual usage resulting from forgetfulness and low levels of responsibility for such work [34].”

• cites [34] sourceNot checked yet

  Not checked in the free run.