Unmet Need: Improving mHealth Evaluation Rigor to Build the Evidence Base
Abstract
mHealth-the use of mobile technologies for health-is a growing element of health system activity globally, but evaluation of those activities remains quite scant, and remains an important knowledge gap for advancing mHealth activities. In 2010, the World Health Organization and Columbia University implemented a small-scale survey to generate preliminary data on evaluation activities used by mHealth initiatives. The authors describe self-reported data from 69 projects in 29 countries. The majority (74%) reported some sort of evaluation activity, primarily nonexperimental in design (62%). The authors developed a 6-point scale of evaluation rigor comprising information on use of comparison groups, sample size calculation, data collection timing, and randomization. The mean score was low (2.4); half (47%) were conducting evaluations with a minimum threshold (4+) of rigor, indicating use of a comparison group, while less than 20% had randomized the mHealth intervention. The authors were unable to assess whether the rigor score was appropriate for the type of mHealth activity being evaluated. What was clear was that although most data came from mHealth projects pilots aimed for scale-up, few had designed evaluations that would support crucial decisions on whether to scale up and how. Whether the mHealth activity is a strategy to improve health or a tool for achieving intermediate outcomes that should lead to better health, mHealth evaluations must be improved to generate robust evidence for cost-effectiveness assessment and to allow for accurate identification of the contribution of mHealth initiatives to health systems strengthening and the impact on actual health outcomes.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Equipping frontline health workers with mobile phone applications may enhance the delivery of maternal, neonatal, and child health services as well as the accuracy of data capture and recording.
“Equipping frontline health workers with mobile phone applications may enhance the delivery of maternal, neonatal, and child health services as well as the accuracy of data capture and recording [16, 26, 35, 36].”
Cites
Utilization-Focused Evaluation
(2018) · The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation · doi:10.4135/9781506326139.n729 · Cited by 2,315 · see citing works
Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use
Patton, M. Q. (2010) · Cited by 1,740 · see citing works
Text messaging as a tool for behavior change in disease prevention and management
Cole-Lewis, H., & Kershaw, T. (2010) · Epidemiologic Reviews · doi:10.1093/epirev/mxq004 · Cited by 1,190 · see citing works
Mobile Health Technology Evaluation
Kumar, S., Nilsen, W. J., Abernethy, A., Atienza, A., Patrick, K., Pavel, M., Riley, W. T., Shar, A., Spring, B., Spruijt-Metz, D., et al. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.03.017 · Cited by 900 · see citing works
Scaling Up mHealth: Where Is the Evidence?
Tomlinson, M., Rotheram-Borus, M. J., Swartz, L., & Tsai, A. C. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001382 · Cited by 673 · see citing works
Utilization-Focused evaluation
Patton, M. Q. (2003) · International Handbook of Educational Evaluation · doi:10.1007/978-94-010-0309-4_15 · Cited by 334 · see citing works
Prgomet, M., Georgiou, A., & Westbrook, J. I. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3215 · Cited by 319 · see citing works
Advancing the science of mHealth
Nilsen, W., Kumar, S., Shar, A., Varoquiers, C., Wiley, T., Riley, W. T., Pavel, M., & Atienza, A. A. (2012) · Journal of Health Communication · doi:10.1080/10810730.2012.677394 · Cited by 226 · see citing works
Bloomfield, G. S., Vedanthan, R., Vasudevan, L., Kithei, A., Were, M., & Velazquez, E. J. (2014) · Globalization and Health · doi:10.1186/1744-8603-10-49 · Cited by 154 · see citing works
Data Collection Outcomes Comparing Paper Forms With PDA Forms in an Office-Based Patient Survey
Galliher, J. M., Stewart, T. V., Pathak, P. K., Werner, J. J., Dickinson, L. M., & Hickner, J. M. (2008) · The Annals of Family Medicine · doi:10.1370/afm.762 · Cited by 56 · see citing works
Blaya, J. A., Cohen, T., Rodríguez, P., Kim, J., & Fraser, H. (2008) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2008.09.015 · Cited by 34 · see citing works
Capitalizing on the Characteristics of mHealth to Evaluate Its Impact
Mechael, P., Nemser, B., Cosmaciuc, R., Cole-Lewis, H., Ohemeng-Dapaah, S., Dusabe, S., Kaonga, N. N., Namakula, P., Shemsanga, M., Burbach, R., et al. (2012) · Journal of Health Communication · doi:10.1080/10810730.2012.679847 · Cited by 26 · see citing works
Self-citation
Flagged first, then by citations
3 of the 15 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Albert, H., Nathavitharana, R. R., Isaacs, C., Pai, M., Denkinger, C. M., & Boehme, C. C. (2016) · European Respiratory Journal · doi:10.1183/13993003.00543-2016 · Cited by 302 · see citing works
Rothstein, J. D., Jennings, L., Moorthy, A., Yang, F., Gee, L., Romano, K., Hutchful, D., Labrique, A. B., & LeFevre, A. E. (2016) · International Journal of Telemedicine and Applications · doi:10.1155/2016/2515420 · Cited by 74 · see citing works
Alfonsi, J., Choi, E., Arshad, T., Sammott, S.-A. S., Pais, V., Nguyen, C., Maguire, B. R., Stinson, J., & Palmert, M. R. (2020) · JMIR mhealth and uhealth · doi:10.2196/22074 · Cited by 84 · see citing works
Parkes-Ratanshi, R. M., Kimeze, J., Nakku-Joloba, E., Hamill, M. M., Namawejje, M., Kiragga, A. N., Byamugisha, J. K., Rompalo, A. M., Gaydos, C. A., & Manabe, Y. C. (2020) · Sexual Health · doi:10.1071/sh19092 · Cited by 45 · see citing works
Huben, A. V., Howell, M., Howard, K., Carrello, J., & Norris, S. (2021) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462321000362 · Cited by 40 · see citing works
Segur-Ferrer, J., Moltó-Puigmartí, C., Pastells‐Peiró, R., & Vivanco‐Hidalgo, R. M. (2024) · Journal of Medical Internet Research · doi:10.2196/48694 · Cited by 33 · see citing works
Green, N., Mathur, S., Kiguli, S., Makani, J., Fashakin, V., LaRussa, P. S., Lyimo, M. A., Abrams, E. J., Mulumba, L., & Mupere, E. (2016) · Global Pediatric Health · doi:10.1177/2333794x16637767 · Cited by 31 · see citing works
Information and Communication Technologies and Global Health Challenges
Gutierrez, M. A., Moreno, R. A., & Rebelo, M. S. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-804591-6.00004-5 · Cited by 32 · see citing works
Innovative Healthcare Applications of ICT for Developing Countries
Ariani, A., Koesoema, A. P., & Soegijoko, S. (2017) · Understanding complex systems · doi:10.1007/978-3-319-55774-8_2 · Cited by 28 · see citing works
Considerations for Improved Mobile Health Evaluation: Retrospective Qualitative Investigation
Dick, S., O’Connor, Y., Thompson, M. J., O’Donoghue, J., Hardy, V., Wu, T.-S. J., O’Sullivan, T., Chirambo, G. B., & Heavin, C. (2019) · JMIR mhealth and uhealth · doi:10.2196/12424 · Cited by 21 · see citing works
Leon, N., Namadingo, H., Bobrow, K., Cooper, S., Crampin, A., Pauly, B., Levitt, N., & Farmer, A. (2021) · BMC Public Health · doi:10.1186/s12889-020-10089-6 · Cited by 18 · see citing works
Design Lessons from a Micro-Randomized Pilot Study in Mobile Health
Smith, S. N., Lee, J. S., Hall, K., Seewald, N. J., Boruvka, A., Murphy, S. A., & Klasnja, P. (2017) · Springer eBooks · doi:10.1007/978-3-319-51394-2_4 · Cited by 17 · see citing works
Leon, N., Namadingo, H., Cooper, S., Bobrow, K., Mwantisi, C., Nyasulu, M., Sicwebu, N., Crampin, A., Levitt, N., & Farmer, A. (2021) · BMC Public Health · doi:10.1186/s12889-021-11552-8 · Cited by 17 · see citing works
Westgard, C. M., & Orrego-Ferreyros, L. A. (2022) · PLOS Global Public Health · doi:10.1371/journal.pgph.0001118 · Cited by 10 · see citing works
Digital Health Research Methods and Tools: Suggestions and Selected Resources for Researchers
Gray, K. M., & Gilbert, C. A. (2017) · Intelligent systems reference library · doi:10.1007/978-3-319-67513-8_2 · Cited by 9 · see citing works
Mechael, P., Kaonga, N., Chandrasekharan, S., Muthuperumal, P., Peter, J., Ganju, A., & Murthy, N. (2019) · JMIR mhealth and uhealth · doi:10.2196/14668 · Cited by 9 · see citing works
Self-citationVakili, N., Curran, J., Walls, R. A., Phillips, D., Miller, A., Cassidy, C. E., & Wozney, L. (2024) · JMIR Formative Research · doi:10.2196/51690 · Cited by 3 · see citing works
Jurkeviciute, M., Enam, A., Torres-Bonilla, J., & Eriksson, H. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01399-9 · Cited by 3 · see citing works
Design Frameworks for Mobile Health Technology
Zhang, K., & Aslan, A. B. (2022) · International Journal of Health Systems and Translational Medicine · doi:10.4018/ijhstm.302653 · Cited by 3 · see citing works
Roberts, W., Lyson, H., Speer, C., Tovar, E., Paz, E., & Zimlichman, E. (2025) · Value in Health · doi:10.1016/j.jval.2025.01.025 · Cited by 3 · see citing works
Jurkeviciute, M. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0975-9 · Cited by 3 · see citing works
Goldberg, S. B., Torous, J. B., & Sun, S. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000127 · Cited by 2 · see citing works
Westgard, C. M., & Orrego-Ferreyros, L. A. (2021) · Research Square · doi:10.21203/rs.3.rs-995871/v1 · Cited by 1 · see citing works
Royston, G. (2017) · International Conference on Digital Health · doi:10.1145/3079452.3079466
Westgard, C. M. (2021) · Research Square · doi:10.21203/rs.3.rs-491310/v1
Leon, N., Namadingo, H., Cooper, S., Bobrow, K., Mwantisi, C., Nyasulu, M., Sicwebu, N., Crampin, A., Levitt, N. S., & Farmer, A. (2021) · Figshare · doi:10.6084/m9.figshare.16305799
Leon, N., Namadingo, H., Cooper, S., Bobrow, K., Mwantisi, C., Nyasulu, M., Sicwebu, N., Crampin, A., Levitt, N. S., & Farmer, A. (2021) · Figshare · doi:10.6084/m9.figshare.16305802
- Authors
- Mookherji, S., Mehl, G., Kaonga, N., & Mechael, P.
- Affiliations
- George Washington University (US)
- World Health Organization (CH)
- Columbia University (US)
- Corresponding author
- Mookherji, Sangeeta
- Venue
- Journal of Health Communication
- Publisher
- Informa UK Limited
- Published
- 2015-06-04
- Volume
- 20
- Issue
- 10
- Pages
- 1224-1229
- ISSN
- 1081-0730 (ISSN-L)
- 1087-0415
- OpenAlex
- W2167049058
- Cited by
- 27
- References
- 15
- Citations by year
- 2025: 1 · 2024: 2 · 2022: 3 · 2021: 8 · 2020: 2 · 2019: 3 · 2017: 5 · 2016: 3
- FWCI
- 4.90
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Health Communication (publishedVersion)
- PubMed (publishedVersion)
- Funders
- World Health Organization
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Physical Sciences › Computer Science › Information Systems › ICT in Developing Communities
- Keywords
- mobile health, evaluation rigor, randomized controlled trials, comparison groups, sample size determination, scale-up, evidence base, health system strengthening, cost-effectiveness
- MeSH terms
- Health Services Needs and Demand, Health Services Research, Humans, Global Health, Telemedicine, Health Care Surveys, Evidence-Based Practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
