The DOI matches its Crossref record; 98 of the 98 references it cites were found in a registry or an index.
Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches
Open accessAbstract
The field of digital health, and its meaning, has evolved rapidly over the last 20 years. For this article we followed the most recent definition provided by FDA in 2020. Emerging solutions offers tremendous potential to positively transform the healthcare sector. Despite the growing number of applications, however, the evolution of methodologies to perform timely, cost-effective and robust evaluations have not kept pace. It remains an industry-wide challenge to provide credible evidence, therefore, hindering wider adoption. Conventional methodologies, such as clinical trials, have seldom been applied and more pragmatic approaches are needed. In response, several academic centers such as researchers from the Institute of Global Health Innovation at Imperial College London have initiated a digital health clinical simulation test bed to explore new approaches for evidence gathering relevant to solution type and maturity. The aim of this article is to: (1) Review current research approaches and discuss their limitations; (2) Discuss challenges faced by different stakeholders in undertaking evaluations; and (3) Call for new approaches to facilitate the safe and responsible growth of the digital health sector.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · An Integrated Approach to Communication Theory and Research · doi:10.4324/9780203710753-35 · Cited by 37,322
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk
Goff, D. C., Lloyd-Jones, D. M., Bennett, G., Coady, S., D’Agostino, R. B., Gibbons, R., Greenland, P., Lackland, D. T., Levy, D., O’Donnell, C. J., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.005 · Cited by 4,945
How we design feasibility studies
Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D., Bakken, S., Kaplan, C. P., Squiers, L., Fabrizio, C., et al. (2009) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2009.02.002 · Cited by 4,021 · cited 1×
“…– post design involves: pre-phase, which provides control data; “ washout ” period 55 (i.e., with no interventions implemented with a time gap up to several months), to allow familiarization and to limit bias related to implementation 39,56 ; post-phase to collect data on solution effectiveness.”
MethodAssessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 1×
“Prospective RCTs are the most accepted method for healthcare interventions 31 .”
MethodCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Clinical Practice Guidelines We Can Trust
Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Board on Health Care Services, & Institute of Medicine (2011) · National Academies Press eBooks · doi:10.17226/13058 · Cited by 3,081
Research works printed but not linked: 7
- Ohl, et al. Impact of availability of telehealth programs on documented HIV viral suppression: A Cluster-Randomized Program Evaluation in the Veterans Health Administration. Open Forum Infect Dis. 6, ofz206 (2019). · doi:10.1093/ofid/ofz206
- Sauerland, S. et al. Retrospective clinical studies in surgery: potentials and pitfalls. J. Hand Surg. Br. 27, 117–121 (2002). · doi:10.1054/jhsb.2001.0703
- Dahabreh, I. J. et al. Modeling and Simulation in the Context of Health Technology Assessment: Review of Existing Guidance, Future Research Needs, and Validity Assessment (2017).
- Anderson, J. G. et al. Evaluation in health informatics: computer simulation. Comput. Biol. Med. 32, 151–164 (2002). · doi:10.1016/s0010-4825(02)00012-4
- Roberts, S. D. Tutorial on the simulation of healthcare systems. Proceedings of the 2011 winter simulation conference (wsc), 1403–1414 (2011). · doi:10.1109/wsc.2011.6147860
- Yao, H. et al. Research and design on distributed remote simulation based on Web. In IEEE International Conference on Information Management and Engineering. pp. 522–525 (2010). · doi:10.1109/icime.2010.5477791
- Calabria, et al. Open triple therapy for chronic obstructive pulmonary disease: Patterns of prescription, exacerbations and healthcare costs from a large Italian claims database. Pulmon. Pharmacol. Therap. 61, 101904 (2020). · doi:10.1016/j.pupt.2020.101904
Cited by
Whitelaw, S., Pellegrini, D. M., Mamas, M. A., Cowie, M., & Van Spall, H. G. C. (2021) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztab005 · Cited by 241
Digital health competencies in medical school education: a scoping review and Delphi method study
Khurana, M. P., Raaschou-Pedersen, D. E., Kurtzhals, J., Bardram, J. E., Ostrowski, S. R., & Bundgaard, J. S. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03163-7 · Cited by 129
Zangger, G., Bricca, A., Liaghat, B., Juhl, C. B., Mortensen, S. R., Andersen, R. M., Damsted, C., Hamborg, T. G., Ried-Larsen, M., Tang, L. H., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/46439 · Cited by 116
Iyamu, I., Gómez-Ramírez, O., Xu, A. X., Chang, H.-J., Watt, S., Mckee, G., & Gilbert, M. (2022) · DIGITAL HEALTH · doi:10.1177/20552076221102255 · Cited by 118
“7,8 Many of these challenges apply to digital health generally and are not speci fi c to digital public health.”
BackgroundJayakumar, S., Sounderajah, V., Normahani, P., Harling, L., Markar, S. R., Ashrafian, H., & Darzi, A. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00544-y · Cited by 110
Self-citation
Metadata
- Authors
- Guo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner, C., & Prime, M.
- Venue
- npj Digital Medicine
- Published
- 2020-08-27
- Volume
- 3
- Issue
- 1
- Pages
- 110-110
- ISSN
- 2398-6352 (ISSN-L)
- OpenAlex
- W3080862813
- Cited by
- 316
- References
- 98
- FWCI
- 37.16
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches. https://app.cochat.ai/papers/10.1038/s41746-020-00314-2
- The paper
- Guo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner, C., & Prime, M. (2020). Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches. npj Digital Medicine. https://doi.org/10.1038/s41746-020-00314-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.