Assessing the Clinical Robustness of Digital Health Startups: Cross-sectional Observational Analysis
Open accessPublished version of a preprint
The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Background: The digital health sector has experienced rapid growth over the past decade. However, health care technology stakeholders lack a comprehensive understanding of clinical robustness and claims across the industry. Objective: This analysis aimed to examine the clinical robustness and public claims made by digital health companies. Methods: A cross-sectional observational analysis was conducted using company data from the Rock Health Digital Health Venture Funding Database, the US Food and Drug Administration, and the US National Library of Medicine. Companies were included if they sell products targeting the prevention, diagnosis, or treatment phases of the care continuum. Clinical robustness was defined using regulatory filings and clinical trials completed by each company. Public claims data included clinical, economic, and engagement claims regarding product outcomes made by each company on its website. Results: A total of 224 digital health companies with an average age of 7.7 years were included in our cohort. Average clinical robustness was 2.5 (1.8 clinical trials and 0.8 regulatory filings) with a median score of 1. Ninety-eight (44%) companies had a clinical robustness score of 0, while 45 (20%) companies had a clinical robustness score of 5 or more. The average number of public claims was 1.3 (0.5 clinical, 0.4 economic, and 0.4 engagement); the median number of claims was 1. No correlation was observed between clinical robustness and number of clinical claims (r2=0.02), clinical robustness and total funding (r2=0.08), or clinical robustness and company age (r2=0.18). Conclusions: Many digital health companies have a low level of clinical robustness and do not make many claims as measured by regulatory filings, clinical trials, and public data shared online. Companies and customers may benefit from investing in greater clinical validation efforts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Effectiveness of Telemental Health: A 2013 Review
Hilty, D. M., Ferrer, D. C., Parish, M. B., Johnston, B., Callahan, E. J., & Yellowlees, P. M. (2013) · Telemedicine and e-Health · doi:10.1089/tmj.2013.0075 · Cited by 1,048
Digital health: a path to validation
Mathews, S. C., McShea, M. J., Hanley, C. L., Ravitz, A., Labrique, A. B., & Cohen, A. B. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0111-3 · Cited by 508
Self-citationWidmer, R. J., Collins, N. M., Collins, C. S., West, C. P., Lerman, L. O., & Lerman, A. (2015) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2014.12.026 · Cited by 416
Effectiveness of mHealth interventions for patients with diabetes: An overview of systematic reviews
Kitsiou, S., Paré, G., Jaana, M., & Gerber, B. (2017) · PLOS ONE · doi:10.1371/journal.pone.0173160 · Cited by 388
Using science to sell apps: Evaluation of mental health app store quality claims
Larsen, M. E., Huckvale, K., Nicholas, J., Torous, J., Birrell, L., Li, E., & Reda, B. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0093-1 · Cited by 334
Systematic Review of Economic Impact Studies of Artificial Intelligence in Health Care
Wolff, J., Pauling, J., Keck, A., & Baumbach, J. (2020) · Journal of Medical Internet Research · doi:10.2196/16866 · Cited by 261
Howarth, A., Quesada, J., Silva, J., Judycki, S., & Mills, P. R. (2018) · DIGITAL HEALTH · doi:10.1177/2055207618770861 · Cited by 240
(2019) · doi:10.2196/13166 · Cited by 222
Valuing Young, Start-Up and Growth Companies: Estimation Issues and Valuation Challenges
Damodaran, A. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1418687 · Cited by 191
Safety of patient-facing digital symptom checkers
Fraser, H., Coiera, E., & Wong, D. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32819-8 · Cited by 180
Research works printed but not linked: 2
- 10.1377/hlthaff.2018.05081 · DOI · Paper page
- 10.1038/s41746-020-0276-9 · DOI
Cited by
Golden, G. M., Popescu, C., Israel, S. A., Perlman, K., Armstrong, C., Fratila, R., Tanguay-Sela, M., & Benrimoh, D. A. (2024) · Health Policy and Technology · doi:10.1016/j.hlpt.2024.100844 · Cited by 95
Sun, S., Simonsson, O., McGarvey, S. T., Torous, J. B., & Goldberg, S. B. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00119-5 · Cited by 78
Galderisi, S., Appelbaum, P. S., Gill, N., Gooding, P., Herrman, H., Melillo, A., Myrick, K., Pathare, S., Savage, M., Szmukler, G., et al. (2024) · World Psychiatry · doi:10.1002/wps.21230 · Cited by 76
“Another study of digital health startups found no correlation between clinical robustness and the number of clinical claims, total funding, or company age 92 .”
O'Daffer, A., Colt, S. F., Wasil, A. R., & Lau, N. (2022) · JMIR Mental Health · doi:10.2196/40924 · Cited by 72
Jacob, C., Lindeque, J. P., Klein, A., Ivory, C. J., Heuss, S., & Peter, M. K. (2023) · JMIR Human Factors · doi:10.2196/45143 · Cited by 62
Choreographing for public value in digital health?
Gross, N., & Geiger, S. (2023) · Big Data & Society · doi:10.1177/20539517231220622 · Cited by 53
Clinical Validation of Digital Healthcare Solutions: State of the Art, Challenges and Opportunities
Gomis‐Pastor, M., Berdún, J., Borràs-Santos, A., López, A. D. D., Rama, B. F.-M., García-Esquirol, Ó., Gratacòs, M., Ontiveros, G., Cruz, R. P., Real, J., et al. (2024) · Healthcare · doi:10.3390/healthcare12111057 · Cited by 53
What is coming next in health technology startups? Some insights and practice guidelines
Chakraborty, I., Edirippulige, S., & Ilavarasan, P. V. (2023) · Digital Health · doi:10.1177/20552076231178435 · Cited by 48
Vase, L., Wager, T. D., & Eccleston, C. (2025) · The Lancet · doi:10.1016/s0140-6736(25)00404-0 · Cited by 47
MacKay, D., Chan, C. B., Dasgupta, K., Dominy, C., Gagner, M., Jin, S., Kim, J., Little, J. P., MacDonald, B., McInnes, N., et al. (2022) · Canadian Journal of Diabetes · doi:10.1016/j.jcjd.2022.10.004 · Cited by 46
Digital health in pain assessment, diagnosis, and management: Overview and perspectives
Hadjiat, Y., & Arendt‐Nielsen, L. (2023) · Frontiers in Pain Research · doi:10.3389/fpain.2023.1097379 · Cited by 45
The Application of Preventive Medicine in the Future Digital Health Era
la Torre, K. D., Min, S., Lee, H., & Kang, D. (2024) · Journal of Medical Internet Research · doi:10.2196/59165 · Cited by 45
Digital Health Programs to Reduce Readmissions in Coronary Artery Disease
Braver, J., Marwick, T. H., Oldenburg, B. F., Issaka, A., & Carrington, M. J. (2023) · JACC Advances · doi:10.1016/j.jacadv.2023.100591 · Cited by 38
Healthcare entrepreneurship: An integrative framework for future research
Glover, W. J., Crocker, A., & Brush, C. G. (2024) · Journal of Business Venturing Insights · doi:10.1016/j.jbvi.2024.e00476 · Cited by 38
Can digital health democratize health care?
Hendl, T., & Shukla, A. (2024) · Bioethics · doi:10.1111/bioe.13266 · Cited by 36
Cevasco, K. E., Morrison Brown, R. E., Woldeselassie, R., & Kaplan, S. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02059-x · Cited by 35
“One challenge is that most eHealth applications are marketed without documented effectiveness, and few have regulatory approval [2–4].”
BackgroundSánchez-Robles, M., Saura, J. R., & Soriano, D. R. (2023) · Review of Managerial Science · doi:10.1007/s11846-023-00670-9 · Cited by 31
Social Determinants of Digital Health Adoption: Pilot Cross-sectional Survey
Patel, S. P., Sun, E., Reinhardt, A., Geevarghese, S., He, S., & Gazmararian, J. A. (2022) · JMIR Formative Research · doi:10.2196/39647 · Cited by 30
Weimar, S. N., Martjan, R. S., & Terzidis, O. (2024) · Management Review Quarterly · doi:10.1007/s11301-024-00417-0 · Cited by 30
Digital Mental Health’s Unstable Dichotomy—Wellness and Health
Torous, J. B., Firth, J., & Goldberg, S. B. (2024) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2024.0532 · Cited by 24
Muharremi, G., Meçani, R., & Muka, T. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm14010053 · Cited by 20
Zhou, T., Salman, D., & McGregor, A. H. (2024) · JMIR mhealth and uhealth · doi:10.2196/53262 · Cited by 18
The case for inclusive co-creation in digital health innovation
Nickel, G. C., Wang, S., Kwong, J. C. C., & Kvedar, J. C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01256-9 · Cited by 17
MedTech start-ups: A comprehensive scoping review of current research trends and future directions
Kalinowska-Beszczyńska, O., & Prędkiewicz, K. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307959 · Cited by 15
Eccleston, C., Fisher, E., Keefe, F. J., Palermo, T. M., & Toelle, T. R. (2024) · Pain · doi:10.1097/j.pain.0000000000003348 · Cited by 12
Clinical Simulation in the Regulation of Software as a Medical Device: An eDelphi Study
Ghafur, S., O’Brien, N., O’Brien, N., Guo, C., Prime, M. S., & Darzi, A. W. (2024) · JMIR Formative Research · doi:10.2196/56241 · Cited by 11
Key Considerations for Designing Clinical Studies to Evaluate Digital Health Solutions
Bolinger, E., & Tyl, B. (2024) · Journal of Medical Internet Research · doi:10.2196/54518 · Cited by 10
Lazarus, J. V., Villota‐Rivas, M., Jiménez-González, C., Santos‐Laso, Á., Iruzubieta, P., Arias, M., Rice-Duek, L., Leigh, S., Kopka, C. J., Turnés, J., et al. (2023) · Clinics in Liver Disease · doi:10.1016/j.cld.2023.01.021 · Cited by 10
Gai, Y., Crocker, A., Brush, C. G., & Glover, W. J. (2024) · International Journal of Entrepreneurial Behaviour & Research · doi:10.1108/ijebr-02-2023-0204 · Cited by 10
Sawicki, G. S., Goodman, A. S., Bacon, C., Collins, L., Greenberg, J., Duncan, C. L., Frederick, C., Luebbert, S., Mazuera, S., Polineni, D., et al. (2023) · Journal of Cystic Fibrosis · doi:10.1016/j.jcf.2023.04.015 · Cited by 9
Emel’yanov, A. V., Анатольевна, Ж. Е., Kozhevnikova, M. V., Zektser, V., Kamalova, A., Privalova, E. V., & Belenkov, Y. N. (2024) · Biomedical Engineering · doi:10.1007/s10527-023-10322-7 · Cited by 8
Morley, J., Laitila, J., Ross, J. S., Schamroth, J., Zhang, J., & Floridi, L. (2025) · Minds and Machines · doi:10.1007/s11023-025-09710-7 · Cited by 8
Belfort, E. L., MACMILLAN, C. M., & Weigle, P. E. (2024) · Child and Adolescent Psychiatric Clinics of North America · doi:10.1016/j.chc.2024.03.010 · Cited by 7
Telehealth’s Role Enabling Sustainable Innovation and Circular Economies in Health
Kalogeropoulos, D. (2023) · Telehealth and Medicine Today · doi:10.30953/thmt.v8.409 · Cited by 7
Growth of Remote Therapeutic Monitoring Lands New Opportunities for Case Management
Moreo, K., & Sapir, T. (2023) · Professional Case Management · doi:10.1097/ncm.0000000000000688 · Cited by 3
Vinay, R., Probst, J., Huynh, P., Schlögl, M., Kowatsch, T., & Nißen, M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0323390 · Cited by 4
Modern view on the possibilities of patients with rheumatoid arthritis remote treatment monitoring
Prokofeva, Y. A., Вадимовна, М. И., & Анатольевна, Ж. Е. (2024) · Russian Journal of Preventive Medicine · doi:10.17116/profmed20242711140 · Cited by 3
Technology exceptionalism in medicine: Is it time to rethink the paradigm?
Achtyes, E. D., Glenn, T., Monteith, S., Geddes, J. R., Bauer, M., & Whybrow, P. C. (2023) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2023.05.002 · Cited by 2
Entrepreneurial Founding Conditions and the Survival of Startups in Ghana
Apprey, P. L. (2025) · Kent Academic Repository (University of Kent) · doi:10.22024/unikent/01.02.111130 · Cited by 2
Remote monitoring of patients with chronic heart failure: a non invasive approach
Emelianov, A. V., Kozhevnikova, M. V., Анатольевна, Ж. Е., Panova, A. L., Privalova, E. V., & Belenkov, Y. N. (2024) · Digital Diagnostics · doi:10.17816/dd633033 · Cited by 1
de Groot, P., Fadaei, S., Luime, J. J., Wagenaar, W., Vis, M., Kok, M. R., Bos, W. H., & Tchetverikov, I. (2025) · EULAR Rheumatology Open · doi:10.1016/j.ero.2025.11.021 · Cited by 1
Yassaee, A., Reed, A. B., Swanson, B., Neves, A. L., & Hargreaves, D. S. (2025) · Journal of Medical Internet Research · doi:10.2196/68910 · Cited by 1
Automatische Bildanalyse und künstliche Intelligenz in der Nephropathologie
Bülow, R. D., Hölscher, D. L., & Boor, P. (2022) · Die Nephrologie · doi:10.1007/s11560-022-00598-3
Foundations for safer evidence-based digital health technologies under construction
Valenzuela‐Pascual, C., Anmella, G., Sanabra, M., Mas, A., & Hidalgo‐Mazzei, D. (2023) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2023.06.009
Mathews, S. C. (2023) · Journal of Medical Internet Research · doi:10.2196/51438
Self-citationA Narrow Definition of Clinical Robustness
Marshall, J., Yurkovic, A., Thames, T., & Parekh, A. (2023) · Journal of Medical Internet Research · doi:10.2196/41715
Shoucair, S., Downing, G. J., O’Rourke, J., Greenberg, L., Yea, M., Vatas, G., Kim, P., Fishbein, T., Unger, K. R., & Winslow, E. R. (2023) · BMC Digital Health · doi:10.1186/s44247-023-00049-6
Challenges in collecting evidence for the safe use of digital health solutions
Balenović, A., & Mesarić, J. (2024) · Liječnički vjesnik · doi:10.26800/lv-145-11-12-2
Identifying and Prioritizing Key Challenges in Launching Startups in the Health Sector in Iran
Amiri, S., Rezaei, B., & Naderi, N. (2025) · Journal of Health Reports and Technology · doi:10.5812/jhrt-150048
Siddiqui, M. Z., Khadim, N., Asab, M. Z., Zulfaqar, F., & Abrar, M. M. (2025) · Contemporary Journal of Social Science Review · doi:10.63878/cjssr.v3i3.974
Burns, K., Bondaronek, P., Low, J., Banerjee, C., Kyffin, S., Sheldon, H., & Wallace, P. (2025) · BMJ Innovations · doi:10.1136/bmjinnov-2024-001341
Nagappan, A., Zhu, X., Moucheraud, C., & Jung, O. (2025) · JMIR Formative Research · doi:10.2196/78431
Evaluation of Digital Health Technologies
Prime, M. S., Ghafur, S., Fontana, G., & van Haasteren, A. (2025) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780198882442.013.0021
van Gelder, N. E. (2024) · Radboud Repository (Radboud University)
Digital Health Entrepreneurship: A Founder’s Perspective on Ecosystem-Supported Growth
Giunti, G., Guardado, S., & Tripathi, N. (2025) · Springer eBooks · doi:10.1007/978-3-032-04294-1_15
Proteção e Sustentabilidade de Diretrizes de UX Inclusiva para Neurodivergentes
Junior, A. S. D. C., Nunes, E. P. S., & Figueiredo, J. M. D. (2025) · doi:10.5753/eri-mt.2025.17015
The United States Leads the Globe in Venture Capital Funding for Orthopaedic Surgery
Sridharan, M. J., Wahle, C. F., Walker, P., Barry, L. E., Yu, J. S. C., Petrigliano, F. A., Wessel, L. E., & Jensen, A. R. (2026) · Arthroscopy Sports Medicine and Rehabilitation · doi:10.1002/ars2.70002
Developer evidence and user satisfaction in mental health apps across China and the United States
Liang, J., Ma, J., Li, X., Peng, X., Fang, Q., Zhang, Z., Pan, H., Zhao, L., Liu, Y., He, Y. Q., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03026-1
The effectiveness of AI applications in healthcare startups
Petrenko, Z. O., & Samus, P. O. (2024) · eNTUKhPIIR Repository (Kharkiv Polytechnic Institute)
Теоретичні та практичні дослідження молодих вчених
(2020) · eNTUKhPIIR Repository (Kharkiv Polytechnic Institute)
Fedorova, E., Volkov, A., Morozova, I., & Kuznetsov, D. A. (2026) · Archives of Pharmacy Practice · doi:10.51847/rjpzvoq1ql
Artificial Intelligence in Cardiovascular Care
Valentin, C., Cámara, Ó., Bijnens, B. H., Alan, F., Karen, M., Alessandro, Z., Lorenzo, B. A., Victoria, R. S., Nikolaos, S., Griesinger, C. B., et al. (2026) · Joint Research Centre (European Commission) · doi:10.2760/4314186
Advancing Patient‐Centred AI With Adaptive Machine Learning
Kalogeropoulos, D., Barach, P. R., Downing, A., Verhulst, S., & Lustberg, M. B. (2026) · Cancer Medicine · doi:10.1002/cam4.72371
Metadata
- Authors
- Day, S., Shah, V., Kaganoff, S., Powelson, S., & Mathews, S. C.
- Affiliations
- University of Cambridge (GB)
- Johns Hopkins University (US)
- Corresponding author
- Mathews, Simon C
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2022-06-20
- Volume
- 24
- Issue
- 6
- Pages
- e37677
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC9253972
- OpenAlex
- W4283164418
- Cited by
- 57
- References
- 28
- Citations by year
- 2026: 3 · 2025: 10 · 2024: 23 · 2023: 17 · 2022: 4
- FWCI
- 10.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Keywords
- digital health startups, regulatory submissions, clinical trials, public claims, health technology companies, clinical validation, venture financing
- MeSH terms
- Child, Cross-Sectional Studies, Data Collection, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.