CoChat

Assessing the Clinical Robustness of Digital Health Startups: Cross-sectional Observational Analysis

Open access
Day, Sean, Shah, Veeraj, Kaganoff, Sari, Powelson, Shannon, Mathews, Simon C · 2022 · Journal of Medical Internet Research

Published version of a preprint

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.

Abstract

Background: The digital health sector has experienced rapid growth over the past decade. However, health care technology stakeholders lack a comprehensive understanding of clinical robustness and claims across the industry. Objective: This analysis aimed to examine the clinical robustness and public claims made by digital health companies. Methods: A cross-sectional observational analysis was conducted using company data from the Rock Health Digital Health Venture Funding Database, the US Food and Drug Administration, and the US National Library of Medicine. Companies were included if they sell products targeting the prevention, diagnosis, or treatment phases of the care continuum. Clinical robustness was defined using regulatory filings and clinical trials completed by each company. Public claims data included clinical, economic, and engagement claims regarding product outcomes made by each company on its website. Results: A total of 224 digital health companies with an average age of 7.7 years were included in our cohort. Average clinical robustness was 2.5 (1.8 clinical trials and 0.8 regulatory filings) with a median score of 1. Ninety-eight (44%) companies had a clinical robustness score of 0, while 45 (20%) companies had a clinical robustness score of 5 or more. The average number of public claims was 1.3 (0.5 clinical, 0.4 economic, and 0.4 engagement); the median number of claims was 1. No correlation was observed between clinical robustness and number of clinical claims (r2=0.02), clinical robustness and total funding (r2=0.08), or clinical robustness and company age (r2=0.18). Conclusions: Many digital health companies have a low level of clinical robustness and do not make many claims as measured by regulatory filings, clinical trials, and public data shared online. Companies and customers may benefit from investing in greater clinical validation efforts.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/37677

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Average number of claims made by start-ups broken out by customer type.
Figure 1 Average number of claims made by start-ups broken out by customer type.CC BY · PubMed Central
Figure 2 Clinical robustness, total claims, and total company funding across the digital health landscape.
Figure 2 Clinical robustness, total claims, and total company funding across the digital health landscape.CC BY · PubMed Central
Figure 3 Funding and clinical robustness across various clinical areas.
Figure 3 Funding and clinical robustness across various clinical areas.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 28 references

Research works printed but not linked: 2

  1. 10.1377/hlthaff.2018.05081 · DOI · Paper page
  2. 10.1038/s41746-020-0276-9 · DOI

Cited by

Metadata

Authors
Day, S., Shah, V., Kaganoff, S., Powelson, S., & Mathews, S. C.
Affiliations
  • University of Cambridge (GB)
  • Johns Hopkins University (US)
Corresponding author
Mathews, Simon C
Venue
Journal of Medical Internet Research
Published
2022-06-20
Volume
24
Issue
6
Pages
e37677
ISSN
1438-8871 (ISSN-L)
OpenAlex
W4283164418
Cited by
57
References
28
Citations by year
2026: 3 · 2025: 10 · 2024: 23 · 2023: 17 · 2022: 4
FWCI
10.11
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
Keywords
digital health startups, regulatory submissions, clinical trials, public claims, health technology companies, clinical validation, venture financing
MeSH terms
Child, Cross-Sectional Studies, Data Collection, Humans

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.