CoChat
Sign Up
Cited by 162Mobile Health and mHealth ApplicationsStatistical Methods in Clinical TrialsDigital Mental Health Interventions

Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods

Open access
Pham, Quynh, Wiljer, David, Cafazzo, Joseph A·2016·JMIR mHealth and uHealth
DOI 10.2196/mhealth.5720PMC PMC5035379
Open this paper in CoChat — free account

Abstract

BACKGROUND: Randomized controlled trials (RCTs) have long been considered the primary research study design capable of eliciting causal relationships between health interventions and consequent outcomes. However, with a prolonged duration from recruitment to publication, high-cost trial implementation, and a rigid trial protocol, RCTs are perceived as an impractical evaluation methodology for most mHealth apps. OBJECTIVE: Given the recent development of alternative evaluation methodologies and tools to automate mHealth research, we sought to determine the breadth of these methods and the extent that they were being used in clinical trials. METHODS: We conducted a review of the ClinicalTrials.gov registry to identify and examine current clinical trials involving mHealth apps and retrieved relevant trials registered between November 2014 and November 2015. RESULTS: Of the 137 trials identified, 71 were found to meet inclusion criteria. The majority used a randomized controlled trial design (80%, 57/71). Study designs included 36 two-group pretest-posttest control group comparisons (51%, 36/71), 16 posttest-only control group comparisons (23%, 16/71), 7 one-group pretest-posttest designs (10%, 7/71), 2 one-shot case study designs (3%, 2/71), and 2 static-group comparisons (3%, 2/71). A total of 17 trials included a qualitative component to their methodology (24%, 17/71). Complete trial data collection required 20 months on average to complete (mean 21, SD 12). For trials with a total duration of 2 years or more (31%, 22/71), the average time from recruitment to complete data collection (mean 35 months, SD 10) was 2 years longer than the average time required to collect primary data (mean 11, SD 8). Trials had a moderate sample size of 112 participants. Two trials were conducted online (3%, 2/71) and 7 trials collected data continuously (10%, 7/68). Onsite study implementation was heavily favored (97%, 69/71). Trials with four data collection points had a longer study duration than trials with two data collection points: F4,56=3.2, P=.021, η(2)=0.18. Single-blinded trials had a longer data collection period compared to open trials: F2,58=3.8, P=.028, η(2)=0.12. Academic sponsorship was the most common form of trial funding (73%, 52/71). Trials with academic sponsorship had a longer study duration compared to industry sponsorship: F2,61=3.7, P=.030, η(2)=0.11. Combined, data collection frequency, study masking, sample size, and study sponsorship accounted for 32.6% of the variance in study duration: F4,55=6.6, P<.01, adjusted r(2)=.33. Only 7 trials had been completed at the time this retrospective review was conducted (10%, 7/71). CONCLUSIONS: mHealth evaluation methodology has not deviated from common methods, despite the need for more relevant and timely evaluations. There is a need for clinical evaluation to keep pace with the level of innovation of mHealth if it is to have meaningful impact in informing payers, providers, policy makers, and patients.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/mhealth.5720

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is a small evidence base to provide guidance assessing chatbot effectiveness.

    “therefore, there is a small evidence base to provide guidance assessing their effectiveness [21, 22].”

    From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [21]

    The Discussion (Principal Findings; page unavailable) says only a fraction of publicly available mHealth apps have been evaluated, supporting a limited evidence base for apps generally. It does not establish that the evidence base for assessing chatbot effectiveness specifically is small. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Systematic reviews have shown that there are few published chatbot randomized control trials in the eHealth market.

    “Systematic reviews have shown that there are few published chatbot randomized control trials (RCT) in the eHealth market [21]”

    From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [21]

    The Methods section describes a review of mHealth app trials registered on ClinicalTrials.gov, not a review of published chatbot trials. The Discussion (Principal Findings) says RCTs were the predominant design among the registered mHealth trials; the Limitations section says the search was not exhaustive and did not cover all published work. A search for chatbot trials and published eHealth RCTs found no passage addressing how many chatbot RCTs had been published. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. [2]Goyal S Morita P Lewis GF Yu C Seto E Cafazzo JA The Systematic Design of a Behavioural Mobile Health Application for the Self-Management of Type 2 Diabetes Can J Diabetes 2016 02 40 1 95 104 10.1016/j.jcjd.2015.06.007 26455762 S1499-2671(15)00496-7 26455762 · DOI
  2. [3]Seto E Leonard KJ Cafazzo JA Barnsley J Masino C Ross HJ Mobile phone-based telemonitoring for heart failure management: a randomized controlled trial J Med Internet Res 2012 14 1 e31 10.2196/jmir.1909 22356799 v14i1e31 22356799 PMC3374537 · DOI · Paper page
  3. [4]Jibb LA Stevens BJ Nathan PC Seto E Cafazzo JA Stinson JN A smartphone-based pain management app for adolescents with cancer: establishing system requirements and a pain care algorithm based on literature review, interviews, and consensus JMIR Res Protoc 2014 3 1 e15 10.2196/resprot.3041 24646454 v3i1e15 24646454 PMC3978558 · DOI
  4. [5]Chan S Torous J Hinton L Yellowlees P Towards a Framework for Evaluating Mobile Mental Health Apps Telemed J E Health 2015 07 14 10.1089/tmj.2015.0002 26171663 26171663 · DOI · Paper page
  5. [6]Singh K Drouin K Newmark L Rozenblum R Lee J Landman A Pabo E Klinger EV Bates DW Developing a Framework for Evaluating the Patient Engagement, Quality, and Safety of Mobile Health Applications Issue Brief (Commonw Fund) 2016 02 5 1 11 26934758 26934758
  6. [8]Kumar S Nilsen WJ Abernethy A Atienza A Patrick K Pavel M Riley WT Shar A Spring B Spruijt-Metz D Hedeker D Honavar V Kravitz R Lefebvre RC Mohr DC Murphy SA Quinn C Shusterman V Swendeman D Mobile health technology evaluation: the mHealth evidence workshop Am J Prev Med 2013 08 45 2 228 236 10.1016/j.amepre.2013.03.017 23867031 S0749-3797(13)00277-8 23867031 PMC3803146 · DOI · Paper page
  7. [9]Piantadosi S Clinical Trials: A Methodologic Perspective 2013 New Jersey Wiley-Interscience
  8. [11]DeVito DA Song M Myers B Hawkins RP Aubrecht J Begey A Connolly M Li R Pilewski JM Bermudez CA Dew MA Clinical trials of health information technology interventions intended for patient use: unique issues and considerations Clin Trials 2013 10 6 896 906 10.1177/1740774513493149 23867222 1740774513493149 23867222 PMC3808467 · DOI
  9. [12]Mohr DC Schueller SM Riley WT Brown CH Cuijpers P Duan N Kwasny MJ Stiles-Shields C Cheung K Trials of Intervention Principles: Evaluation Methods for Evolving Behavioral Intervention Technologies J Med Internet Res 2015 17 7 e166 10.2196/jmir.4391 26155878 v17i7e166 26155878 PMC4526981 · DOI · Paper page
  10. [13]Riley WT Glasgow RE Etheredge L Abernethy AP Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise Clin Transl Med 2013 2 1 10 10.1186/2001-1326-2-10 23663660 2001-1326-2-10 23663660 PMC3658895 · DOI · Paper page
  11. [15]Moehr JR Evaluation: salvation or nemesis of medical informatics? Comput Biol Med 2002 05 32 3 113 125 11922929 S0010482502000094 11922929
  12. [16]Mohr DC Cheung K Schueller SM Hendricks BC Duan N Continuous evaluation of evolving behavioral intervention technologies Am J Prev Med 2013 10 45 4 517 523 10.1016/j.amepre.2013.06.006 24050429 S0749-3797(13)00387-5 24050429 PMC3828034 · DOI
  13. [17]Collins LM Murphy SA Nair VN Strecher VJ A strategy for optimizing and evaluating behavioral interventions Ann Behav Med 2005 08 30 1 65 73 10.1207/s15324796abm3001_8 16097907 16097907 · DOI · Paper page
  14. [19]Klasnja P Hekler EB Shiffman S Boruvka A Almirall D Tewari A Murphy SA Microrandomized trials: An experimental design for developing just-in-time adaptive interventions Health Psychol 2015 12 34 Suppl 1220 1228 10.1037/hea0000305 26651463 2015-56045-003 PMC4732571 26651463 · DOI · Paper page
  15. [20]Goyal S Morita PP Picton P Seto E Zbib A Cafazzo JA Uptake of a Consumer-Focused mHealth Application for the Assessment and Prevention of Heart Disease: The <30 Days Study JMIR Mhealth Uhealth 2016 4 1 e32 10.2196/mhealth.4730 27012937 v4i1e32 27012937 PMC4824871 · DOI
  16. [21]Morita PP Cafazzo JA Challenges and Paradoxes of Human Factors in Health Technology Design JMIR Hum Factors 2016 3 1 e11 10.2196/humanfactors.4653 27025862 v3i1e11 27025862 PMC4811664 · DOI
  17. [22]Uddin AA Morita PP Tallevi K Armour K Li J Nolan RP Cafazzo JA Development of a Wearable Cardiac Monitoring System for Behavioral Neurocardiac Training: A Usability Study JMIR Mhealth Uhealth 2016 4 2 e45 10.2196/mhealth.5288 27106171 v4i2e45 27106171 PMC4859872 · DOI
  18. [23]Hendela T Apple Press Info 2015 03 09 Apple Introduces ResearchKit, Giving Medical Researchers the Tools to Revolutionize Medical Studies http://www.apple.com/ca/pr/library/2015/03/09Apple-Introduces-ResearchKit-Giving-Medical-Researchers-the-Tools-to-Revolutionize-Medical-Studies.html
  19. [24]Jardine J Fisher J Carrick B Apple's ResearchKit: smart data collection for the smartphone era? J R Soc Med 2015 08 108 8 294 296 10.1177/0141076815600673 26268915 108/8/294 26268915 PMC4535444 · DOI
  20. [25]Mohammadi D ResearchKit: A clever tool to gather clinical data The Pharmaceutical Journal 2015 294 781 782 10.1211/PJ.2015.20068358 · DOI
  21. [26]Steinhubl SR Muse ED Topol EJ The emerging field of mobile health Sci Transl Med 2015 04 15 7 283 283rv3 10.1126/scitranslmed.aaa3487 25877894 7/283/283rv3 PMC4748838 25877894 · DOI · Paper page
  22. [27]Friend SH App-enabled trial participation: Tectonic shift or tepid rumble? Sci Transl Med 2015 07 22 7 297 297ed10 10.1126/scitranslmed.aab1206 26203077 7/297/297ed10 PMC5440077 26203077 · DOI
  23. [29]Pagliari C Design and evaluation in eHealth: challenges and implications for an interdisciplinary field J Med Internet Res 2007 9 2 e15 10.2196/jmir.9.2.e15 17537718 v9i2e15 17537718 PMC1913937 · DOI · Paper page
  24. [30]Jadad AR Moore RA Carroll D Jenkinson C Reynolds DJ Gavaghan DJ McQuay HJ Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996 02 17 1 1 12 8721797 0197-2456(95)00134-4 8721797
  25. [31]Sun X Briel M Busse JW You JJ Akl EA Mejza F Bala MM Bassler D Mertz D Diaz-Granados N Vandvik PO Malaga G Srinathan SK Dahm P Johnston BC Alonso-Coello P Hassouneh B Truong J Dattani ND Walter SD Heels-Ansdell D Bhatnagar N Altman DG Guyatt GH The influence of study characteristics on reporting of subgroup analyses in randomised controlled trials: systematic review BMJ 2011 342 d1569 21444636 21444636 PMC6173170
  26. [32]Powell AC Landman AB Bates DW In search of a few good apps JAMA 2014 05 14 311 18 1851 1852 10.1001/jama.2014.2564 24664278 1852662 24664278 · DOI
  27. [34]Marchal B Westhorp G Wong G Van Belle S Greenhalgh T Kegels G Pawson R Realist RCTs of complex interventions - an oxymoron Soc Sci Med 2013 10 94 124 128 10.1016/j.socscimed.2013.06.025 23850482 S0277-9536(13)00369-9 23850482 · DOI
  28. [37]Ben-Zeev D Schueller SM Begale M Duffecy J Kane JM Mohr DC Strategies for mHealth research: lessons from 3 mobile intervention studies Adm Policy Ment Health 2015 03 42 2 157 167 10.1007/s10488-014-0556-2 24824311 24824311 PMC4232479 · DOI
  29. [38]Coiera E Four rules for the reinvention of health care BMJ 2004 05 15 328 7449 1197 1199 10.1136/bmj.328.7449.1197 15142933 328/7449/1197 15142933 PMC411109 · DOI
  30. [41]Mirkovic J Kaufman DR Ruland CM Supporting cancer patients in illness management: usability evaluation of a mobile app JMIR Mhealth Uhealth 2014 2 3 e33 10.2196/mhealth.3359 25119490 v2i3e33 25119490 PMC4147703 · DOI
  31. [43]Choo S Kim JY Jung SY Kim S Kim JE Han JS Kim S Kim JH Kim J Kim Y Kim D Steinhubl S Development of a Weight Loss Mobile App Linked With an Accelerometer for Use in the Clinic: Usability, Acceptability, and Early Testing of its Impact on the Patient-Doctor Relationship JMIR Mhealth Uhealth 2016 4 1 e24 10.2196/mhealth.4546 27032541 v4i1e24 27032541 PMC4832121 · DOI
  32. [44]O'Malley G Dowdall G Burls A Perry IJ Curran N Exploring the usability of a mobile app for adolescent obesity management JMIR Mhealth Uhealth 2014 2 2 e29 10.2196/mhealth.3262 25098237 v2i2e29 25098237 PMC4114470 · DOI

Cited by

Sorted by citations

Authors
Pham, Q., Wiljer, D., & Cafazzo, J. A.
Affiliations
  • Centre for Addiction and Mental Health (CA)
  • University Health Network (CA)
ORCID
Venue
JMIR mHealth and uHealth
Publisher
JMIR Publications Inc.
Published
2016-09-09
Volume
4
Issue
3
Pages
e107
ISSN
2291-5222 (ISSN-L)
DOI
10.2196/mhealth.5720
PMC
PMC5035379
OpenAlex
W2510814766
Cited by
162
References
8
Citations by year
2026: 6 · 2025: 14 · 2024: 12 · 2023: 12 · 2022: 20 · 2021: 16 · 2020: 33 · 2019: 18 · 2018: 20 · 2017: 11
FWCI
30.43
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR mhealth and uhealth (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
mobile health apps, alternative trial designs, randomized controlled trials, research design, trial duration, data collection frequency, sample size, blinding, continuous data collection, research automation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review