Methodologic Issues in Health Informatics Trials: The Complexities of Complex Interventions
Open accessAbstract
OBJECTIVE All electronic health (e-health) interventions require validation as health information technologies, ideally in randomized controlled trial settings. However, as with other types of complex interventions involving various active components and multiple targets, health informatics trials often experience problems of design, methodology, or analysis that can influence the results and acceptance of the research. Our objective was to review selected key methodologic issues in conducting and reporting randomized controlled trials in health informatics, provide examples from a recent study, and present practical recommendations. DESIGN For illustration, we use the COMPETE III study, a large randomized controlled clinical trial investigating the impact of a shared decision-support system on the quality of vascular disease management in Ontario, Canada. RESULTS We describe a set of methodologic, logistic, and statistical issues that should be considered when planning and implementing trials of complex e-health interventions, and provide practical recommendations for health informatics trialists. CONCLUSIONS Our recommendations emphasize validity and pragmatic considerations and would be useful for health informaticians conducting or evaluating e-health studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Randomization was performed at the unit level and stratified by site before trial initiation.
“Randomization was performed at the unit level and stratified by site before trial initiation53.”
From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [53]
In “Blinding of subjects” (page unavailable), the paper explicitly says the COMPETE III trialists randomized individual patients, not units. Its “Choice of randomization” discussion treats unit-level allocation as an option, but the passages do not report stratification by site before the trial. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredEach study unit was randomly allocated to CONCERN EWS intervention or usual-care control using a computer-generated randomization scheme.
“Each study unit (ACU or ICU) was randomly allocated to one of two groups (CONCERN EWS intervention or usual-care control) using a computer-generated randomization scheme53.”
From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [53]
The “Blinding of subjects” section (page not supplied) says COMPETE III randomized individual patients, not ACU or ICU study units. The “COMPETE III Trial” section describes a vascular-risk decision-support intervention, not CONCERN EWS. Although the “Blinding” section mentions a computer-generated allocation schedule, it is for that different trial. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 36 · Cited by 27
Research works printed but not linked: 30
- Eisenstein EL, Ortiz M, Anstrom KJ, Crosslin DR, Lobach DF. Economic evaluation in medical information technology: why the numbers don't add up. AMIA Annu Symp Proc. 2006;914.
- Rosenbloom ST. Approaches to evaluating electronic prescrib- ing. J Am Med Inform Assoc. 2006 Jul-Aug;13(4):399 -401.
- Friedman CP, Wyatt JC. Evaluation Methods in Biomedical Informatics. 2nd ed. New York: Springer-Verlag; 2006.
- Garg AX, Adhikari NK, McDonald H, et al. Effects of comput- erized clinical decision support systems on practitioner perfor- mance and patient outcomes: a systematic review. JAMA 2005 Mar 9;293(10):1223-38.
- Guyatt GH, Haynes RB, Jaeschke R, et al. The philosophy of evidence-based medicine. In: Guyatt GH, Rennie D, editors. Users' guides to the medical literature: a manual for evidence- based clinical practice. Chicago, IL: AMA Press; 2001. p. 3-12.
- Campbell M, Fitzpatrick R, Haines A, et al. Framework for design and evaluation of complex interventions to improve health. BMJ 2000 Sep 16;321(7262):694 -6.
- Randolph AG, Haynes RB, Wyatt JC, Cook DJ, Guyatt GH. Therapy and validity: computer decision support systems. In: Guyatt GH, Rennie D, editors. Users' guides to the medical literature: a manual for evidence-based clinical practice. Chi- cago, IL: AMA Press; 2001. p. 291-308.
- Altman DG, Schulz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001 Apr 17;134(8):663-94.
- Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of random- ised controlled trials: the QUOROM statement. Lancet. 1999 Nov 27;354(9193):1896 -900.
- Torgerson DJ. Contamination in trials: is cluster randomisation the answer? BMJ 2001 Feb 10;322(7282):355-7.
- Chuang JH, Hripcsak G, Heitjan DF. Design and analysis of controlled trials in naturally clustered environments: implica- tions for medical informatics. J Am Med Inform Assoc. 2002 May-Jun;9(3):230 -8.
- Puffer S, Torgerson D, Watson J. Evidence for risk of bias in cluster randomized trials: review of recent trials published in three general medical journals. BMJ 2003 Oct 4;327(7418):785-9.
- Devereaux PJ, Bhandari M, Montori VM, Manns BJ, Ghall WA, Guyatt GH. Double blind, you have been voted off the island! Evid Based Ment Health 2002 May;5(2):36 -7.
- Schulz KF, Grimes DA. Blinding in randomised trials: hiding who got what. Lancet 2002 Feb 23;359(9307):696 -700.
- Altman DG, Schulz KF. Statistics notes: Concealing treatment allocation in randomized trials. BMJ 2001 Aug 25;323(7310): 446 -7.
- Tamblyn R, Huang A, Perreault R, et al. The medical office of the 21st century (MOXXI): effectiveness of computerized deci- sion-making support in reducing inappropriate prescribing in primary care. CMAJ 2003 Sep 16;169(6):549 -56.
- McGregor JC, Weekes E, Forrest GN, et al. Impact of a comput- erized clinical decision support system on reducing inappropri- ate antimicrobial use: a randomized controlled trial. J Am Med Inform Assoc. 2006 Jul-Aug;13(4):378 -84.
- Delpierre C, Cuzin L, Fillaux J, Alvarez M, Massip P, Lang T. A systematic review of computer-based patient record systems and quality of care: more randomized clinical trials or a broader approach? Int J Qual Health Care. 2004;16(5):407-16.
- Eddy DM. Performance measurement: problems and solutions. Health Aff (Millwood). 1998 Jul-Aug;17(4):7-25.
- Freemantle N, Calvert M, Wood J, Eastaugh J, Griffin C. Composite outcomes in randomized trials: greater precision but with greater uncertainty? JAMA 2003 May 21;289(19):2554 -9.
- Montori VM, Permanyer-Miralda G, Ferreira-Gonzalez I, et al. Validity of composite end points in clinical trials. BMJ 2005 Mar 12;330(7491):594 -6.
- Ferreira-González I, Busse JW, Heels-Ansdell D, et al. Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials. BMJ 2007 Apr 14;334(7597):786.
- Majumdar SR, Johnson JA, Bowker SL, et al. A Canadian consensus for the standardized evaluation of quality improve- ment interventions in Type-2 diabetes: Development of a qual- ity indicator set. Can J Diabetes. 2005;29:220 -9.
- Yusuf S, Hawken S, Ounpuu S, et al. Effect of potentially modifiable risk factors associated with myocardial infarction in 52 countries (the INTERHEART study): case-control study. Lancet. 2004 Sep 11-17;364(9438):937-52.
- Beaton DE, Boers M, Wells GA. Many faces of the minimal clinically important difference (MCID): a literature review and directions for future research. Curr Opin Rheumatol 2002 Mar; 14(2):109 -14.
- Dansky KH, Thompson D, Sanner T. A framework for evaluat- ing eHealth research. Eval Program Plann. 2006 Nov;29(4): 397-404.
- Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005 Mar 9;293(10):1197-203.
- Ioannidis JPA, Evans SJW, Gotzsche PC, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141:781-8.
- Berner ES, Kasiraman RK, Yu F, Ray MN, Houston TK. Data quality in the outpatient setting: impact on clinical decision support systems. AMIA Annu Symp Proc. 2005;41-5.
- Shrive FM, Stuart H, Quan H, Ghali WA. Dealing with missing data in a multi-question depression scale: a comparison of imputation methods. BMC Med Res Methodol. 2006 Dec 13;6:57.
- Authors
- Shcherbatykh, I., Holbrook, A., Thabane, L., Dolovich, L., & COMPETE III investigators
- Affiliations
- St. Joseph’s Healthcare Hamilton (CA)
- Corresponding author
- Shcherbatykh, I.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-09-01
- Volume
- 15
- Issue
- 5
- Pages
- 575-580
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC2528041
- OpenAlex
- W2101984653
- Cited by
- 27
- References
- 36
- Citations by year
- 2025: 1 · 2024: 1 · 2022: 1 · 2021: 1 · 2020: 2 · 2018: 1 · 2017: 1 · 2016: 2 · 2014: 2 · 2013: 4 · 2012: 1
- FWCI
- 8.08
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ontario Ministry of Health and Long-Term Care
- Canadian Institutes of Health Research
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- complex interventions, digital health interventions, randomized controlled trials, trial design, vascular disease management, trial validity, reporting recommendations
- MeSH terms
- Cost-Benefit Analysis, Data Interpretation, Statistical, Humans, Ontario, Research Design, Vascular Diseases, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care, Decision Support Systems, Clinical, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
