The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Methodological Challenges in Online Trials
Open accessAbstract
Health care and health care services are increasingly being delivered over the Internet. There is a strong argument that interventions delivered online should also be evaluated online to maximize the trial's external validity. Conducting a trial online can help reduce research costs and improve some aspects of internal validity. To date, there are relatively few trials of health interventions that have been conducted entirely online. In this paper we describe the major methodological issues that arise in trials (recruitment, randomization, fidelity of the intervention, retention, and data quality), consider how the online context affects these issues, and use our experience of one online trial evaluating an intervention to help hazardous drinkers drink less (DownYourDrink) to illustrate potential solutions. Further work is needed to develop online trial methodology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Program engagement and retention are expected challenges with transition-age youth in general (Kim et al., 2012) and with remote delivery as well (Hanna et al., 2014; Murray et al., 2009; Watson et al., 2018).”
From Intervention development to improve foster youth mental health by targeting coping self-efficacy and help-seeking (2023) · cites this paper as Murray et al., 2009
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Clinical Trials: A Practical Approach.
Hill, I. D., & Pocock, S. (1984) · Biometrics · doi:10.2307/2531186 · Cited by 2,918
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524
Developing and Evaluating Complex Interventions: New Guidance
Craig, P., Dieppe, P. A., MacIntyre, S., Michie, S., Nazareth, I., & Petticrew, M. P. (2006) · UCL Discovery (University College London) · Cited by 1,878
Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: A meta-analysis
SPEK, V., CUIJPERS, P., NYKLÍČEK, I., RIPER, H., KEYZER, J., & POP, V. (2007) · Psychological Medicine · doi:10.1017/s0033291706008944 · Cited by 1,586
Designing and evaluating complex interventions to improve health care
Campbell, N. C., Murray, E., Darbyshire, J., Emery, J., Farmer, A., Griffiths, F., Guthrie, B., Lester, H., Wilson, P., & Kinmonth, A. L. (2007) · BMJ · doi:10.1136/bmj.39108.379965.be · Cited by 1,331
Self-citationEysenbach, G., Powell, J., Englesakis, M., Rizo, C., & Stern, A. (2004) · BMJ · doi:10.1136/bmj.328.7449.1166 · Cited by 1,268
Clinical trials: A practical approach
Daly, M. J. (1984) · Trends in Pharmacological Sciences · doi:10.1016/0165-6147(84)90515-7 · Cited by 1,057
Wantland, D. J., Portillo, C. J., Holzemer, W. L., Slaughter, R., & McGhee, E. M. (2004) · Journal of Medical Internet Research · doi:10.2196/jmir.6.4.e40 · Cited by 966 · cited 1×
“These range from simple health information sites to complex self-management programs incorporating interactive components such as decision support, social support, behavior change support [6-8], and computerized cognitive behavioral therapy [9,10].”
BackgroundUsing the Internet for Surveys and Health Research
Eysenbach, G., & Wyatt, J. (2002) · Journal of Medical Internet Research · doi:10.2196/jmir.4.2.e13 · Cited by 790
Interactive Health Communication Applications for people with chronic disease
Murray, E., Burns, J., See Tai, S., Lai, R., & Nazareth, I. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004274.pub4 · Cited by 722 · cited 1×
“These range from simple health information sites to complex self-management programs incorporating interactive components such as decision support, social support, behavior change support [6-8], and computerized cognitive behavioral therapy [9,10].”
BackgroundSelf-citationWood, A., White, I. R., & Thompson, S. G. (2004) · Clinical Trials · doi:10.1191/1740774504cn032oa · Cited by 559
Self-citationeHealth Trends in Europe 2005-2007: A Population-Based Survey
Kummervold, P. E., Chronaki, C. E., Lausen, B., Prokosch, H.-U., Rasmussen, J., Santana, S., Staniszewski, A., & Wangberg, S. C. (2008) · Journal of Medical Internet Research · doi:10.2196/jmir.1023 · Cited by 449
Campbell, M., Snowdon, C., Francis, D., Elbourne, D., McDonald, A., Knight, R., Entwistle, V, Garcia, J., Roberts, I., Grant, A., et al. (2007) · Health Technology Assessment · doi:10.3310/hta11480 · Cited by 433
Internet interventions: In review, in use, and into the future.
Ritterband, L. M., Gonder-Frederick, L. A., Cox, D. J., Clifton, A. D., West, R. W., & Borowitz, S. M. (2003) · Professional Psychology: Research and Practice · doi:10.1037/0735-7028.34.5.527 · Cited by 352
Inclusion of patients in clinical trial analysis: the intention‐to‐treat principle
Heritier, S. R., Gebski, V., & KEECH, A. C. (2003) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.2003.tb05627.x · Cited by 271
Web‐based self‐help for problem drinkers: a pragmatic randomized trial
Riper, H., Kramer, J., Smit, F., Conijn, B., Schippers, G., & Cuijpers, P. (2008) · Addiction · doi:10.1111/j.1360-0443.2007.02063.x · Cited by 267
Review of the effectiveness of treatment for alcohol problems.
Raistrick, D., Heather, N., & Godfrey, C. (2006) · Cited by 261
Self-citationGlasgow, R. E., Nelson, C. C., Kearney, K. A., Reid, R., Ritzwoller, D. P., Strecher, V. J., Couper, M. P., Green, B., & Wildenhaus, K. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.2.e11 · Cited by 216
Altman, D. G., & Bland, M. (2007) · BMJ · doi:10.1136/bmj.38977.682025.2c · Cited by 196
Verheijden, M. W., Jans, M. P., Hildebrandt, V. H., & Hopman-Rock, M. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.1.e1 · Cited by 195
Strategies to improve recruitment to research studies
Mapstone, J., Elbourne, D. R., & Roberts, I. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000013.pub3 · Cited by 194
Gerber, W. H. D. A. E. J. H. A. M. M. (2009) · Cited by 180
Challenges of Internet Recruitment: A Case Study with Disappointing Results
Koo, M., & Skinner, H. A. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e6 · Cited by 177
Hewitt, C., Hahn, S., Torgerson, D. J., Watson, J., & Bland, J. M. (2005) · BMJ · doi:10.1136/bmj.38413.576713.ae · Cited by 176
Murray, E. J., Burns, J., Tai, S. S., Lai, R., & Nazareth, I. (2005) · UCL Discovery (University College London) · Cited by 171
Self-citationThe Story of Subject Naught: A Cautionary but Optimistic Tale of Internet Survey Research
Konstan, J. A., Rosser, B. R. S., Ross, M. W., Stanton, J. M., & Edwards, W. M. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2005.tb00248.x · Cited by 170
Shepherd, J., Jones, J. E., Hartwell, D., Davidson, P., Price, A., & Waugh, N. (2007) · Health Technology Assessment · doi:10.3310/hta11110 · Cited by 164
Bull, S. S., Lloyd, L. V., RIETMEIJER, C. A. M., & McFarlane, M. (2004) · AIDS Care · doi:10.1080/09540120412331292507 · Cited by 147
Mills, N., Donovan, J. L., Smith, M., Jacoby, A., Neal, D. E., & Hamdy, F. C. (2003) · Controlled Clinical Trials · doi:10.1016/s0197-2456(03)00020-5 · Cited by 142
Linke, S., Murray, E., Butler, C., & Wallace, P. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.2.e10 · Cited by 127
Self-citationKaltenthaler, E., Shackley, P., Stevens, K., Beverley, C., Parry, G., & Chilcott, J. (2002) · Health Technology Assessment · doi:10.3310/hta6220 · Cited by 109 · cited 1×
“These range from simple health information sites to complex self-management programs incorporating interactive components such as decision support, social support, behavior change support [6-8], and computerized cognitive behavioral therapy [9,10].”
BackgroundIssues in evaluating health websites in an Internet-based randomized controlled trial
Eysenbach, G. (2002) · Journal of Medical Internet Research · doi:10.2196/jmir.4.3.e17 · Cited by 99 · cited 1×
“However, there are grounds for exploring methods of evaluation that rely totally on the Internet [13].”
BackgroundUses and limitations of randomization-based efficacy estimators
White, I. R. (2005) · Statistical Methods in Medical Research · doi:10.1191/0962280205sm406oa · Cited by 98
Self-citationBuchanan, T., Ali, T., Heffernan, T. M., Ling, J., Parrott, A. C., Rodgers, J., & Scholey, A. B. (2005) · Behavior Research Methods · doi:10.3758/bf03206409 · Cited by 98
Coleman, K. (2002) · Family Practice · doi:10.1093/fampra/19.2.183 · Cited by 97
Down your drink: a web-based intervention for people with excessive alcohol consumption
Linke, S. (2004) · Alcohol and Alcoholism · doi:10.1093/alcalc/agh004 · Cited by 91
The Internet and Clinical Trials: Background, Online Resources, Examples and Issues
Paul, J., Seib, R. K., & Prescott, T. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e5 · Cited by 90
Conducting clinical trials over the internet: feasibility study
McAlindon, T. E., Formica, M., Kabbara, K., LaValley, M. P., & Lehmer, M. (2003) · BMJ · doi:10.1136/bmj.327.7413.484 · Cited by 88
Development of a Psychologically Enhanced Interactive Online Intervention for Hazardous Drinking
Linke, S., McCambridge, J., Khadjesari, Z., Wallace, P., & Murray, E. (2008) · Alcohol and Alcoholism · doi:10.1093/alcalc/agn066 · Cited by 71
Self-citationOxford Internet Survey 2005 Report: The Internet in BritainPreprint
Dutton, W. H., di Gennaro, C., & Hargrave, A. M. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1327035 · Cited by 66
Couper, M. P., Peytchev, A., Strecher, V. J., Rothert, K., & Anderson, J. P. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.2.e16 · Cited by 63
Thomas, B. A., & McCambridge, J. (2008) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2008.02.010 · Cited by 53
Self-citationMurray, E., McCambridge, J., Khadjesari, Z., White, I. R., Thompson, S. G., Godfrey, C., Linke, S., & Wallace, P. (2007) · BMC Public Health · doi:10.1186/1471-2458-7-306 · Cited by 38
Self-citationKhadjesari, Z., Murray, E., Kalaitzaki, E., White, I. R., McCambridge, J., Godfrey, C., & Wallace, P. (2008) · Addictive Behaviors · doi:10.1016/j.addbeh.2008.11.010 · Cited by 35
Self-citationFormica, M., Kabbara, K., Clark, R. A., & McAlindon, T. E. (2004) · Journal of Medical Internet Research · doi:10.2196/jmir.6.1.e6 · Cited by 23
eHealth recruitment challenges
Thompson, D. I., Canada, A., Bhatt, R., Davis, J., Plesko, L., Baranowski, T., Cullen, K. W., & Zakeri, I. (2006) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2006.08.004 · Cited by 20
Research works printed but not linked: 6
- Fox, S, 2002, The online health care revolution: how the Web helps Americans take better care of themselves
- Murray, E, 2005, Cochrane Database Syst Rev, CD004274
- MRC Health Services and Public Health Research Board, 2000, A Framework for Development and Evaluation of RCTs for Complex Interventions to Improve Health, 1
- Mapstone, J, 2007, Cochrane Database Syst Rev, MR000013
- 10.1001/jama.273.5.408 · DOI
- Konstan, J, 2005, J Comput Mediated Commun, 10
Cited by
Sorted by citations
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517
Systematic Literature Review on the Spread of Health-related Misinformation on Social Media
Wang, Y., McKee, M., Torbica, A., & Stuckler, D. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.112552 · Cited by 1,806
Evaluating Digital Health Interventions
Murray, E., Hekler, E. B., Andersson, G., Collins, L. M., Doherty, A., Hollis, C., Rivera, D. E., West, R., & Wyatt, J. C. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.008 · Cited by 847
Self-citationInternet-based interventions for smoking cessation
Taylor, G. M. J., Dalili, M. N., Semwal, M., Čivljak, M., Sheikh, A., & Car, J. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007078.pub5 · Cited by 676
Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks
Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.5.p472 · Cited by 329
Metadata
- Authors
- Murray, E., Khadjesari, Z., White, I. R., Kalaitzaki, E., Godfrey, C., McCambridge, J., Thompson, S. G., & Wallace, P.
- Affiliations
- University College London (GB)
- MRC Biostatistics Unit (GB)
- Medical Research Council (GB)
- University of York (GB)
- London School of Hygiene & Tropical Medicine (GB)
- Corresponding author
- Murray, Elizabeth
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2009-04-03
- Volume
- 11
- Issue
- 1
- Pages
- e9
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC2762798
- OpenAlex
- W2165002752
- Cited by
- 195
- References
- 48
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 9 · 2023: 13 · 2022: 8 · 2021: 13 · 2020: 11 · 2019: 8 · 2018: 12 · 2017: 17 · 2016: 16 · 2015: 16 · 2014: 6 · 2013: 19 · 2012: 11
- FWCI
- 15.30
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- other-oa
- Locations
- Journal of Medical Internet Research (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Portal (King's College London) (publishedVersion, other-oa)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Cancer Research UK
- National Institute for Health and Care Research
- British Heart Foundation (RG/08/014/24067)
- Diabetes UK
- Alcohol Change UK
- Medical Research Council (U.1052.00.006, U.1052.00.001)
- Economic and Social Research Council
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- online trials, online intervention, clinical trial design, participant recruitment, randomization, intervention fidelity, participant retention, data quality, generalizability, internal validity, hazardous drinking
- MeSH terms
- Clinical Trials as Topic, Delivery of Health Care, Humans, Online Systems, Reproducibility of Results, Randomized Controlled Trials as Topic, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
