The DOI matches its Crossref record; 40 of the 41 references it cites were found in a registry or an index.
Influence of Reported Study Design Characteristics on Intervention Effect Estimates From Randomized, Controlled Trials
Abstract
Published evidence suggests that aspects of trial design lead to biased intervention effect estimates, but findings from different studies are inconsistent. This study combined data from 7 meta-epidemiologic studies and removed overlaps to derive a final data set of 234 unique meta-analyses containing 1973 trials. Outcome measures were classified as "mortality," "other objective," "or subjective," and Bayesian hierarchical models were used to estimate associations of trial characteristics with average bias and between-trial heterogeneity. Intervention effect estimates seemed to be exaggerated in trials with inadequate or unclear (vs. adequate) random-sequence generation (ratio of odds ratios, 0.89 [95% credible interval {CrI}, 0.82 to 0.96]) and with inadequate or unclear (vs. adequate) allocation concealment (ratio of odds ratios, 0.93 [CrI, 0.87 to 0.99]). Lack of or unclear double-blinding (vs. double-blinding) was associated with an average of 13% exaggeration of intervention effects (ratio of odds ratios, 0.87 [CrI, 0.79 to 0.96]), and between-trial heterogeneity was increased for such studies (SD increase in heterogeneity, 0.14 [CrI, 0.02 to 0.30]). For each characteristic, average bias and increases in between-trial heterogeneity were driven primarily by trials with subjective outcomes, with little evidence of bias in trials with objective and mortality outcomes. This study is limited by incomplete trial reporting, and findings may be confounded by other study design characteristics. Bias associated with study design characteristics may lead to exaggeration of intervention effect estimates and increases in between-trial heterogeneity in trials reporting subjectively assessed outcomes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The use of VAS for pain scores, while validated, is subjective and may increase heterogeneity between studies [25].”
From Effect of Coenzyme Q10 on statin-associated myalgia and adherence to statin therapy: A systematic review and meta-analysis (2020) · cites this paper as [25]
In the citing paper
“incomplete reporting may be associated with biased estimates of intervention effects [4]”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [4]
In the citing paper
“However, most of the underlying studies are open-label trials with subjective health outcomes (eg, self-reported diarrhoea or respiratory symptoms)”
In the citing paper
“so there is a risk of overestimating their effectiveness.52”
In the citing paper
“Poor randomisation methods with no concealment can lead to exaggerated treatment effects,79 80 which can then lead to further problems with overexaggerated systematic reviews.81”
From Novel ACT-based eHealth psychoeducational intervention for students with mental distress: A study protocol for a mixed-methodology pilot trial (2019) · cites this paper as [79]
In the citing paper
“There is empirical evidence showing that trials without appropriate randomization, concealment of allocation, and blinding tend to report an inaccurate treatment effect compared with trials that include these features.109”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [109]
In the citing paper
“In general, these non-random allocation methods introduce selection bias and biased estimates of an intervention’s effect size.323 324 325 326 327 328”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [325]
In the citing paper
“Similar to failure of random sequence generation, inadequate or unclear allocation concealment in trial publications has been associated with inflated effect size estimates.325 327 328 336 349 350”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [325]
In the citing paper
“and that incomplete reporting may be associated with biased estimates of intervention effects.4”
From CONSORT 2025 statement: updated guideline for reporting randomised trials (2025) · cites this paper as [4]
In the citing paper
“so there is a great risk that the results of the review overestimate the beneficial effects and underestimate the harmful effects of SSRIs.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [32]
In the citing paper
“Trials at high or unclear risk of bias tend to overestimate benefits and underestimate harms.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [32]
In the citing paper
“so there is a great risk that the results of the review overestimate the beneficial effects and underestimate the harmful effects of antidepressants.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [32]
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationCONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
Self-citationCONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · cited 1×
“Future trials should report the blinding status of these groups, as specified in the CONSORT (Consolidated Standards of Reporting Trials) Statement (35, 36).”
MethodSelf-citationSchulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Self-citationWinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. J. (2000) · Statistics and Computing · doi:10.1023/a:1008929526011 · Cited by 5,397
Gelman, A. (2006) · Bayesian Analysis · doi:10.1214/06-ba117a · Cited by 4,050
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Self-citationWood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Self-citationGluud, L. L., Villumsen, J., & Gluud, C. N. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,266
Self-citationPrior distributions for variance parameters in hierarchical modelsPreprint
Gelman, A. (2004) · RePEc: Research Papers in Economics · Cited by 1,558
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546
Self-citationHróbjartsson, A., & Gøtzsche, P. C. (2001) · New England Journal of Medicine · doi:10.1056/nejm200105243442106 · Cited by 1,504
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · cited 6×
“The study by Egger and associates (2) was based on quality assessments by authors of the included Cochrane reviews, which were generally performed in duplicate by 2 observers.”
BackgroundResultInfluentialSelf-citationKirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
Self-citationBias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636
Self-citationLambert, P. C., Sutton, A. J., Burton, P. R., Abrams, K. R., & Jones, D. R. (2005) · Statistics in Medicine · doi:10.1002/sim.2112 · Cited by 511
Hopewell, S., Dutton, S., Yu, L.-M., Chan, A.-W., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c723 · Cited by 504
Self-citationBalk, E. M. (2002) · JAMA · doi:10.1001/jama.287.22.2973 · Cited by 478
Self-citationImpact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal, J., Hrobjartsson, A., Jorgensen, K., Hilden, J., Altman, D., & Gotzsche, P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym087 · Cited by 472
Self-citationKrogsbøll, L. T., Hróbjartsson, A., & Gøtzsche, P. C. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-1 · Cited by 435
Quality of Reporting of Randomized Trials as a Measure of Methodologic Quality
Huwiler-Müntener, K. (2002) · JAMA · doi:10.1001/jama.287.21.2801 · Cited by 397
Soares, H. P., Daniels, S., Kumar, A., Clarke, M. J., Scott, C., Swann, S., & Djulbegović, B. (2004) · BMJ · doi:10.1136/bmj.328.7430.22 · Cited by 310
Dechartres, A., Boutron, I., Trinquart, L., Charles, P., & Ravaud, P. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-1-201107050-00006 · Cited by 307
Devereaux, P. J. (2001) · JAMA · doi:10.1001/jama.285.15.2000 · Cited by 258
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Burgi, E., Scherer, M., Altman, D. G., & Juni, P. (2009) · BMJ · doi:10.1136/bmj.b3244 · Cited by 248 · cited 1×
“There is recent empirical evidence of bias in the results of randomized, controlled trials due to attrition (26, 27) and selective outcome reporting (28, 29).”
BackgroundSelf-citationMoja, L., Telaro, E., D'Amico, R., Moschetti, I., Coe, L., & Liberati, A. (2005) · British medical journal · doi:10.1136/bmj.38414.515938.8f · Cited by 226
Pildal, J., Chan, A.-W., Hróbjartsson, A., Forfang, E., Altman, D. G., & Gøtzsche, P. C. (2005) · BMJ · doi:10.1136/bmj.38414.422650.8f · Cited by 209
Self-citationIn the dark The reporting of blinding status in randomized controlled trials
Montori, V. M., Bhandari, M., Devereaux, P. J., Manns, B., Ghali, W. A., & Guyatt, G. H. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00446-8 · Cited by 175
Nüesch, E., Reichenbach, S., Trelle, S., Rutjes, A. W., Liewald, K., Sterchi, R., Altman, D. G., & Jüni, P. (2009) · Arthritis Care & Research · doi:10.1002/art.24894 · Cited by 150
Self-citationModels for Potentially Biased Evidence in Meta-Analysis using Empirically Based Priors
Welton, N. J., Ades, A. E., Carlin, J. B., Altman, D. G., & Sterne, J. A. C. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2008.00548.x · Cited by 147
Self-citationSiontis, G. C. M. (2011) · Archives of Internal Medicine · doi:10.1001/archinternmed.2011.334 · Cited by 129
Discrepancy between published report and actual conduct of randomized clinical trials
Hill, C. L., LaValley, M. P., & Felson, D. T. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00440-7 · Cited by 111
Bias due to withdrawal in long‐term randomised trials in COPD: Evidence from the TORCH study
Vestbo, J., Anderson, J. A., Calverley, P. M. A., Celli, B. R., Ferguson, G. T., Jenkins, C. R., Yates, J., & Jones, P. (2010) · The Clinical Respiratory Journal · doi:10.1111/j.1752-699x.2010.00198.x · Cited by 87
Siersma, V., Als-Nielsen, B. E. T., Chen, W., Hilden, J., Gluud, L. L., & Gluud, C. N. (2006) · Statistics in Medicine · doi:10.1002/sim.2752 · Cited by 83
Self-citationMetabias: A Challenge for Comparative Effectiveness Research
Goodman, S. N., & Dickersin, K. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-1-201107050-00010 · Cited by 70
Allocation concealment and blinding: when ignorance is bliss
Forder, P., Gebski, V., & Keech, A. (2005) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2005.tb06584.x · Cited by 67
Hróbjartsson, A., Pildal, J., Chan, A.-W., Haahr, M. T., Altman, D. G., & Gøtzsche, P. C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.04.003 · Cited by 56
Self-citationComparison of Large Versus Smaller Randomized Trials for Mental Health-Related Interventions
Contopoulos‐Ioannidis, D. G., Gilbody, S. M., Trikalinos, T. A., Churchill, R. C., Wahlbeck, K., Ioannidis, J. P. A., & The EU-PSI Project (2005) · American Journal of Psychiatry · doi:10.1176/appi.ajp.162.3.578 · Cited by 38
Self-citationDevelopment of a combined database for meta‐epidemiological research
Savović, J., Harris, R., Wood, L., Beynon, R., Altman, D. G., Als-Nielsen, B. E. T., Balk, E. M., Deeks, J. J., Gluud, L. L., Gluud, C. N., et al. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.18 · Cited by 25
Self-citation
Cited by
Sorted by citations
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Self-citationSPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251
Self-citationIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationPrüss-Ustün, A., Wolf, J., Bartram, J., Clasen, T., Cumming, O., Freeman, M. C., Gordon, B., Hunter, P. R., Medlicott, K., & Johnston, R. (2019) · International Journal of Hygiene and Environmental Health · doi:10.1016/j.ijheh.2019.05.004 · Cited by 1,308
Prüss‐Ustün, A., Bartram, J., Clasen, T., Colford, J. M., Cumming, O., Curtis, V., Bonjour, S., Dangour, A. D., De France, J., Fewtrell, L., et al. (2014) · Tropical Medicine & International Health · doi:10.1111/tmi.12329 · Cited by 1,198
Metadata
- Authors
- Savović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., Ioannidis, J. P. A., Schulz, K. F., Beynon, R., Welton, N. J., Wood, L., Moher, D., Deeks, J. J., & Sterne, J. A. C.
- Affiliations
- University of Bern (CH)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2012-09-18
- Volume
- 157
- Issue
- 6
- Pages
- 429-438
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W1964502734
- Semantic Scholar
- 5667b851bbf788508a249be0a5243029fcf25bdd
- Cited by
- 1,049
- References
- 41
- Citations by year
- 2026: 21 · 2025: 44 · 2024: 51 · 2023: 44 · 2022: 50 · 2021: 68 · 2020: 67 · 2019: 84 · 2018: 99 · 2017: 100 · 2016: 93 · 2015: 112 · 2014: 113 · 2013: 87 · 2012: 9
- FWCI
- 115.35
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- BORIS (University Library Bern) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Funders
- National Institute for Health and Care Research (NF-SI-0611-10168)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- trial design bias, random sequence generation, allocation concealment, double blinding, between-trial heterogeneity, meta-epidemiology, subjective outcomes, mortality, trial reporting quality
- MeSH terms
- Bayes Theorem, Double-Blind Method, Humans, Research Design, Meta-Analysis as Topic, Bias, Odds Ratio, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
