The DOI matches its Crossref record; 40 of the 41 references it cites were found in a registry or an index.
Influence of Reported Study Design Characteristics on Intervention Effect Estimates From Randomized, Controlled Trials
Abstract
Published evidence suggests that aspects of trial design lead to biased intervention effect estimates, but findings from different studies are inconsistent. This study combined data from 7 meta-epidemiologic studies and removed overlaps to derive a final data set of 234 unique meta-analyses containing 1973 trials. Outcome measures were classified as "mortality," "other objective," "or subjective," and Bayesian hierarchical models were used to estimate associations of trial characteristics with average bias and between-trial heterogeneity. Intervention effect estimates seemed to be exaggerated in trials with inadequate or unclear (vs. adequate) random-sequence generation (ratio of odds ratios, 0.89 [95% credible interval {CrI}, 0.82 to 0.96]) and with inadequate or unclear (vs. adequate) allocation concealment (ratio of odds ratios, 0.93 [CrI, 0.87 to 0.99]). Lack of or unclear double-blinding (vs. double-blinding) was associated with an average of 13% exaggeration of intervention effects (ratio of odds ratios, 0.87 [CrI, 0.79 to 0.96]), and between-trial heterogeneity was increased for such studies (SD increase in heterogeneity, 0.14 [CrI, 0.02 to 0.30]). For each characteristic, average bias and increases in between-trial heterogeneity were driven primarily by trials with subjective outcomes, with little evidence of bias in trials with objective and mortality outcomes. This study is limited by incomplete trial reporting, and findings may be confounded by other study design characteristics. Bias associated with study design characteristics may lead to exaggeration of intervention effect estimates and increases in between-trial heterogeneity in trials reporting subjectively assessed outcomes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The use of VAS for pain scores, while validated, is subjective and may increase heterogeneity between studies [25].”
From Effect of Coenzyme Q10 on statin-associated myalgia and adherence to statin therapy: A systematic review and meta-analysis (2020) · cites this paper as [25]
In the citing paper
“incomplete reporting may be associated with biased estimates of intervention effects [4]”
From CONSORT 2025 statement: Updated guideline for reporting randomised trials (2025) · cites this paper as [4]
In the citing paper
“However, most of the underlying studies are open-label trials with subjective health outcomes (eg, self-reported diarrhoea or respiratory symptoms)”
In the citing paper
“so there is a risk of overestimating their effectiveness.52”
In the citing paper
“Poor randomisation methods with no concealment can lead to exaggerated treatment effects,79 80 which can then lead to further problems with overexaggerated systematic reviews.81”
From Novel ACT-based eHealth psychoeducational intervention for students with mental distress: A study protocol for a mixed-methodology pilot trial (2019) · cites this paper as [79]
In the citing paper
“There is empirical evidence showing that trials without appropriate randomization, concealment of allocation, and blinding tend to report an inaccurate treatment effect compared with trials that include these features.109”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [109]
In the citing paper
“In general, these non-random allocation methods introduce selection bias and biased estimates of an intervention’s effect size.323 324 325 326 327 328”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [325]
In the citing paper
“Similar to failure of random sequence generation, inadequate or unclear allocation concealment in trial publications has been associated with inflated effect size estimates.325 327 328 336 349 350”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [325]
In the citing paper
“and that incomplete reporting may be associated with biased estimates of intervention effects.4”
From CONSORT 2025 statement: updated guideline for reporting randomised trials (2025) · cites this paper as [4]
In the citing paper
“so there is a great risk that the results of the review overestimate the beneficial effects and underestimate the harmful effects of SSRIs.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [32]
In the citing paper
“Trials at high or unclear risk of bias tend to overestimate benefits and underestimate harms.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [32]
In the citing paper
“so there is a great risk that the results of the review overestimate the beneficial effects and underestimate the harmful effects of antidepressants.”
From Should antidepressants be used for major depressive disorder? (2020) · cites this paper as [32]
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationCONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
Self-citationCONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · cited 1×
“Future trials should report the blinding status of these groups, as specified in the CONSORT (Consolidated Standards of Reporting Trials) Statement (35, 36).”
MethodSelf-citationSchulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Self-citationWinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. J. (2000) · Statistics and Computing · doi:10.1023/a:1008929526011 · Cited by 5,397
Gelman, A. (2006) · Bayesian Analysis · doi:10.1214/06-ba117a · Cited by 4,050
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Self-citationWood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Self-citationGluud, L. L., Villumsen, J., & Gluud, C. N. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,266
Self-citation
Flagged first, then by citations
1 of the 41 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Self-citationSPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251
Self-citationIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationPrüss-Ustün, A., Wolf, J., Bartram, J., Clasen, T., Cumming, O., Freeman, M. C., Gordon, B., Hunter, P. R., Medlicott, K., & Johnston, R. (2019) · International Journal of Hygiene and Environmental Health · doi:10.1016/j.ijheh.2019.05.004 · Cited by 1,308
Prüss‐Ustün, A., Bartram, J., Clasen, T., Colford, J. M., Cumming, O., Curtis, V., Bonjour, S., Dangour, A. D., De France, J., Fewtrell, L., et al. (2014) · Tropical Medicine & International Health · doi:10.1111/tmi.12329 · Cited by 1,198
Metadata
- Authors
- Savović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., Ioannidis, J. P. A., Schulz, K. F., Beynon, R., Welton, N. J., Wood, L., Moher, D., Deeks, J. J., & Sterne, J. A. C.
- Affiliations
- University of Bern (CH)
- ORCID
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2012-09-18
- Volume
- 157
- Issue
- 6
- Pages
- 429-438
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- OpenAlex
- W1964502734
- Semantic Scholar
- 5667b851bbf788508a249be0a5243029fcf25bdd
- Cited by
- 1,049
- References
- 41
- Citations by year
- 2026: 21 · 2025: 44 · 2024: 51 · 2023: 44 · 2022: 50 · 2021: 68 · 2020: 67 · 2019: 84 · 2018: 99 · 2017: 100 · 2016: 93 · 2015: 112 · 2014: 113 · 2013: 87 · 2012: 9
- FWCI
- 115.35
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- BORIS (University Library Bern) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Funders
- National Institute for Health and Care Research (NF-SI-0611-10168)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- trial design bias, random sequence generation, allocation concealment, double blinding, between-trial heterogeneity, meta-epidemiology, subjective outcomes, mortality, trial reporting quality
- MeSH terms
- Bayes Theorem, Double-Blind Method, Humans, Research Design, Meta-Analysis as Topic, Bias, Odds Ratio, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
