Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies
Open accessThe DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To synthesise evidence on the average bias and heterogeneity associated with reported methodological features of randomized trials. DESIGN: Systematic review of meta-epidemiological studies. METHODS: We retrieved eligible studies included in a recent AHRQ-EPC review on this topic (latest search September 2012), and searched Ovid MEDLINE and Ovid EMBASE for studies indexed from Jan 2012-May 2015. Data were extracted by one author and verified by another. We combined estimates of average bias (e.g. ratio of odds ratios (ROR) or difference in standardised mean differences (dSMD)) in meta-analyses using the random-effects model. Analyses were stratified by type of outcome ("mortality" versus "other objective" versus "subjective"). Direction of effect was standardised so that ROR < 1 and dSMD < 0 denotes a larger intervention effect estimate in trials with an inadequate or unclear (versus adequate) characteristic. RESULTS: We included 24 studies. The available evidence suggests that intervention effect estimates may be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (ROR 0.93, 95% CI 0.86 to 0.99; 7 studies) and allocation concealment (ROR 0.90, 95% CI 0.84 to 0.97; 7 studies). For these characteristics, the average bias appeared to be larger in trials of subjective outcomes compared with other objective outcomes. Also, intervention effects for subjective outcomes appear to be exaggerated in trials with lack of/unclear blinding of participants (versus blinding) (dSMD -0.37, 95% CI -0.77 to 0.04; 2 studies), lack of/unclear blinding of outcome assessors (ROR 0.64, 95% CI 0.43 to 0.96; 1 study) and lack of/unclear double blinding (ROR 0.77, 95% CI 0.61 to 0.93; 1 study). The influence of other characteristics (e.g. unblinded trial personnel, attrition) is unclear. CONCLUSIONS: Certain characteristics of randomized trials may exaggerate intervention effect estimates. The average bias appears to be greatest in trials of subjective outcomes. More research on several characteristics, particularly attrition and selective reporting, is needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig 3 Random-effects meta-analysis of RORs associated with inadequate/unclear (versus adequate) sequence generation. The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR is (95% CrI) 0.89 (0.82, 0.96)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.89 (0.75, 1.05)]. The Unverzagt 2013c ROR is based on a multivariable analysis with adjustment for allocation concealment, double blinding, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.98 (0.8, 1.21)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.99 (0.84, 1.16)]. The BRANDO 2012e ROR is based on a multivariable analysis with adjustment for allocation concealment and double blinding [the corresponding univariable ROR (95% CrI) is 0.83 (0.74, 0.94)].](https://figures.cochat.ai/paper-figures/2983b5b33b19494280d6784de9530e9d/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-3.jpg)
![Fig 4 Random-effects meta-analysis of RORs associated with inadequate/unclear (versus adequate) allocation concealment. The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.93 (0.87, 0.99)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.98 (0.88, 1.10)]. The BRANDO 2012c ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.97 (0.85, 1.10)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for sequence generation and double blinding [the corresponding univariable ROR (95% CrI) is 0.85 (0.75, 0.95)].](https://figures.cochat.ai/paper-figures/594b60d931a944d4b7cd0812e7e54cf5/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-4.jpg)
![Fig 5 Random-effects meta-analysis of RORs and dSMDs associated with presence (versus absence) of baseline imbalance. The Unverzagt 2013a ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.92 (0.80, 1.06)].](https://figures.cochat.ai/paper-figures/2117adfe36b54af3aedfc58feb8e5a77/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-5.jpg)


![Fig 8 Estimated RORs and dSMDs associated with any (versus no or minimal) attrition. The boxed section displays the average bias estimates, where available, from the four meta-epidemiological studies contributing to the BRANDO 2012 study. The Abraha 2015a ROR is based on a multivariable analysis with adjustment for use of placebo comparison, sample size, type of centre, items of risk of bias, post-randomisation exclusions, funding, and publication bias [the corresponding univariable ROR (95% CI) is 0.83 (0.71, 0.97)]. The Unverzagt 2013b ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 1.19 (0.98, 1.45)]. The Nuesch 2009bc dSMD is based on a multivariable analysis with adjustment for allocation concealment [the corresponding multivariable dSMD (95% CI) with adjustment for blinding of participants is -0.15 (-0.30, 0.00), and the corresponding univariable dSMD (95% CI) is -0.13 (-0.29, 0.04)].](https://figures.cochat.ai/paper-figures/0e8ce72c7fdb470c826c8be718f13f64/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-8.jpg)

![Fig 10 Random-effects meta-analysis of RORs associated with lack of/unclear double blinding (versus double blinding). The boxed section displays the average bias estimates, where available, from the seven meta-epidemiological studies contributing to the BRANDO 2012a study (however only the BRANDO 2012a ROR was included in our meta-analysis). The BRANDO 2012a ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.87 (0.79, 0.96)]. The BRANDO 2012b ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.92 (0.80, 1.04)]. The Unverzagt 2013c ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, attrition, selective outcome reporting, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.84 (0.69, 1.02)]. The BRANDO 2012d ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.93 (0.74, 1.18)]. The BRANDO 2012e ROR is based on a multivariable analysis with adjustment for sequence generation and allocation concealment [the corresponding univariable ROR (95% CrI) is 0.78 (0.65, 0.92)].](https://figures.cochat.ai/paper-figures/2137b7d01f5c4ed3952ef80ca31122dd/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-10.jpg)
![Fig 11 Random-effects meta-analysis of RORs and dSMDs associated with high/unclear (versus low) risk of bias due to selective reporting. The Unverzagt 2013a ROR is based on a multivariable analysis with adjustment for sequence generation, allocation concealment, double blinding, attrition, early stopping, pre-intervention, competing interests, baseline imbalance, switching interventions, sufficient follow-up, and single- versus multi-centre status [the corresponding univariable ROR (95% CI) is 0.73 (0.54, 0.98)].](https://figures.cochat.ai/paper-figures/518f65e477414b18b41b5c661142ba2f/empirical-evidence-of-study-design-biases-in-randomized-trials-systematic-review-figure-11.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical evidence and theoretical considerations suggest that several features of study design are associated with larger intervention effect estimates in studies; these features include inadequate generation and concealment of a random sequence to assign participants to groups, substantial loss to follow-up of participants, and unblinded outcome assessment.”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [80]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“based on meta-epidemiological studies [22][23][24][25] that found little to no effect of blinding on treatment results among randomised trials (including a review of oral health trials 26 )”
From Management of chronic pain secondary to temporomandibular disorders: a systematic review and network meta-analysis of randomised trials (2023) · cites this paper as [23]
Experimental — accuracy not yet measuredIn the citing paper
“There is a suggestion of missing results in the left-hand side of the plot, where results would be unfavourable to SSRIs and in the area of statistical nonsignificance (i.e. the white area where P > 0.10), which adds credence to the possibility that the asymmetry is due to reporting biases.”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [59]
In the citing paper
“There is a suggestion of missing results in the lefthand side of the plot, where results would be unfavourable to SSRIs, and in the area of statistical nonsignificance, which adds credence to the possibility that the asymmetry is due to reporting biases.”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Self-citationPreferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · cited 1×
“We used DerSimonian and Laird’s method of moments estimator to estimate the betweenstudy variance [22].”
MethodThe Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Zhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2015) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,533
Research works printed but not linked: 3
- OCEBM Levels of Evidence Working Group. " The Oxford 2011 Levels of Evidence". Oxford Centre for Evidence-Based Medicine Available: http://www.cebm.net/index.aspx?o=5653.
- Berkman ND, Santaguida PL, Viswanathan M, Morton SC. AHRQ Methods for Effective Health Care. The Empirical Evidence of Bias in Trials Measuring Treatment Differences. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014. 25392898
- Als-Nielsen B, Chen W, Gluud LL, Siersma V, Hilden J, Gluud C. Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials [abstract]. 12th Cochrane Colloquium: Bridging the Gaps; 2004 Oct 2–6; Ottawa, Ontario, Canada [Internet]. 2004:[102–3 pp.]. Available: http://onlinelibrary.wiley.com/o/cochrane/clcmr/articles/CMR-6643/frame.html.
Cited by
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
“…the original tool was used in practice, 6 a systematic review and meta-analysis of meta-epidemiological studies of empirical evidence for biases associated with characteristics of randomised trials 12 and a cross-sectional study of how selective outcome reporting was assessed in Cochrane reviews.”
MethodSelf-citationPage, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Self-citationPage, M. J., Moher, D., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/gwdhk · Cited by 916
Self-citationCONSORT 2010 statement: extension to randomised crossover trials
Dwan, K., Li, T., Altman, D. G., & Elbourne, D. (2019) · BMJ · doi:10.1136/bmj.l4378 · Cited by 785
Why clinical trial outcomes fail to translate into benefits for patients
Heneghan, C., Goldacre, B., & Mahtani, K. R. (2017) · Trials · doi:10.1186/s13063-017-1870-2 · Cited by 404
“In addition, trials with inadequate or unclear sequence generation also biased estimates when outcomes were subjective [20].”
BackgroundAssessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402
Self-citationSavović, J., Turner, R. M., Mawdsley, D., Jones, H. E., Beynon, R., Higgins, J. P. T., & Sterne, J. A. C. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwx344 · Cited by 382
Self-citationQUADAS-C: A Tool for Assessing Risk of Bias in Comparative Diagnostic Accuracy Studies
Yang, B., Mallett, S., Takwoingi, Y., Davenport, C. F., Hyde, C. J., Whiting, P. F., Deeks, J. J., Leeflang, M. M. G., & the QUADAS-C Group† (2021) · Annals of Internal Medicine · doi:10.7326/m21-2234 · Cited by 324
“Studies investigating bias in randomized trials of interventions (15) were consulted as indirect evidence for items relating to the randomization process.”
MethodMoustgaard, H., Clayton, G. L., Jones, H. E., Boutron, I., Jørgensen, L., Laursen, D. R. T., Olsen, M. F., Paludan-Müller, A., Ravaud, P., Savović, J., et al. (2020) · BMJ · doi:10.1136/bmj.l6802 · Cited by 300
“7 32 We suggest careful consideration of the risk of confounding and of bias, such as bias due to misclassification of methodological characteristics or due to erroneous identification of treatments as experimental and control, in meta-epidemiological studies.”
Self-citationPage, M. J., Sterne, J. A. C., Higgins, J. P. T., & Egger, M. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1468 · Cited by 294
Self-citationBero, L., Chartres, N., Diong, J., Fabbri, A., Ghersi, D., Lam, J., Lau, A., McDonald, S., Mintzes, B., Sutton, P., et al. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0915-2 · Cited by 278
“For example, randomisation and blinding are part of the Cochrane risk of bias tool for randomised trials because there is evidence that inadequate application of these methods overestimates efficacy estimates [42, 43].”
Considering bias and conflicts of interest among the included studies
Boutron, I., Page, M. J., Higgins, J. P., Altman, D. G., Lundh, A., Hróbjartsson, A., & on behalf of the Cochrane Bias Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch7 · Cited by 258
Self-citationNaci, H., Davis, C., Savović, J., Higgins, J. P. T., Sterne, J. A. C., Gyawali, B., Romo-Sandoval, X., Handley, N., & Booth, C. M. (2019) · BMJ · doi:10.1136/bmj.l5221 · Cited by 243
Self-citationDechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171
Psychotherapies for borderline personality disorder: a focused systematic review and meta-analysis
Stoffers-Winterling, J. M., Storebø, O. J., Kongerslev, M. T., Faltinsen, E., Todorovac, A., Sedoc Jørgensen, M., Sales, C. P., Edemann Callesen, H., Pereira Ribeiro, J., Völlm, B. A., et al. (2022) · The British Journal of Psychiatry · doi:10.1192/bjp.2021.204 · Cited by 156
Munkholm, K., Paludan-Müller, A. S., & Boesen, K. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-024886 · Cited by 151
“Trials at high or unclear risk of bias within these two domains are likely to report inflated effect estimates, especially of subjective outcomes.(19) Cipriani et al did not describe how they assessed the risk of bias in relation to the randomisation sequence generation or the allocation concealment, and we were there…”
MethodBlinding in Clinical Trials: Seeing the Big Picture
Monaghan, T. F., Agudelo, C. W., Rahman, S. N., Wein, A. J., Lazar, J., Everaert, K., & Dmochowski, R. R. (2021) · Medicina · doi:10.3390/medicina57070647 · Cited by 169
“Moreover, compared to participants and outcome assessors, the impact of blinding of other trial personnel and healthcare professionals on estimated treatment effect is even less wellestablished [32,33].”
ResultCurcumin Supplementation and Human Disease: A Scoping Review of Clinical Trials
Panknin, T., Howe, C. L., Hauer, M., Bucchireddigari, B., Rossi, A., & Funk, J. L. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24054476 · Cited by 133
“Most important among these was an analysis of the prevale f it ti s reporting results from double-blind, randomized placebo-controlled trials ( -R T), a gold standard design for clinical trials [419], albeit one that tends to minimize treatment effects [420].”
BackgroundUhre, C. F., Uhre, V. F., Lønfeldt, N. N., Pretzmann, L., Vangkilde, S., Plessen, K. J., Gluud, C., Jakobsen, J. C., & Pagsberg, A. K. (2020) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2019.08.480 · Cited by 103
Smedemark, S. A., Aabenhus, R. M., Llor, C., Fournaise, A., Olsen, O., & Jørgensen, K. J. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010130.pub3 · Cited by 145
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Pre-Clinical Testing of Therapies for Traumatic Brain Injury
DeWitt, D. S., Hawkins, B. E., Dixon, C. E., Kochanek, P. M., Armstead, W. M., Bass, C. R., Bramlett, H. M., Büki, A., Dietrich, W. D., Ferguson, A. R., et al. (2018) · Journal of Neurotrauma · doi:10.1089/neu.2018.5778 · Cited by 93
“14; Page et al.(68) Sample size Inadequate sample size to produce adequate statistical power Burke et al.”
Eick, S. M., Goin, D. E., Chartres, N., Lam, J., & Woodruff, T. J. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01490-8 · Cited by 82
Interventions for improving sleep quality in people with chronic kidney disease
Natale, P., Ruospo, M., Saglimbene, V. M., Palmer, S. C., & Strippoli, G. F. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012625.pub2 · Cited by 92
Poolsup, N., Suksomboon, N., Kurnianta, P. D. M., & Deawjaroen, K. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215840 · Cited by 83
RetractedCiani, O., Piepoli, M. F., Smart, N. A., Uddin, J., Walker, S., Warren, F. C., Zwisler, A., Davos, C. H., & Taylor, R. S. (2018) · JACC Heart Failure · doi:10.1016/j.jchf.2018.03.017 · Cited by 79
Young, I., Greig, J., Wilhelm, B. J., & Waddell, L. A. (2019) · Journal of Food Protection · doi:10.4315/0362-028x.jfp-19-108 · Cited by 68
Hengartner, M. P., Jakobsen, J. C., Sørensen, A., & Plöderl, M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229381 · Cited by 66
“Fourth, most studies were rated at high or unclear risk of bias with respect to allocation concealment and blinding of participants [11], which also results in exaggerated effect size estimates in trials of subjective outcomes such as the HDRS or MADRS [35, 36].”
BackgroundRespiratory muscle training in children and adults with neuromuscular disease
Silva, I. S., Pedrosa, R., Azevedo, I. G., Forbes, A., Fregonezi, G., Júnior, M. E. D., Lima, S. R. H., & Ferreira, G. M. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011711.pub2 · Cited by 85
Conde‐Agudelo, A., & Romero, R. J. (2022) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2022.04.023 · Cited by 71
Haigler, M. C., Abdulrehman, E., Siddappa, S., Kishore, R., Padilla, M., & Enciso, R. (2018) · The Journal of the American Dental Association · doi:10.1016/j.adaj.2018.07.025 · Cited by 78
Urgently seeking efficiency and sustainability of clinical trials in global health
Park, J., Grais, R. F., Taljaard, M., Nakimuli‐Mpungu, E., Jehan, F., Nachega, J. B., Ford, N. P., Xavier, D., Kengne, A. P., Ashorn, P., et al. (2021) · The Lancet Global Health · doi:10.1016/s2214-109x(20)30539-8 · Cited by 62
Schmaußer, M., Hoffmann, S., Raab, M., & Laborde, S. (2022) · Journal of Neuroscience Research · doi:10.1002/jnr.25062 · Cited by 59
Disher, T. C., Gullickson, C., Singh, B., Cameron, C., Boulos, L., Beaubien, L., & Campbell-Yeo, M. L. (2019) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2018.5044 · Cited by 68
Wang, Y., Parpia, S., Couban, R., Wang, Q., Armijo-Olivo, S., Bassler, D., Briel, M., Brignardello-Petersen, R., Gluud, L. L., Keitz, S. A., et al. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.11.001 · Cited by 58
Yao, L., Sadeghirad, B., Li, M., Li, J., Wang, Q., Crandon, H. N., Martin, G., Morgan, R., Florez, I. D., Hunskaar, B. S., et al. (2023) · BMJ · doi:10.1136/bmj-2023-076226 · Cited by 79
Büttner, F., Winters, M., Delahunt, E., Elbers, R., Lura, C. B., Khan, K. M., Weir, A., & Ardern, C. L. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-101675 · Cited by 55
“Many systematic reviews assessed study quality instead of risk of bias, which impairs accurate inferences about the credibility of study outcomes.”
Loibl, S., Loirat, D., Tolaney, S. M., Punie, K., Oliveira, M., Rugo, H. S., Bardia, A., Hurvitz, S. A., Brufsky, A., Kalinsky, K. M., et al. (2022) · European Journal of Cancer · doi:10.1016/j.ejca.2022.10.003 · Cited by 56
“The open-label design could also have influenced patient responses by biasing patient responses in favour of one intervention [24].”
Wilshire, C. E., Kindlon, T., Matthees, A., & McGrath, S. J. (2016) · Fatigue Biomedicine Health & Behavior · doi:10.1080/21641846.2017.1259724 · Cited by 62
A systematic review: Tools for assessing methodological quality of human observational studies
Wang, Z., Taylor, K., Allman-Farinelli, M., Armstrong, B., Askie, L., Ghersi, D., McKenzie, J., Norris, S. L., Page, M., Rooney, A. A., et al. (2019) · doi:10.31222/osf.io/pnqmy · Cited by 52
Self-citationPhysical rehabilitation approaches for the recovery of function and mobility following stroke
Todhunter‐Brown, A., Sellers, C., Baer, G. D., Choo, P. L., Cowie, J., Cheyne, J. D., Langhorne, P., Brown, J., Morris, J. H., & Campbell, P. (2025) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001920.pub4 · Cited by 157
Respiratory muscle training in children and adults with neuromuscular disease
Pedrosa, R., Silva, I. S., Azevedo, I. G., Forbes, A., Fregonezi, G., Júnior, M. E. D., Lima, S. R. H., Guerra, R. O., & Ferreira, G. M. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011711 · Cited by 50
Mocellin, M. C., Fernandes, R., Chagas, T. R., & Trindade, E. B. S. D. M. (2017) · Clinical Nutrition · doi:10.1016/j.clnu.2017.05.008 · Cited by 51
Büttner, F., Winters, M., Delahunt, E., Elbers, R., Lura, C. B., Khan, K. M., Weir, A., & Ardern, C. L. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-100806 · Cited by 47
“Study limitations that inform judgements of ‘some concerns’ or ‘high’ risk of bias can distort outcome measures differently.(5) For example, pain is more likely to be over-estimated when a patient is aware of their allocation to a specific intervention group (due to lack of patient blinding) than if they were not awar…”
BackgroundResultInfluentialBüttner, F., Toomey, E., McClean, S., Roe, M., & Delahunt, E. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-101863 · Cited by 47
A systematic review and meta-analysis of strategies to promote vaccination uptake
Liu, S., Durantini, M. R., Calabrese, C., Sánchez, F., & Albarracín, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01940-6 · Cited by 55
Damen, J. A., Debray, T. P. A., Pajouheshnia, R., Reitsma, J. B., Scholten, R. J. P. M., Moons, K. G. M., & Hooft, L. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-026160 · Cited by 50
Moustgaard, H., Jones, H. E., Savović, J., Clayton, G., Sterne, J. A. C., Higgins, J. P. T., & Hróbjartsson, A. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1392 · Cited by 45
“Variation between results of different metaepidemiological studies is also a challenge for other aspects of trial methodology, notably the impact of inadequate concealment of allocation.(7) Although such variation partly reflects imprecision, it may also imply varying degrees of systematic error or heterogeneity cause…”
BackgroundSelf-citationNejstgaard, C. H., Bero, L. A., Hróbjartsson, A., Jørgensen, A. W., Jørgensen, K. J., Le, M., & Lundh, A. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000040.pub3 · Cited by 59
“This cut-o? was based on e?ect sizes of important study design biases in trials ( Page 2016 ).”
BackgroundSelf-citationRhodes, K. M., Turner, R. M., Savović, J., Jones, H. E., Mawdsley, D., & Higgins, J. P. T. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.025 · Cited by 44
“Within-study biases can lead to overestimation or underestimation of the true intervention effect in a study and are expected to contribute to between-study variation in metaanalyses [5,6,16].”
Self-citationSaltaji, H., Armijo-Olivo, S., Cummings, G. G., Amin, M., & Flores-Mir, C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0190089 · Cited by 43
Efficace, F., Cella, D. F., Aaronson, N. K., Calvert, M., Cottone, F., Maïo, M. D., Perrone, F., Sparano, F., Gamper, E., Vignetti, M., et al. (2021) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djab177 · Cited by 50
Testa, B., Reid, J., Scott, M., Murison, P. J., & Bell, A. M. (2021) · Frontiers in Veterinary Science · doi:10.3389/fvets.2021.751949 · Cited by 43
Ho, R. S. T., Chung, V. C., Wong, C. H. L., Wu, J. C. Y., Wong, S. Y. S., & Wu, X. (2017) · Scientific Reports · doi:10.1038/s41598-017-09856-0 · Cited by 44
Putzu, A., Clivio, S., Belletti, A., & Cassina, T. (2017) · International Journal of Cardiology · doi:10.1016/j.ijcard.2017.10.077 · Cited by 44
Acupuncture and Related Therapies for Obesity: A Network Meta‐Analysis
Zhang, Y., Li, J., Mo, G., Liu, J., Yang, H., Chen, X., Liu, H., Cai, T., Zhang, X., Tian, X., et al. (2018) · Evidence-based Complementary and Alternative Medicine · doi:10.1155/2018/9569685 · Cited by 43
Bertizzolo, L., Bossuyt, P., Atal, I., Ravaud, P., & Dechartres, A. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-028382 · Cited by 35
Vorland, C. J., Brown, A. W., Dawson, J., Dickinson, S., Golzarri‐Arroyo, L., Hannon, B. A., Heo, M., Heymsfield, S. B., Jayawardene, W. P., Kahathuduwa, C. N., et al. (2021) · International Journal of Obesity · doi:10.1038/s41366-021-00909-z · Cited by 43
“Trials with inadequate concealment are associated with larger effect estimates than are those with adequate concealment [144-147], likely reflecting bias.”
Minozzi, S., Gonzalez-Lorenzo, M., Cinquini, M., Berardinelli, D., Cagnazzo, C., Ciardullo, S., De Nardi, P., Gammone, M., Iovino, P., Lando, A., et al. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.09.003 · Cited by 41
Páez, A., Nunan, D., McCulloch, P. G., & Beard, D. J. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111610 · Cited by 44
Østengaard, L., Lundh, A., Tjørnhøj-Thomsen, T., Abdi, S., Gelle, M. H. A., Stewart, L. A., Boutron, I., & Hróbjartsson, A. (2020) · BMJ · doi:10.1136/bmj.m3764 · Cited by 35
Self-citationThe impact of blinding on trial results: A systematic review and meta‐analysis
(2023) · doi:10.1002/cesm.12015 · Cited by 35
“Our review improves on previous reviews by performing an assessment of risk of bias and the certainty of evidence and including several recently published studies not included in previous reviews [10, 13, 17 – 19, 33, 43].”
Blinding in trials of interventional procedures is possible and worthwhile
Wartolowska, K. A., Beard, D. J., & Carr, A. J. (2017) · F1000Research · doi:10.12688/f1000research.12528.1 · Cited by 34
Wu, X., Wong, C. H. L., Ho, R. S. T., Cheung, W. K. W., Ford, A. C., Wu, J. C. Y., Mak, A. D. P., Cramer, H., & Chung, V. C. (2019) · Therapeutic Advances in Gastroenterology · doi:10.1177/1756284818820438 · Cited by 38
Mechanisms and direction of allocation bias in randomised clinical trials
Paludan‐Müller, A. S., Laursen, D. R. T., & Hróbjartsson, A. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0235-y · Cited by 35
“A similar problem could impact on the empirical challenges of assessing other types of bias in randomised trials, for example bias due to attrition, blinding and outcome selection [5, 68].”
BackgroundMethodResultInfluentialSelf-citationThe Nordic Nutrition Recommendations 2022 – structure and rationale of qualified systematic reviews
Arnesen, E., Christensen, J. J., Andersen, R., Eneroth, H., Erkkola, M., Høyer, A., Lemming, E. W., Meltzer, H. M., Halldorsson, T. I., Þórsdóttir, I., et al. (2020) · Food & Nutrition Research · doi:10.29219/fnr.v64.4403 · Cited by 32
Herbert, R. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.024 · Cited by 32
Risk of Bias in Network Meta-Analysis (RoB NMA) tool
Lunny, C., Higgins, J. P. T., White, I. R., Dias, S., Hutton, B., Wright, J. M., Veroniki, A. A., Whiting, P. F., & Tricco, A. C. (2025) · BMJ · doi:10.1136/bmj-2024-079839 · Cited by 41
“72 Therefore, assessing the potential for biases in the results of the included primary studies is important.”
Self-citationPreobrazenski, N., Seigel, J., Halliday, S., Janssen, I., & McGlory, C. (2023) · Journal of Cachexia Sarcopenia and Muscle · doi:10.1002/jcsm.13201 · Cited by 37
Cashin, A. G., Lee, H., Bagg, M. K., O’Hagan, E. T., Traeger, A. C., Kamper, S. J., Folly, T., Jones, M. D., Booth, J., & McAuley, J. H. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.06.025 · Cited by 35
“Quality of physical therapy intervention trials for low back pain Cashin et al. (2020) Page 16 of 18”
BackgroundSystematic Reviews in Health Research
Egger, M., Higgins, J. P. T., & Smith, G. D. (2022) · Wiley eBooks · doi:10.1002/9781119099369.ch1 · Cited by 43
Self-citationAstorino, T. A., Causer, E., Hazell, T. J., Arhen, B. B., & Gurd, B. J. (2022) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000002993 · Cited by 37
Karjalainen, T., Heikkinen, J., Busija, L., Jokihaara, J., Lewin, A. M., Naylor, J., Harris, L., Harris, I. A., Buchbinder, R., & Adie, S. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.23903 · Cited by 32
Hung, A., Nelson, H., & Koehle, M. S. (2021) · Sports Medicine · doi:10.1007/s40279-021-01544-4 · Cited by 33
“Empirical evidence suggests RCTs with inadequate/unclear allocation concealment tend to find exaggerated effect estimates, particularly in subjective outcomes [86].”
BackgroundEfficacy of melatonin in term neonatal models of perinatal hypoxia‐ischaemia
Pang, R., Han, H. J., Meehan, C., Golay, X., Miller, S., & Robertson, N. J. (2022) · Annals of Clinical and Translational Neurology · doi:10.1002/acn3.51559 · Cited by 29
Johnsen, A. T., Petersen, M. A., Sjøgren, P., Pedersen, L., Neergaard, M. A., Damkier, A., Gluud, C. N., Fayers, P. M., Lindschou, J., Strömgren, A. S., et al. (2019) · Supportive Care in Cancer · doi:10.1007/s00520-019-05021-7 · Cited by 31
Bonafiglia, J. T., Islam, H., Preobrazenski, N., & Gurd, B. J. (2021) · Journal of sport and health science/Journal of Sport and Health Science · doi:10.1016/j.jshs.2021.03.005 · Cited by 30
Hunter, B. R., Wang, A. Z., Bucca, A. W., Musey, P. I., Strachan, C. C., Roumpf, S. K., Propst, S. L., Croft, A., Menard, L. M., & Kirschner, J. M. (2022) · JAMA Neurology · doi:10.1001/jamaneurol.2022.1858 · Cited by 44
Blinding in trials of interventional procedures is possible and worthwhile
Wartolowska, K. A., Beard, D. J., & Carr, A. J. (2017) · F1000Research · doi:10.12688/f1000research.12528.2 · Cited by 50
Rufiange, M., Rousseau‐Blass, F., & Pang, D. (2019) · Veterinary Record Open · doi:10.1136/vetreco-2018-000322 · Cited by 26
Cho, Y., Kim, C., & Kang, B. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-023725 · Cited by 24
Haring, R., Ghannad, M., Bertizzolo, L., & Page, M. J. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.02.004 · Cited by 27
Self-citationMeasuring the success of blinding in placebo-controlled trials: Should we be so quick to dismiss it?
Webster, R., Bishop, F. L., Collins, G. S., Evers, A. W. M., Hoffmann, T., Knottnerus, J. A., Lamb, S. E., Macdonald, H., Madigan, C. D., Napadow, V., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.022 · Cited by 24
Sonis, J. H., & Cook, J. M. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.112637 · Cited by 31
Nguyen, T., & Xie, L. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.10.012 · Cited by 25
Lee, E. K. P., Wang, S., Ng, W., Ramdzan, S. N., Tse, E., Chan, L., Rashid, A., Chin, W., Yu, C., Sit, R. W. S., et al. (2024) · Journal of Hypertension · doi:10.1097/hjh.0000000000003783 · Cited by 21
Armijo-Olivo, S. L., Mohamad, N., de Oliveira‐Souza, A. I. S., de Castro‐Carletti, E. M., Ballenberger, N., & Fuentes, J. (2021) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000001893 · Cited by 26
Reporting and communication of randomisation procedures is suboptimal in veterinary trials
Girolamo, N. D., Giuffrida, M. A., Winter, A. L., & Reynders, R. M. (2017) · Veterinary Record · doi:10.1136/vr.104035 · Cited by 22
Berthelsen, D. B., Ginnerup‐Nielsen, E., Juhl, C. B., Lund, H., Henriksen, M., Hróbjartsson, A., Nielsen, S. M., Voshaar, M., & Christensen, R. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.10.016 · Cited by 19
Self-citationDeer, T. R., Costandi, S. J., Washabaugh, E. P., Chafin, T. B., Wahezi, S. E., Jassal, N. S., & Sayed, D. (2022) · Pain Medicine · doi:10.1093/pm/pnac028 · Cited by 20
“5 indicate that the patient is satisfied [17].”
BackgroundDavis, C., Wagner, A. K., Salcher‐Konrad, M., Scowcroft, H., Mintzes, B. J., Pokorny, A. M. J., Lew, J., & Naci, H. (2023) · BMJ · doi:10.1136/bmj-2022-073711 · Cited by 21
Yi, J., Li, H., Li, Y., Li, X., Zheng, W., & Zhao, Z. (2019) · Journal of Endodontics · doi:10.1016/j.joen.2019.10.016 · Cited by 19
Transcutaneous electrical nerve stimulation (TENS) for chronic pain: the opportunity to begin again
Travers, M. J., O'Connell, N. E., Tugwell, P., Eccleston, C., & Gibson, W. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000139 · Cited by 24
Roydhouse, J. K., Mishra-Kalyani, P. S., Bhatnagar, V., Gutman, R., King‐Kallimanis, B. L., Sridhara, R., & Kluetz, P. G. (2021) · Value in Health · doi:10.1016/j.jval.2020.12.015 · Cited by 19
“A recent systematic review that included 24 studies found that lack of blinding for “subjective” outcomes was associated with a larger intervention effect.(30) The association between past study characteristics and treatment effects has varied across therapeutic areas,(31) making it difficult to compare our findings t…”
MethodResultPreobrazenski, N., Islam, H., & Gurd, B. J. (2021) · European Journal of Applied Physiology · doi:10.1007/s00421-021-04669-6 · Cited by 20
Stadelmaier, J., Roux, I., Πετροπούλου, Μ., & Schwingshackl, L. (2022) · BMC Medicine · doi:10.1186/s12916-022-02540-9 · Cited by 20
“Similarly, Page and colleagues [8] synthesised evidence from 24 meta-epidemiological studies and observed that certain characteristics of RCTs such as sequence generation, allocation concealment, and double blinding may exaggerate the intervention effect estimates for subjective compared with objective outcomes.”
BackgroundResultMickenautsch, S., Rupf, S., Miletić, I., & Yengopal, V. (2022) · PLoS ONE · doi:10.1371/journal.pone.0279645 · Cited by 17
“…establish a statistically significant effect estimate difference between trials with a sample size of below and above 200 patients (S2 File MA-ID10 Page 16), despite both studies indicating a statistically significant effect estimate difference each (ROR 0.76; 95%CI: 0.68 to 0.85 [23] and ROR…”
BackgroundMethodResultInfluentialRisk of bias in exercise science: A systematic review of 340 studies
Preobrazenski, N., McCaig, A., Turner, A., Kushner, M., Pacitti, L. J., Mendolia, P., MacDonald, B., Storoschuk, K. L., Bouck, T., Zaza, Y., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109010 · Cited by 25
“…trials in the clinical literature, 19–21 meta-epidemiological studies of papers published in the post-CONSORT era judged that most studies continued to have a high or unclear risk of bias across several other domains of science 14,22–25 including clinical 26 and preclinical 27 literature.”
BackgroundBalaguier, R., Madeleine, P., Rose-Dulcina, K., & Vuillerme, N. (2017) · Journal of Agromedicine · doi:10.1080/1059924x.2017.1317683 · Cited by 25
Niederer, D., Weippert, M., & Behrens, M. (2022) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2022.11149 · Cited by 17
Metadata
- Authors
- Page, M. J., Higgins, J. P. T., Clayton, G., Sterne, J. A. C., Hróbjartsson, A., & Savović, J.
- Affiliations
- University of Bristol (GB)
- University of Southern Denmark (DK)
- University Hospitals Bristol NHS Foundation Trust (GB)
- Corresponding author
- Page, Matthew J.
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-07-11
- Volume
- 11
- Issue
- 7
- Pages
- e0159267
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4939945
- OpenAlex
- W2461177968
- Cited by
- 284
- References
- 64
- Citations by year
- 2026: 24 · 2025: 22 · 2024: 25 · 2023: 29 · 2022: 45 · 2021: 35 · 2020: 33 · 2019: 32 · 2018: 21 · 2017: 14 · 2016: 3
- FWCI
- 7.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- University of Southern Denmark Research Portal (University of Southern Denmark) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Explore Bristol Research (submittedVersion, cc-by)
- Funders
- MRC Network of Hubs for Trials Methodology Research (MR/L004933/1- N61)
- National Health and Medical Research Council (1088535)
- National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- study design bias, meta-epidemiology, sequence generation, allocation concealment, blinding, double blinding, subjective outcomes, objective outcomes, attrition, selective attrition, reporting quality, trial heterogeneity, standardized mean difference
- MeSH terms
- Humans, Bias, Epidemiologic Studies, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.