A methodological review of recent meta-analyses has found significant heterogeneity in age between randomized groups
Abstract
Background There is evidence to suggest that component randomized controlled trials (RCTs) within systematic reviews may be biased. It is important that these reviews are identified to prevent erroneous conclusions influencing health care policies and decisions.Purpose To assess the likelihood of bias in trials in 12 meta-analyses.Design A review of 12 systematic reviews.Data sources Twelve recently published systematic reviews with 503 component randomized trials, published in the British Medical Journal, The Lancet, Journal of the American Medical Association, and The Annals of Internal Medicine before May 2012.Study selection and data extraction Systematic reviews were eligible for inclusion if they included only RCTs. We obtained the full text for the component RCTs of the 12 systematic reviews (in English only). We extracted summary data on age, number of participants in each treatment group, and the method of allocation concealment for each RCT.Data synthesis Five of the 12 meta-analyses exhibited heterogeneity in age differences (I(2) > 0.30), when there should have been none. In two meta-analyses, the age of the intervention group was significantly greater than that of the control group. Inadequate allocation concealment was a statistically significant predictor of heterogeneity in one trial as observed by a metaregression.Conclusions Most of the sample of recent meta-analyses showed that there were signs of imbalance and/or heterogeneity in ages between treatment groups, when there should have been none. Systematic reviewers might consider using the techniques described here to assess the validity of their findings.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Poor randomisation methods can lead to further problems with overexaggerated systematic reviews.
“Poor randomisation methods with no concealment can lead to exaggerated treatment effects,79 80 which can then lead to further problems with overexaggerated systematic reviews.81”
From Novel ACT-based eHealth psychoeducational intervention for students with mental distress: A study protocol for a mixed-methodology pilot trial (2019) · cites this paper as [81]
Cites
Estimating the mean and variance from the median, range, and the size of a sample
Hozo, S. P., Djulbegovic, B., & Hozo, I. (2005) · BMC Medical Research Methodology · doi:10.1186/1471-2288-5-13 · Cited by 9,062 · see citing works
(2008) · Cited by 3,790 · see citing works
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127 · see citing works
Chapter 9: Analysing Data and Undertaking Meta-Analyses
Deeks, J. J., Higgins, J. P. T., & Altman, D. G. (2008) · Cited by 2,782 · see citing works
Language bias in randomised controlled trials published in English and German
Egger, M., Zellweger-Zähner, T., Schneider, M., Junker, C., Lengeler, C., & Antes, G. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02419-7 · Cited by 1,028 · see citing works
Leucht, S., Tardy, M., Komossa, K., Heres, S., Kissling, W., Salanti, G., & Davis, J. M. (2012) · The Lancet · doi:10.1016/s0140-6736(12)60239-6 · Cited by 931 · see citing works
Allocation concealment in randomised trials: defending against deciphering
Schulz, K. F., & Grimes, D. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07750-4 · Cited by 923 · see citing works
Direction and impact of language bias in meta-analyses of controlled trials: empirical study
Jüni, P., Holenstein, F., Sterne, J., Bartlett, C., & Egger, M. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.115 · Cited by 892 · see citing works
Newberry, S. J. (2012) · JAMA · doi:10.1001/jama.2012.3507 · Cited by 870 · see citing works
Coombes, B. K., Bisset, L., & Vicenzino, B. (2010) · The Lancet · doi:10.1016/s0140-6736(10)61160-9 · Cited by 861 · see citing works
Thangaratinam, S., Rogozinska, E., Jolly, K., Glinkowski, S., Roseboom, T., Tomlinson, J. W., Kunz, R., Mol, B. W., Coomarasamy, A., & Khan, K. S. (2012) · BMJ · doi:10.1136/bmj.e2088 · Cited by 802 · see citing works
Orrow, G., Kinmonth, A.-L., Sanderson, S., & Sutton, S. (2012) · BMJ · doi:10.1136/bmj.e1389 · Cited by 739 · see citing works
Viscosupplementation for Osteoarthritis of the Knee: A Systematic Review and Meta-analysis
Rutjes, A. W. S., Jüni, P., da Costa, B. R., Trelle, S., Nüesch, E., & Reichenbach, S. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-3-201208070-00473 · Cited by 717 · see citing works
Randomization and allocation concealment: a practical guide for researchers
Doig, G. S., & Simpson, F. O. (2005) · Journal of Critical Care · doi:10.1016/j.jcrc.2005.04.005 · Cited by 664 · see citing works
Assessing the Quality of Randomized Trials
Clark, H. D., Wells, G. A., Huët, C., McAlister, F. A., Salmi, L., Fergusson, D. A., & Laupacis, A. (1999) · Controlled Clinical Trials · doi:10.1016/s0197-2456(99)00026-4 · Cited by 649 · see citing works
Heneghan, C. J., Ward, A. M., Perera, R., Bankhead, C. R., Fuller, A., Stevens, R., Bradford, K., Tyndel, S., Alonso‐Coello, P., Ansell, J., et al. (2011) · The Lancet · doi:10.1016/s0140-6736(11)61294-4 · Cited by 373 · see citing works
Management of Chronic Prostatitis/ Chronic Pelvic Pain Syndrome
Anothaisintawee, T. (2011) · JAMA · doi:10.1001/jama.2010.1913 · Cited by 277 · see citing works
The analysis of 168 randomised controlled trials to test data integrity
Carlisle, J. B. (2012) · Anaesthesia · doi:10.1111/j.1365-2044.2012.07128.x · Cited by 255 · see citing works
Concealing treatment allocation in randomised trials
Altman, D. G., & Schulz, K. F. (2001) · BMJ · doi:10.1136/bmj.323.7310.446 · Cited by 222 · see citing works
Palmer, S. C., Di Micco, L., Razavian, M., Craig, J. C., Perkovic, V., Pellegrini, F., Copetti, M., Graziano, G., Tognoni, G., Jardine, M., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-6-201203200-00007 · Cited by 204 · see citing works
Hewitt, C., Hahn, S., Torgerson, D. J., Watson, J., & Bland, J. M. (2005) · BMJ · doi:10.1136/bmj.38413.576713.ae · Cited by 176 · see citing works
Self-citationEliminating bias in randomized controlled trials: importance of allocation concealment and masking.
Viera, A. J., & Bangdiwala, S. I. (2007) · PubMed · Cited by 169 · see citing works
A systematic review of the effects of calcium supplementation on body weight
Trowman, R., Dumville, J. C., Hahn, S., & Torgerson, D. J. (2006) · British Journal Of Nutrition · doi:10.1079/bjn20051727 · Cited by 115 · see citing works
Hemmingsen, B., Lundby‐Christensen, L., Wetterslev, J., Vaag, A. A., Gluud, C. N., Lund, S. S., & Almdal, T. (2012) · BMJ · doi:10.1136/bmj.e1771 · Cited by 108 · see citing works
Neumann, I., Rada, G., Claro, J. C., Carrasco-Labra, A., Thorlund, K., Akl, E. A., Bates, S. M., & Guyatt, G. H. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-10-201205150-00421 · Cited by 94 · see citing works
Dechartres, A., Charles, P., Hopewell, S., Ravaud, P., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.015 · Cited by 75 · see citing works
Trowman, R., Dumville, J. C., Torgerson, D. J., & Cranny, G. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.014 · Cited by 59 · see citing works
Do baseline p-values follow a uniform distribution in randomised trials?
Bland, M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0076010 · Cited by 47 · see citing works
Shiwa, S. R., Moseley, A. M., Maher, C. G., & Costa, L. O. P. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.08.004 · Cited by 44 · see citing works
Management of chronic prostatitis–chronic pelvic pain syndrome
Anderson, R. U. (2002) · Urologic Clinics of North America · doi:10.1016/s0094-0143(02)00009-5 · Cited by 37 · see citing works
Hewitt, C. E., Torgerson, D. J., & Berger, V. W. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.06.005 · Cited by 14 · see citing works
Strippoli, G. F., Palmer, S. C., & Craig, J. C. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-4-201208210-00025 · Cited by 9 · see citing works
Flagged first, then by citations
3 of the 36 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Pragmatic trials of pain therapies: a systematic review of methods
Hohenschurz‐Schmidt, D., Kleykamp, B. A., Draper‐Rodi, J., Vollert, J., Chan, J., Ferguson, M. C., McNicol, E. D., Phalip, J., Evans, S. R., Turk, D. C., et al. (2021) · Pain · doi:10.1097/j.pain.0000000000002317 · Cited by 49 · see citing works
A review of cluster randomized trials found statistical evidence of selection bias
Bolzern, J., Mnyama, N., Bosanquet, K., & Torgerson, D. J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.03.010 · Cited by 48 · see citing works
Self-citationSoeters, P., Bozzetti, F., Cynober, L., Elia, M., Shenkin, A., & Sobotka, L. (2016) · Clinical Nutrition · doi:10.1016/j.clnu.2015.08.008 · Cited by 40 · see citing works
Hicks, A., Fairhurst, C. M., & Torgerson, D. J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.001 · Cited by 32 · see citing works
Clark, L., Fairhurst, C. M., Cook, L., & Torgerson, D. J. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.023 · Cited by 17 · see citing works
Cochrane: the unfinished symphony of research synthesis
Roberts, I., & Ker, K. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0290-9 · Cited by 16 · see citing works
Luijendijk, H. J., Page, M. J., Burger, H., & Koolman, X. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01115-7 · Cited by 15 · see citing works
Mickenautsch, S., & Yengopal, V. (2024) · Cureus · doi:10.7759/cureus.63581 · Cited by 11 · see citing works
Mickenautsch, S., & Yengopal, V. (2024) · Cureus · doi:10.7759/cureus.60346 · Cited by 11 · see citing works
Edwards, D. J., Rainey, E., Boukouvala, V., Wells, Y., Bennett, P., Tree, J., & Kemp, A. H. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-029411 · Cited by 9 · see citing works
“Poor randomisation methods with no concealment can lead to exaggerated treatment effects,(79) 80 which can then lead to further problems with overexaggerated systematic reviews.(81) For this reason, individuals involved Table 1 Overview of ‘A Bite of ACT’ intervention”
BackgroundWewege, M. A., Hansford, H. J., Shah, B., Gilanyi, Y. L., Douglas, S. R. G., Parmenter, B. J., McAuley, J. H., & Jones, M. D. (2022) · Hypertension Research · doi:10.1038/s41440-022-00984-3 · Cited by 9 · see citing works
Hulshof, T. A., Zuidema, S. U., van Meer, P. J. K., Wied, C. C. G., & Luijendijk, H. J. (2018) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1757 · Cited by 7 · see citing works
Park, E.-G., & Hahn, S. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01620-x · Cited by 8 · see citing works
Mickenautsch, S., & Yengopal, V. (2024) · Cureus · doi:10.7759/cureus.58961 · Cited by 8 · see citing works
Applications of yoga in oral oncology: A systematic scoping review
Kanmodi, K. K., Braimah, R. O., Amzat, J., Salami, A. A., & Nnyanzi, L. A. (2023) · Health Science Reports · doi:10.1002/hsr2.1208 · Cited by 6 · see citing works
Mitchell, A. S., Moe‐Byrne, T., Cunningham‐Burley, R., Dean, A., Rangan, A., Roche, J., & Torgerson, D. J. (2019) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13313 · Cited by 6 · see citing works
Mickenautsch, S., & Yengopal, V. (2024) · Cureus · doi:10.7759/cureus.68911 · Cited by 4 · see citing works
I2 Statistic as a Test for Selection Bias in Randomised Controlled Trials
Mickenautsch, S., & Yengopal, V. (2025) · Cureus · doi:10.7759/cureus.84769 · Cited by 3 · see citing works
Mickenautsch, S., & Yengopal, V. (2024) · Cureus · doi:10.7759/cureus.71668 · Cited by 3 · see citing works
Hu, W., Guo, S., Meng, X., Wang, F., Ma, S., Zhang, C., Wang, J., Yuan, L., Zhang, L., Jing, Y., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-12312-z · Cited by 2 · see citing works
Seery, C., Twomey, D. M., Dully, J., Breheny, E., Skelly, T., Davenport, L., Bramham, J., Brady, N. P., & O’Keeffe, F. M. (2025) · Neuropsychological Rehabilitation · doi:10.1080/09602011.2025.2499745 · Cited by 3 · see citing works
Ceballos, A. D. A., Cárdenas, J. S. M., González, D. G., & Gómez, J. R. M. (2023) · Medicina UPB · doi:10.18566/medupb.v42n1.a06 · Cited by 1 · see citing works
van Bruggen, F. H., Zuidema, S. U., & Luijendijk, H. J. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02260-z · Cited by 1 · see citing works
Guo, X., Zhang, L., Zhang, Q., Kendrick, K. M., & Yao, S. (2026) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2026.106679 · Cited by 1 · see citing works
The I2 Statistic As Selection Bias Test: Updated Threshold Limits
Mickenautsch, S., & Yengopal, V. (2026) · Cureus · doi:10.7759/cureus.107553 · Cited by 1 · see citing works
Meta-analysis of Rare Diseases in Occupational Epidemiology.
McElvenny, D. (2017) · LSHTM Research Online (London School of Hygiene and Tropical Medicine) · doi:10.17037/pubs.03894558 · Cited by 1 · see citing works
Mickenautsch, S., & Yengopal, V. (2023) · Research Square · doi:10.21203/rs.3.rs-3298532/v1 · Cited by 1 · see citing works
Importance of including harms as well as benefits in all clinical trials
Tugwell, P., & Knottnerus, J. A. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.02.004
A statistical method for removing unbalanced trials with multiple covariates in meta-analysis
Attanasio, M., Aiello, F., & Tinè, F. (2023) · PLoS ONE · doi:10.1371/journal.pone.0295332
Selection bias risk in “low bias” (RoB 2) rated randomised control trials (Preprint)Preprint
Mickenautsch, S., & Yengopal, V. (2024) · Research Square · doi:10.21203/rs.3.rs-4567677/v1
Sources of Bias in Randomised Controlled Trials
Crombie, I. (2021) · doi:10.1002/9781394322183.ch2
van Bruggen, F. H., Zuidema, S. U., & Luijendijk, H. J. (2024) · Figshare · doi:10.6084/m9.figshare.26085140
Park, E.-G., & Hahn, S. (2022) · Figshare · doi:10.6084/m9.figshare.19923270
Le gaspillage évitable de la recherche dans les essais cliniques
Yordanov, Y. (2019) · theses.fr (ABES) · doi:10.70675/130e8831z862fz47f2z8000z2f04c8541788
Miles, A. (2015) · European Journal for Person Centered Healthcare · doi:10.5750/ejpch.v3i1.1009
- Authors
- Clark, L., Fairhurst, C., Hewitt, C. E., Birks, Y., Brabyn, S., Cockayne, S., Rodgers, S., Hicks, K., Hodgson, R., Littlewood, E., & Torgerson, D. J.
- Affiliations
- University of York (GB)
- Corresponding author
- Clark, Laura
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2014-09
- Volume
- 67
- Issue
- 9
- Pages
- 1016-1024
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1989811374
- Cited by
- 36
- References
- 36
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 8 · 2023: 4 · 2022: 3 · 2021: 3 · 2020: 1 · 2019: 3 · 2018: 3 · 2017: 2 · 2016: 1 · 2015: 2 · 2014: 1
- FWCI
- 1.95
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Keywords
- age heterogeneity, allocation concealment, meta-analysis, systematic reviews, trial validity, meta-regression
- MeSH terms
- Humans, Review Literature as Topic, Meta-Analysis as Topic, Bias, Likelihood Functions, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
