The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Issues in Comparisons Between Meta-analyses and Large Trials
Abstract
CONTEXT: The extent of concordance between meta-analyses and large trials on the same topic has been investigated with different protocols. Inconsistent conclusions created confusion regarding the validity of these major tools of clinical evidence. OBJECTIVE: To evaluate protocols comparing meta-analyses and large trials in order to understand if and why they disagree on the concordance of these 2 clinical research methods. DESIGN: Systematic comparison of protocol designs, study selection, definitions of agreement, analysis methods, and reported discrepancies between large trials and meta-analyses. RESULTS: More discrepancies were claimed when large trials were selected from influential journals (which may prefer trials disagreeing with prior evidence) than from already performed meta-analyses (which may target homogeneous trials) and when both primary and secondary (rather than only primary) end points were considered. Depending on how agreement was defined, kappa coefficients varied from 0.22 (low agreement) to 0.72 (excellent agreement). The correlation of treatment effects between large trials and meta-analyses varied from -0.12 to 0.76, but was more similar (0.50-0.76) when only primary end points were considered. When both the magnitude and uncertainty of treatment effects were considered, large trials disagreed with meta-analyses 10% to 23% of the time. Discrepancies were attributed to different disease risks, variable protocols, quality, and publication bias. CONCLUSIONS: Comparisons of large trials with meta-analyses may reach different conclusions depending on how trials and meta-analyses are selected and how end points and agreement are defined. Scrutiny of these 2 major research methods can enhance our appreciation of both for guiding medical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127
Quantitative Synthesis in Systematic Reviews
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-9-199711010-00008 · Cited by 2,517
Self-citationQUANTITATIVE METHODS IN THE REVIEW OF EPIDEMIOLOGIC LITERATURE1
GREENLAND, S. (1987) · Epidemiologic Reviews · doi:10.1093/oxfordjournals.epirev.a036298 · Cited by 1,644
Publication bias and clinical trials
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90155-3 · Cited by 715
Cited by
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
DiMatteo, M. R. (2004) · Medical Care · doi:10.1097/01.mlr.0000114908.90348.f9 · Cited by 2,169
Social support and patient adherence to medical treatment: a meta-analysis.
DiMatteo, M. R. (2004) · Health Psychology · doi:10.1037/0278-6133.23.2.207 · Cited by 2,107
Patient Adherence and Medical Treatment Outcomes: A Meta-Analysis
Robin DiMatteo, M., Giordani, P. J., Lepper, H. S., & Croghan, T. W. (2002) · Medical Care · doi:10.1097/00005650-200209000-00009 · Cited by 1,490
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Self-citationA comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286
Mind-Set Interventions Are a Scalable Treatment for Academic Underachievement
Paunesku, D., Walton, G. M., Romero, C., Smith, E. N., Yeager, D. S., & Dweck, C. S. (2015) · Psychological Science · doi:10.1177/0956797615571017 · Cited by 981
Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies
Ioannidis, J. P. A. (2001) · JAMA · doi:10.1001/jama.286.7.821 · Cited by 919
Self-citationDemystifying trial networks and network meta-analysis
Mills, E. J., Thorlund, K., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f2914 · Cited by 742
Self-citationNetwork meta-analysis: application and practice using Stata
Shim, S., Yoon, B.-H., Shin, I.-S., & Bae, J.-M. (2017) · Epidemiology and Health · doi:10.4178/epih.e2017047 · Cited by 656
Bias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636
Influence of trial sample size on treatment effect estimates: meta-epidemiological study
Dechartres, A., Trinquart, L., Boutron, I., & Ravaud, P. (2013) · BMJ · doi:10.1136/bmj.f2304 · Cited by 544
When and why defaults influence decisions: a meta-analysis of default effects
JACHIMOWICZ, J. M., DUNCAN, S., WEBER, E. U., & JOHNSON, E. J. (2019) · Behavioural Public Policy · doi:10.1017/bpp.2018.43 · Cited by 481
Health Beliefs, Disease Severity, and Patient Adherence: A Meta-Analysis
DiMatteo, M. R., Haskard, K. B., & Williams, S. L. (2007) · Medical Care · doi:10.1097/mlr.0b013e318032937e · Cited by 482
Misleading funnel plot for detection of bias in meta-analysis
Tang, J.-L., & Liu, J. L. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00204-8 · Cited by 420
Brown, K. F., Kroll, J. S., Hudson, M. J., Ramsay, M., Green, J., Long, S. J., Vincent, C. A., Fraser, G., & Sevdalis, N. (2010) · Vaccine · doi:10.1016/j.vaccine.2010.04.052 · Cited by 412
Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease
Puhan, M. A., Gimeno-Santos, E., Scharplatz, M., Troosters, T., Walters, E. H., & Steurer, J. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005305.pub3 · Cited by 379
Bero, L., Oostvogel, F., Bacchetti, P., & Lee, K. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040184 · Cited by 249
Association Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses
Dechartres, A., Altman, D. G., Trinquart, L., Boutron, I., & Ravaud, P. (2014) · JAMA · doi:10.1001/jama.2014.8166 · Cited by 233
Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies
Papanikolaou, P. N. (2006) · Canadian Medical Association Journal · doi:10.1503/cmaj.050873 · Cited by 229
Gallegos, A. M., Crean, H. F., Pigeon, W. R., & Heffner, K. L. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.10.004 · Cited by 181
Shanthanna, H., Gilron, I., Rajarathinam, M., AlAmri, R., Kamath, S., Thabane, L., Devereaux, P. J., & Bhandari, M. (2017) · PLOS Medicine · doi:10.1371/journal.pmed.1002369 · Cited by 177
Evaluating Meta-analyses in the General Surgical Literature: A Critical Appraisal
Dixon, E., Hameed, M., Sutherland, F., Cook, D. J., & Doig, C. (2005) · Annals of Surgery · doi:10.1097/01.sla.0000154258.30305.df · Cited by 129
A mega-trial investigation of goal setting, interest enhancement, and energy on procrastination.
Gröpel, P., & Steel, P. (2008) · Personality and Individual Differences · doi:10.1016/j.paid.2008.05.015 · Cited by 101
Effects of dietary weight loss on obstructive sleep apnea: a meta-analysis
Anandam, A., Akinnusi, M., Kufel, T., Porhomayon, J., & El-Solh, A. A. (2013) · Sleep and Breathing · doi:10.1007/s11325-012-0677-3 · Cited by 95
Primer: strengths and weaknesses of meta-analysis
Finckh, A., & Tramèr, M. R. (2008) · Nature Clinical Practice Rheumatology · doi:10.1038/ncprheum0732 · Cited by 62
Rosenberg, S. D., Swanson, J. W., Wolford, G. L., Osher, F. C., Swartz, M. S., Essock, S. M., Butterfield, M. I., Marsh, B. J., & the Five-Site Health and Risk Study Research Committee (2003) · Psychiatric Services · doi:10.1176/appi.ps.54.6.827 · Cited by 58
How to critically read ecological meta-analyses
Lortie, C. J., Stewart, G., Rothstein, H., & Lau, J. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1109 · Cited by 39
Haidich, A.-B., & Ioannidis, J. P. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-4 · Cited by 34
Self-citationVariation in results from randomized, controlled trials: stochastic or systematic?
Jane-wit, D., Horwitz, R. I., & Concato, J. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.02.010 · Cited by 33
Metadata
- Authors
- Ioannidis, J. P. A.
- Venue
- JAMA
- Publisher
- American Medical Association (AMA)
- Published
- 1998-04-08
- Volume
- 279
- Issue
- 14
- Pages
- 1089
- ISSN
- 0002-9955 (ISSN-L)
- 0098-7484
- OpenAlex
- W2024119733
- Cited by
- 253
- References
- 6
- FWCI
- 28.02
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
