Editors Can Lead Researchers to Confidence Intervals, but Can't Make Them Think: Statistical Reform Lessons From Medicine
The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Since the mid-1980s, confidence intervals (CIs) have been standard in medical journals. We sought lessons for psychology from medicine's experience with statistical reform by investigating two attempts by Kenneth Rothman to change statistical practices. We examined 594 American Journal of Public Health (AJPH) articles published between 1982 and 2000 and 110 Epidemiology articles published in 1990 and 2000. Rothman's editorial instruction to report CIs and not p values was largely effective: In AJPH, sole reliance on p values dropped from 63% to 5%, and CI reporting rose from 10% to 54%; Epidemiology showed even stronger compliance. However, compliance was superficial: Very few authors referred to CIs when discussing results. The results of our survey support what other research has indicated: Editorial policy alone is not a sufficient mechanism for statistical reform. Achieving substantial, desirable change will require further guidance regarding use and interpretation of CIs and appropriate effect size measures. Necessary steps will include studying researchers' understanding of CIs, improving education, and developing empirically justified recommendations for improved statistical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings.
Raudenbush, S. W., Hunter, J. E., & Schmidt, F. L. (1991) · Journal of the American Statistical Association · doi:10.2307/2289738 · Cited by 2,327
Null Hypothesis Testing: Problems, Prevalence, and an Alternative
Anderson, D. R., Burnham, K. P., & Thompson, W. L. (2000) · The Journal of Wildlife Management · doi:10.2307/3803199 · Cited by 1,930
Smithson, M. (2000) · SAGE Publications Ltd eBooks · doi:10.4135/9781446218525 · Cited by 1,871
Practical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · cited 1×
“1999). Kirk (1996) , for example, identified and explained over 40 different measures of effect size.”
BackgroundPublication Manual of the American Psychological Association: BOOK REVIEW
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,451
Inference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449 · cited 2×
“Williams, & Cumming, 2005; Cumming, Williams, & Fidler, 2004); • empirical and conceptual studies of how CIs can most effectively be presented and used to interpret research results (e.g., Cumming & Finch, 2005 ); • empirical studies of how to teach CIs, and the consequences of giving CIs a foundational role in statis…”
BackgroundSelf-citationSchmidt, F. L. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.115 · Cited by 830 · cited 1×
“Although some reformers have claimed that CIs are easier to teach and less frequently misinterpreted than NHST (e.g., Hunter & Schmidt, 1997; Schmidt, 1996 ), these claims are supported only by anecdotal evidence.”
BackgroundSchmidt, F. L. (1996) · Psychological Methods · doi:10.1037//1082-989x.1.2.115 · Cited by 783
What If There Were No Significance Tests?
(2013) · Psychology Press eBooks · doi:10.4324/9781315827353 · Cited by 667
Thompson, B. (2002) · Educational Researcher · doi:10.3102/0013189x031003025 · Cited by 538 · cited 1×
“Twenty journals in psychology and education now require effect size reporting ( Thompson, 2002 ).”
BackgroundCumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528
Self-citationThe Significance Test Controversy.
Carter, L. F., Morrison, D. E., & Henkel, R. E. (1971) · Social Forces · doi:10.2307/2576949 · Cited by 437
Researchers Misunderstand Confidence Intervals and Standard Error Bars.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.389 · Cited by 390
Self-citationAERA Editorial Policies regarding Statistical Significance Testing: Three Suggested Reforms
Thompson, B. (1996) · Educational Researcher · doi:10.2307/1176337 · Cited by 339
Rothman, J. (1978) · New England Journal of Medicine · doi:10.1056/nejm197812142992410 · Cited by 303
UNIFORM REQUIREMENTS FOR MANUSCRIPTS SUBMITTED TO BIOMEDICAL JOURNALS
Davidoff, F., Godlee, F., Hoey, J., Glass, R. M., Overbeke, J., Utiger, R. D., Nicholls, M. G., Horton, R. C., Nylenna, M., Højgaard, L., et al. (1988) · Indian Journal of Pharmacology · doi:10.4103/ijpha_2003355_330 · Cited by 303
Estimation of confidence limits for the cumulative probability of survival in life table analysis
Rothman, K. J. (1978) · Journal of Chronic Diseases · doi:10.1016/0021-9681(78)90043-7 · Cited by 282
THE FALLACY OF EMPLOYING STANDARDIZED REGRESSION COEFFICIENTS AND CORRELATIONS AS MEASURES OF EFFECT
GREENLAND, S., SCHLESSELMAN, J. J., & CRIQUI, M. H. (1986) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a114229 · Cited by 256
Thompson, B. (1996) · Educational Researcher · doi:10.3102/0013189x025002026 · Cited by 211
Educational and psychological measurement
(1982) · Serials Review · doi:10.1016/0098-7913(82)90053-3 · Cited by 186
Should Significance Tests be Banned? Introduction to a Special Section Exploring the Pros and Cons
Shrout, P. E. (1997) · Psychological Science · doi:10.1111/j.1467-9280.1997.tb00533.x · Cited by 180 · cited 2×
“If you do not agree with my standards (concerning the inappropriateness of significance tests) you should feel free to argue the point, or simply ignore what you may consider to be my misguided view, by publishing elsewhere ( Shrout, 1997, p.1, citing Fleiss, 1986, citing Rothman).”
BackgroundReplication and Researchers' Understanding of Confidence Intervals and Standard Error Bars
Cumming, G., Williams, J., & Fidler, F. (2004) · Understanding Statistics · doi:10.1207/s15328031us0304_5 · Cited by 172 · cited 1×
“Williams, & Cumming, 2005; Cumming, Williams, & Fidler, 2004 ); • empirical and conceptual studies of how CIs can most effectively be presented and used to interpret research results (e.g., Cumming & Finch, 2005 ); • empirical studies of how to teach CIs, and the consequences of giving CIs a foundational role in stati…”
BackgroundSelf-citationReporting Practices and APA Editorial Policies Regarding Statistical Significance and Effect Size
Vacha-Haase, T., Nilsson, J. E., Reetz, D. R., Lance, T. S., & Thompson, B. (2000) · Theory & Psychology · doi:10.1177/0959354300103006 · Cited by 126 · cited 1×
“Vacha-Haase et al (2000) reviewed 10 independent studies of reporting practices in 23 psychology and education journals.”
BackgroundFidler, F. (2002) · Educational and Psychological Measurement · doi:10.1177/001316402236876 · Cited by 102
Self-citationStatistical Techniques Employed in AERJ and JCP Articles from 1988 to 1997: A Methodological Review
Kieffer, K. M., Reese, R. J., & Thompson, B. (2001) · The Journal of Experimental Education · doi:10.1080/00220970109599489 · Cited by 91 · cited 1×
“While it is too early to assess the impact of the new APA recommendation, previous decades of encouraging researchers to report CIs have had little effect . They remain relatively little-used (Finch, Cumming, & Thomason, 2001; Kieffer, Reese, & Thompson, 2001 ).”
BackgroundUniform requirements for manuscripts submitted to biomedical journals
International Committee of Medical Journal Editors (1988) · BMJ · doi:10.1136/bmj.296.6619.401 · Cited by 87
Reporting of Statistical Inference in the Journal of Applied Psychology: Little Evidence of Reform
Finch, S. J., Cumming, G., & Thomason, N. R. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971167 · Cited by 86 · cited 2×
“While it is too early to assess the impact of the new APA recommendation, previous decades of encouraging researchers to report CIs have had little effect . They remain relatively little-used ( Finch, Cumming, & Thomason, 2001; Kieffer, Reese, & Thompson, 2001).”
BackgroundSelf-citationSignificance tests have a role in epidemiologic research: reactions to A. M. Walker.
Fleiss, J. L. (1986) · American Journal of Public Health · doi:10.2105/ajph.76.5.559 · Cited by 78
Towards estimation and confidence intervals.
Langman, M. J. S. (1986) · BMJ · doi:10.1136/bmj.292.6522.716 · Cited by 76 · cited 1×
“By 1986, the British Medical Journal (BMJ) had a policy encouraging CIs ( Langman, 1986 ).”
Background(1993) · Memory & Cognition · doi:10.3758/bf03211158 · Cited by 75
Finch, S., Cumming, G., & Thomason, N. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401612001 · Cited by 52
Self-citationReform of statistical inference in psychology: The case ofMemory & Cognition
Finch, S. J., Cumming, G., Williams, J., Palmer, L., Griffith, E., Alders, C., Anderson, J. L., & Goodman, O. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195577 · Cited by 43 · cited 2×
“Finch, Cumming, Williams, et al (2004) studied the Loftus reform attempt.”
BackgroundSelf-citationPast and Future American Psychological Association Guidelines for Statistical Practice
Finch, S., Thomason, N., & Cumming, G. (2002) · Theory & Psychology · doi:10.1177/0959354302126005 · Cited by 40
Self-citationStatistical significance testing in the American Journal of Epidemiology, 1970-1990.
Savitz, D., Tolo, K.-A., & Poole, C. (1994) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116944 · Cited by 27
Rennie, D. (1978) · New England Journal of Medicine · doi:10.1056/nejm197810122991509 · Cited by 22
Cited by
A working guide to boosted regression trees
Elith, J., Leathwick, J. R., & Hastie, T. (2008) · Journal of Animal Ecology · doi:10.1111/j.1365-2656.2008.01390.x · Cited by 6,484
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Self-citationEllis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
Recommended effect size statistics for repeated measures designs
Bakeman, R. (2005) · Behavior Research Methods · doi:10.3758/bf03192707 · Cited by 1,809
Bayesian Versus Orthodox Statistics: Which Side Are You On?
Dienes, Z. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406920 · Cited by 938
Metadata
- Authors
- Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J.
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2004-02
- Volume
- 15
- Issue
- 2
- Pages
- 119-126
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2144239896
- Cited by
- 244
- References
- 36
- FWCI
- 18.74
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.