Inference by Eye: Confidence Intervals and How to Read Pictures of Data.
Abstract
Wider use in psychology of confidence intervals (CIs), especially as error bars in figures, is a desirable development. However, psychologists seldom use CIs and may not understand them well. The authors discuss the interpretation of figures with error bars and analyze the relationship between CIs and statistical significance testing. They propose 7 rules of eye to guide the inferential use of figures with error bars. These include general principles: Seek bars that relate directly to effects of interest, be sensitive to experimental design, and interpret the intervals. They also include guidelines for inferential interpretation of the overlap of CIs on independent group means. Wider use of interval estimation in psychology has the potential to improve research communication substantially.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657 · see citing works
Hunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023 · see citing works
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791 · see citing works
Using confidence intervals in within-subject designs
Loftus, G. R., & Masson, M. E. J. (1994) · Psychonomic Bulletin & Review · doi:10.3758/bf03210951 · Cited by 2,448 · see citing works
Mcgill, R., Tukey, J. W., & Larsen, W. A. (1978) · The American Statistician · doi:10.1080/00031305.1978.10479236 · Cited by 2,250 · see citing works
Statistics for the social sciences
Hays, W. L. (1973) · Medical Entomology and Zoology · Cited by 2,100 · see citing works
Edgington, E. S., Edgington, E., & Onghena, P. (1964) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess2169 · Cited by 2,034 · see citing works
Confidence intervals rather than P values: estimation rather than hypothesis testing.
Gardner, M. J., & Altman, D. G. (1986) · BMJ · doi:10.1136/bmj.292.6522.746 · Cited by 2,014 · see citing works
Smithson, M. J. (2000) · SAGE Publications Ltd eBooks · doi:10.4135/9781446218525 · Cited by 1,871 · see citing works
Practical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · see citing works
THE SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO MEANS WHEN THE POPULATION VARIANCES ARE UNEQUAL
Welch, B. L. (1938) · Biometrika · doi:10.1093/biomet/29.3-4.350 · Cited by 1,506 · see citing works
Cleveland, W. S. (1985) · Medical Entomology and Zoology · Cited by 1,367 · see citing works
Statistics with Confidence: Confidence Intervals and Statistical Guidelines.
JOLLIFFE, F. R., Gardner, M. J., & Altman, D. G. (1990) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2982808 · Cited by 1,319 · see citing works
On judging the significance of differences by examining the overlap between confidence intervals.
Schenker, N., & Gentleman, J. F. (2001) · The American Statistician · doi:10.1198/000313001317097960 · Cited by 1,218 · see citing works
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090 · see citing works
Beyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034 · see citing works
Schmidt, F. L. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.115 · Cited by 1,031 · see citing works
Using confidence intervals for graphically based data interpretation.
Masson, M. E., & Loftus, G. (2003) · Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale · doi:10.1037/h0087426 · Cited by 819 · see citing works
What If There Were No Significance Tests?
(2013) · Psychology Press eBooks · doi:10.4324/9781315827353 · Cited by 674 · see citing works
McGill, R., Tukey, J. W., & Larsen, W. A. (1978) · The American Statistician · doi:10.2307/2683468 · Cited by 666 · see citing works
STEVENS' HANDBOOK OF EXPERIMENTAL PSYCHOLOGY
Pashler, H. (2002) · Wiley eBooks · doi:10.1002/0471214426 · Cited by 576 · see citing works
Steiger, J. H. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.164 · Cited by 565 · see citing works
Thompson, B. (2002) · Educational Researcher · doi:10.3102/0013189x031003025 · Cited by 546 · see citing works
What If There Were No Significance Tests?
(2016) · Routledge eBooks · doi:10.4324/9781315629049 · Cited by 543 · see citing works
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528 · see citing works
Self-citationNormative comparisons for the evaluation of clinical significance.
Kendall, P. C., Marrs, A. L., Nath, S. R., & Sheldrick, R. C. (1999) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.67.3.285 · Cited by 468 · see citing works
Tryon, W. (2001) · Psychological methods · doi:10.1037/1082-989x.6.4.371 · Cited by 423 · see citing works
Applying Contemporary Statistical Techniques
(2003) · Elsevier eBooks · doi:10.1016/b978-0-12-751541-0.x5021-4 · Cited by 421 · see citing works
Statistical Inference: A Commentary for the Social and Behavioural Sciences
Oakes, M. W. (1986) · Medical Entomology and Zoology · Cited by 392 · see citing works
Researchers Misunderstand Confidence Intervals and Standard Error Bars.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.389 · Cited by 390 · see citing works
Self-citationThe Significance of the Difference Between Two Means when the Population Variances are Unequal
Welch, B. L. (1938) · Biometrika · doi:10.2307/2332010 · Cited by 374 · see citing works
(1995) · Choice Reviews Online · doi:10.5860/choice.32-6272 · Cited by 310 · see citing works
How many discoveries have been lost by ignoring modern statistical methods?
Wilcox, R. R. (1998) · American Psychologist · doi:10.1037/0003-066x.53.3.300 · Cited by 295 · see citing works
The Hospital Admission Decision for Patients With Community-Acquired Pneumonia
Fine, M. J. (1997) · Archives of Internal Medicine · doi:10.1001/archinte.1997.00440220040006 · Cited by 291 · see citing works
Smithson, M. J. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971392 · Cited by 275 · see citing works
Edgington, E. S. (1998) · Technometrics · doi:10.1080/00401706.1998.10485213 · Cited by 245 · see citing works
Applying Contemporary Statistical Techniques
Nembhard, H. B. (2004) · Technometrics · doi:10.1198/tech.2004.s233 · Cited by 187 · see citing works
Statistics with Confidence. Confidence Intervals and Statistical Guidelines
Newcombe, R. G. (1990) · Journal of Medical Genetics · doi:10.1136/jmg.27.1.71 · Cited by 181 · see citing works
Replication and Researchers' Understanding of Confidence Intervals and Standard Error Bars
Cumming, G., Williams, J., & Fidler, F. M. (2004) · Understanding Statistics · doi:10.1207/s15328031us0304_5 · Cited by 172 · see citing works
Self-citationThe Null Hypothesis Testing Controversy in Psychology
Krantz, D. H. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473888 · Cited by 158 · see citing works
If we're so different, why do we keep overlapping? When 1 plus 1 doesn't make 2.
Wolfe, R. S., & Hanley, J. (2002) · PubMed · Cited by 122 · see citing works
Fidler, F. (2002) · Educational and Psychological Measurement · doi:10.1177/001316402236876 · Cited by 111 · see citing works
Computing Correct Confidence Intervals for ANOVA Fixed- and Random-Effects Effect Sizes
Fidler, F. M., & Thompson, B. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614003 · Cited by 107 · see citing works
The earth is round (p < .05): Rejoinder.
Cohen, J. (1995) · American Psychologist · doi:10.1037/0003-066x.50.12.1103 · Cited by 100 · see citing works
Confidence Intervals for Effect Sizes in Analysis of Variance
Bird, K. (2002) · Educational and Psychological Measurement · doi:10.1177/0013164402062002001 · Cited by 86 · see citing works
Kendall, P. C. (1999) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.67.3.283 · Cited by 86 · see citing works
On the communication of information by displays of standard errors and confidence intervals
Estes, W. K. (1997) · Psychonomic Bulletin & Review · doi:10.3758/bf03210790 · Cited by 73 · see citing works
Basic statistics and the inconsistency of multiple comparison procedures.
Saville, D. J. (2003) · Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale · doi:10.1037/h0087423 · Cited by 65 · see citing works
Analysis, Interpretation, and Visual Presentation of Experimental Data
Loftus, G. R. (2002) · doi:10.1002/0471214426.pas0409 · Cited by 56 · see citing works
Fine, M. J. (1997) · Archives of Internal Medicine · doi:10.1001/archinte.157.1.36 · Cited by 52 · see citing works
Past and Future American Psychological Association Guidelines for Statistical Practice
Finch, S., Thomason, N., & Cumming, G. (2002) · Theory & Psychology · doi:10.1177/0959354302126005 · Cited by 44 · see citing works
Self-citationReform of statistical inference in psychology: The case ofMemory & Cognition
Finch, S. J., Cumming, G., Williams, J., Palmer, L., Griffith, E., Alders, C., Anderson, J. L., & Goodman, O. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195577 · Cited by 39 · see citing works
Self-citationDisplaying your findings : a practical guide for creating figures, posters, and presentations
Nicol, A. A. M., & Pexman, P. M. (2003) · Cited by 35 · see citing works
The Null Hypothesis Testing Controversy in Psychology
Krantz, D. H. (1999) · Journal of the American Statistical Association · doi:10.2307/2669949 · Cited by 34 · see citing works
Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (2016) · doi:10.4324/9781315629049-20 · Cited by 22 · see citing works
Vaux, D. L. (2004) · Nature · doi:10.1038/428799c · Cited by 16 · see citing works
Cumming, D. (2004) · doi:10.1049/ic:20040588 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303 · see citing works
“For excellent examples, see ESCI (Cumming and Finch, 2005), confidence interval software by Smithson (2001), and G ∗ Power (Faul et al., 2009).”
BackgroundMethodInfluentialDurlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011) · Child Development · doi:10.1111/j.1467-8624.2010.01564.x · Cited by 8,746 · see citing works
“Multiple papers evaluating the same intervention but containing different outcome data at post or follow-up for the same cohort were combined into a single study.”
BackgroundMethodseaborn: statistical data visualization
Waskom, M. (2021) · Journal of Open Source Software · doi:10.21105/joss.03021 · Cited by 7,310 · see citing works
“The confidence intervals are computed with a bootstrap algorithm, allowing them to generalize over many different statistics, and the default level allows the user to perform “inference by eye” (Cumming & Finch, 2005).”
MethodThe New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200 · see citing works
“The best way to think of SE bars on a mean is usually to double the whole length—so the bars extend 2 SE above and 2 SE below the mean—and interpret them as being, approximately, the 95% CI, as Figure 2 illustrates (Cumming & Finch, 2005).”
Self-citationEllis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461 · see citing works
Hulsheger, U. R., Anderson, N. R., & Salgado, J. F. (2009) · Journal of Applied Psychology · doi:10.1037/a0015978 · Cited by 1,508 · see citing works
“Thus, even if confidence intervals overlap, values can still be significantly different from one another (for interpretation of confidence intervals, see Cumming & Finch, 2005).”
BackgroundSchmider, E., Ziegler, M. D., Danay, E., Beyer, L. M., & Bühner, M. (2010) · Methodology · doi:10.1027/1614-2241/a000016 · Cited by 1,300 · see citing works
Hanson, R., & Morton-Bourgon, K. E. (2009) · Psychological Assessment · doi:10.1037/a0014421 · Cited by 1,034 · see citing works
“Substantive interpretations can be based on values anywhere in the interval (Cumming & Finch, 2005).”
BackgroundStudent Motivation and Associated Outcomes: A Meta-Analysis From Self-Determination Theory
Howard, J. L., Bureau, J., Guay, F., Chong, J. X. Y., & Ryan, R. M. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966789 · Cited by 1,001 · see citing works
“To compare the effect sizes between regulation types, we relied on the confidence interval method proposed by Cumming and Finch (2005).”
MethodResultMann, R. E., Hanson, R. K., & Thornton, D. (2010) · Sexual Abuse · doi:10.1177/1079063210366039 · Cited by 960 · see citing works
“When the observations are uncorrelated, two values can be considered to differ at the p .01 level if their CIs do not overlap (Cumming & Finch, 2005).”
BackgroundError bars in experimental biology
Cumming, G., Fidler, F., & Vaux, D. L. (2007) · The Journal of Cell Biology · doi:10.1083/jcb.200611141 · Cited by 940 · see citing works
“, from n knock-out mice and n wild-type mice, you can compare the SE bars or the 95% CIs on the two means (6).”
BackgroundSelf-citationThe psychological effects of meditation: a meta-analysis
Sedlmeier, P., Eberth, J., Schwarz, M., Zimmermann, D., Haarig, F., Jaeger, S., & Kunze, S. (2012) · Psychological Bulletin · doi:10.1037/a0028168 · Cited by 910 · see citing works
“Cumming and Finch (2005) showed that if 95% CIs for independent means do not overlap, this is equivalent to a significant difference between the two respective means at .01 (see also Cumming, 2011).”
BackgroundEvaluative conditioning in humans: A meta-analysis.
Hofmann, W., Houwer, J. D., Perugini, M., Baeyens, F., & Crombez, G. (2010) · Psychological Bulletin · doi:10.1037/a0018916 · Cited by 926 · see citing works
Moffitt, T. E., Caspi, A., Taylor, A., Kokaua, J., Milne, B. J., Polanczyk, G., & Poulton, R. (2010) · Psychological Medicine · doi:10.1017/s0033291709991036 · Cited by 938 · see citing works
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 818 · see citing works
“If the key parameter (e.g. a correlation) of the replication study falls within the CI of the initial study (or if the two CIs overlap substantially, Cumming & Finch, 2005), one can argue more strongly that the result is replicated.”
BackgroundThe Effects of Mindfulness Meditation: A Meta-Analysis
Eberth, J., & Sedlmeier, P. (2012) · Mindfulness · doi:10.1007/s12671-012-0101-x · Cited by 791 · see citing works
“Please note that inferences on significance are somewhat limited if the sizes of the two CIs to be compared differ by more than a factor of 2 (Cumming and Finch 2005).”
BackgroundToward using confidence intervals to compare correlations.
Zou, G. (2007) · Psychological Methods · doi:10.1037/1082-989x.12.4.399 · Cited by 744 · see citing works
Madigan, D. J., & Kim, L. E. (2021) · Teaching and Teacher Education · doi:10.1016/j.tate.2021.103425 · Cited by 798 · see citing works
Inference by eye: Reading the overlap of independent confidence intervals
Cumming, G. (2008) · Statistics in Medicine · doi:10.1002/sim.3471 · Cited by 673 · see citing works
“An informal adjustment towards conservatism, to protect against inflated Type 1 error rates [ 5 ], may be to use p 5 ] recommended a range of ways to think about CIs without invoking p values.”
BackgroundMethodInfluentialSelf-citationReaction time variability in ADHD: A meta-analytic review of 319 studies
Kofler, M. J., Rapport, M. D., Sarver, D. E., Raiker, J. S., Orban, S. A., Friedman, L. M., & Kolomeyer, E. G. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.06.001 · Cited by 648 · see citing works
MorePower 6.0 for ANOVA with relational confidence intervals and Bayesian analysis
Campbell, J. I. D., & Thompson, V. A. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0186-0 · Cited by 661 · see citing works
“For example graphs and interpre-tative strategies, see Cumming and Finch (2005), Jarmasz and Hollands (2009), and Masson and Loftus (2003).”
MethodThe fallacy of placing confidence in confidence intervals
Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E.-J. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0947-8 · Cited by 577 · see citing works
“…data by making dichotomous statements (Neyman 1937, 1941), eventually becoming a method that many believe is the best way to reason from data (e.g. Cumming & Finch, 2005; Cumming & Fidler, 2009) and a way to avoid dichotomous statements (e.g. Cumming, 2014; Hoekstra, Finch, Kiers, & Johnson,…”
MethodA Meta-Analysis of the Impact of Service-Learning on Students
Celio, C. I., Durlak, J., & Dymnicki, A. (2011) · Journal of Experiential Education · doi:10.1177/105382591103400205 · Cited by 603 · see citing works
“Cumming and Finch’s (2005) procedure was used to detect significant differences between group means by examining the extent of any overlap in the two groups’ mean confidence intervals.”
MethodPenner, I., Raselli, C., Stöcklin, M., Opwis, K., Kappos, L., & Calabrese, P. (2009) · Multiple Sclerosis Journal · doi:10.1177/1352458509348519 · Cited by 595 · see citing works
PlotsOfData—A web app for visualizing data together with their summaries
Postma, M., & Goedhart, J. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000202 · Cited by 554 · see citing works
Risks and decisions for conservation and environmental management
Burgman, M. A. (2005) · Choice Reviews Online · doi:10.5860/choice.43-2184 · Cited by 524 · see citing works
Rapport, M. D., Orban, S. A., Kofler, M. J., & Friedman, L. M. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.08.005 · Cited by 517 · see citing works
Risks and Decisions for Conservation and Environmental Management
Burgman, M. A. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511614279 · Cited by 498 · see citing works
Mayfield, M. M., Bonser, S. P., Morgan, J. W., Aubin, I., McNamara, S., & Vesk, P. A. (2010) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2010.00532.x · Cited by 464 · see citing works
Small is beautiful: In defense of the small-N design
Smith, P. L., & Little, D. R. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-1451-8 · Cited by 466 · see citing works
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451 · see citing works
“Many advocates of statistical reform, including Kline (2004, Chapter 3) and Cumming and Finch (2005), have presented reasons why psychologists should use CIs wherever possible.”
BackgroundSelf-citationHadlaczky, G., Hökby, S., Mkrtchian, A., Carli, V., & Wasserman, D. (2014) · International Review of Psychiatry · doi:10.3109/09540261.2014.924910 · Cited by 433 · see citing works
“IF in the ‘ knowledge ’ outcome was the only moderator variable for which there was no observed CI overlap between conditions and thus a signifi cant difference (Cumming & Finch, 2005).”
BackgroundThe Prediction of Violence in Adult Offenders
Campbell, M. A., French, S. A., & Gendreau, P. (2009) · Criminal Justice and Behavior · doi:10.1177/0093854809333610 · Cited by 432 · see citing works
“When two CIs overlapped by no more than one quarter of the average length of the two intervals, then these mean effects were also interpreted as representing two different population parameters and were statistically different at approximately p ≤ .05 (Cumming & Finch, 2005).”
BackgroundA Meta-analysis of the Impact of Service-Learning on Students
Celio, C. I., Durlak, J. A., & Dymnicki, A. B. (2011) · Journal of Experiential Education · doi:10.5193/jee34.2.164 · Cited by 430 · see citing works
Using reversed items in Likert scales: A questionable practice
Suárez‐Álvarez, J., Pedrosa, I., Fernández, L. M. L., Garcı́a-Cueto, E., Cuesta, M., & Muñiz, J. (2018) · Psicothema · doi:10.7334/psicothema2018.33 · Cited by 435 · see citing works
“The standard error of the estimate was .10 (CI = ± .196), confi dence intervals of the items did not overlap in 14 out of the 20 items when comparing regular and combined forms, so statistically signifi cant differences were found in these 14 items (α = .05; Cumming & Finch, 2006).”
Arthur, W., Bell, S. T., Villado, A. J., & Doverspike, D. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.4.786 · Cited by 426 · see citing works
“The confidence interval reflects the extent to which sampling error remains in the effect size estimate (Whitener, 1990) and suggests the range of plausible values for the true effect size (Cumming & Finch, 2005).”
BackgroundBureau, J. S., Howard, J. L., Chong, J. X. Y., & Guay, F. (2022) · Review of Educational Research · doi:10.3102/00346543211042426 · Cited by 414 · see citing works
“In these analyses, similar estimated effects and overlapping confidence interval would indicate that published and unpublished samples are similar (Cumming & Finch, 2005) and therefore do not present evidence for publication bias.”
BackgroundResultComputation of measures of effect size for neuroscience data sets
Hentschke, H., & Stüttgen, M. C. (2011) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07902.x · Cited by 405 · see citing works
“The misuse of NHST has perhaps been most hotly debated in psychology (Loftus, 1996; Cumming & Finch, 2005).”
BackgroundFunctional neuroanatomy of aversion and its anticipation
Nitschke, J. B., Sarinopoulos, I., Mackiewicz, K. L., Schaefer, H. S., & Davidson, R. J. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2005.06.068 · Cited by 398 · see citing works
“More recently, LeDoux (2002) described aversive stimuli and warning signals that precede them as being served by a single brain system (pp.”
BackgroundInfluentialVan den Broeck, A., Howard, J. L., Van Vaerenbergh, Y., Leroy, H., & Gagné, M. (2021) · Organizational Psychology Review · doi:10.1177/20413866211006173 · Cited by 409 · see citing works
“In accordance with Cumming and Finch (2005), non-overlapping CIs indicated differences bet-ween values at a probability approximately equal to .01, and CIs which overlapped less than 50 % were considered indicative of differences in values of approximately p .05.”
InfluentialResearchers Misunderstand Confidence Intervals and Standard Error Bars.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.389 · Cited by 390 · see citing works
“In fact, when sample sizes are similar and not small, and CI widths are similar, then if 95% CIs on independent means just touch, the two-tail p value is about .006 ( Cumming & Finch, 2005; Payton, Greenstone, & Schenker, 2003), not .05 as many believe.”
BackgroundMethodInfluentialSelf-citationBayesian Estimation and Inference
Zyphur, M. J., & Oswald, F. L. (2013) · Journal of Management · doi:10.1177/0149206313501200 · Cited by 392 · see citing works
“Based on credibility intervals, this approach is stronger than the many recommendations for using confidence intervals (e.g., American Psychological Association, 2010; Cumming & Finch, 2005).”
MethodEffect sizes, confidence intervals, and confidence intervals for effect sizes
Thompson, B. (2007) · Psychology in the Schools · doi:10.1002/pits.20234 · Cited by 392 · see citing works
The Flynn effect: A meta-analysis.
Trahan, L. H., Stuebing, K. K., Fletcher, J. M., & Hiscock, M. (2014) · Psychological Bulletin · doi:10.1037/a0037173 · Cited by 364 · see citing works
Dynamic Capability Building in Service Value Networks for Achieving Service Innovation
Agarwal, R., & Selen, W. J. (2009) · Decision Sciences · doi:10.1111/j.1540-5915.2009.00236.x · Cited by 375 · see citing works
“A first management implication of our study relates to the multidimensional notion of innovation in services.”
BackgroundResultUnderstanding the Results of Multiple Linear Regression
Nimon, K., & Oswald, F. L. (2013) · Organizational Research Methods · doi:10.1177/1094428113493929 · Cited by 352 · see citing works
“Our software program also graphs CIs, keeping in mind that the statistical differences between statistics may still be significant even when their respective CIs overlap (Cumming & Finch, 2005).”
MethodReducing Overconfidence in the Interval Judgments of Experts
Speirs‐Bridge, A., Fidler, F. M., McBride, M. F., Flander, L. B., Cumming, G., & Burgman, M. A. (2009) · Risk Analysis · doi:10.1111/j.1539-6924.2009.01337.x · Cited by 351 · see citing works
“Cumming and Finch(16) list four advantages of using CIs: they give point interval estimates in easily comprehensible units; p values can be easily estimated using CIs, with 14 overlap of 95% CI width being approximately equivalent to p<.05 (for independent groups); CIs help combine evidence from multiple studies, thus…”
BackgroundSelf-citationLowenkamp, C. T., Latessa, E. J., & Smith, P. (2006) · Criminology & Public Policy · doi:10.1111/j.1745-9133.2006.00388.x · Cited by 351 · see citing works
“For a discussion of the different ways the CI can be interpreted, see Cumming and Finch (2005). It should be noted that the CI can be interpreted as a significance test; that is, a CI that contains 0 is not statistically significant.”
BackgroundIdeology, Motivated Reasoning, and Cognitive Reflection: An Experimental Study
Kahan, D. M. (2012) · doi:10.2139/ssrn.2182588 · Cited by 331 · see citing works
Sun, S.-Y., Pan, W., & Wang, L. (2010) · Journal of Educational Psychology · doi:10.1037/a0019507 · Cited by 329 · see citing works
“…placed in the context of additional information, such as confidence intervals and effect size (AERA, 2006; Ander- son et al., 2000; APA, 2001, 2010; Cumming & Finch, 2002; Fan, 2001; Kirk, 1996, 2001; McLean & Ernest, 1998; Snyder & Lawson, 1993; Thompson, 1996, 1997, 2002b; Vacha-Haase &…”
A meta-analysis of universal mental health prevention programs for higher education students
Conley, C. S., Durlak, J. A., & Kirsch, A. C. (2015) · Prevention Science · doi:10.1007/s11121-015-0543-1 · Cited by 340 · see citing works
“Effect Size Comparisons We followed the guidelines of Cumming and Finch (2005), wherein mean ESs whose CIs do not include zero are considered statistically significant, pairs of CIs with no overlap are statistically different at the p<.”
MethodConfigural processing in autism and its relationship to face processing
Behrmann, M., Avidan, G., Leonard, G. L., Kimchi, R., Luna, B., Humphreys, K., & Minshew, N. (2006) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2005.04.002 · Cited by 334 · see citing works
“…the RTs of each individual and ascertained whether the individual data fall outside of the 95% confidence interval of the control subjects, following the recommendations for single case study analysis (Crawford & Garthwaite, 2004) and other statistical recommendations (Cumming & Finch, 2005).”
MethodTesting a continuum structure of self-determined motivation: A meta-analysis.
Howard, J. L., Gagné, M., & Bureau, J. S. (2017) · Psychological Bulletin · doi:10.1037/bul0000125 · Cited by 447 · see citing works
Fair Statistical Communication in HCI
Dragicevic, P. (2016) · Human-computer interaction series · doi:10.1007/978-3-319-26633-6_13 · Cited by 339 · see citing works
“If needed, a p-value can always be roughly inferred from a confidence interval by looking at how far it is from zero (Cumming and Finch, 2005; Cumming, 2012, pp.106–108).”
BackgroundFinal Collapse of the Neyman-Pearson Decision Theoretic Framework and Rise of the neoFisherian
Hurlbert, S. H., & Lombardi, C. M. (2009) · Annales Zoologici Fennici · doi:10.5735/086.046.0501 · Cited by 277 · see citing works
The Supply Chain Position Paradox: Green Practices and Firm Performance
Schmidt, C. G., Foerstl, K. D., & Schaltenbrand, B. (2016) · Journal of Supply Chain Management · doi:10.1111/jscm.12113 · Cited by 286 · see citing works
Lying Takes Time: A Meta-Analysis on Reaction Time Measures of Deception
Suchotzki, K., Verschuere, B., van Bockstaele, B., Ben-Shakhar, G., & Crombez, G. (2017) · Psychological bulletin · doi:10.1037/bul0000087 · Cited by 242 · see citing works
The development of children’s environmental attitude and behavior
Otto, S., Evans, G. W., Moon, M. J., & Kaiser, F. G. (2019) · Global Environmental Change · doi:10.1016/j.gloenvcha.2019.101947 · Cited by 278 · see citing works
Discrimination through Optimization
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359301 · Cited by 268 · see citing works
“However, the converse is not true: overlapping con dence intervals do not necessarily mean that the means are not di erent (see Figure 4 in [17] for explanation).”
BackgroundError Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error
Correll, M., & Gleicher, M. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346298 · Cited by 258 · see citing works
Wadsworth, M. E., & Achenbach, T. M. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.6.1146 · Cited by 271 · see citing works
Wigelsworth, M., Lendrum, A., Oldfield, J., Scott, A., ten Bokkel, I., Tate, K., & Emery, C. (2016) · Cambridge Journal of Education · doi:10.1080/0305764x.2016.1195791 · Cited by 263 · see citing works
“As described by Cumming and Finch (2005), this approach does not require a specialised understanding of meta-analytic techniques, and instead assesses how closely related two confidence intervals are, such as those presented in Figures 1–3 (see ‘results’).”
BackgroundQuality of Care of People With Type 2 Diabetes in Eight European Countries
Stone, M. A., Charpentier, G., Doggen, K., Kuß, O., Lindblad, U., Kellner, C., Nolan, J. J., Pazderska, A., Rutten, G. E. H. M., Trento, M., et al. (2013) · Diabetes Care · doi:10.2337/dc12-1759 · Cited by 258 · see citing works
Krauss, S., Baumert, J., & Blum, W. (2008) · ZDM · doi:10.1007/s11858-008-0141-9 · Cited by 293 · see citing works
“The error bars represent 95% confidence intervals (according to Cumming and Finch, 2005, two samples differ significantly at, for example, p \ 0.01 if the corresponding intervals do not overlap).”
BackgroundInfluentialUndergraduate Research Experiences Broaden Diversity in the Scientific Workforce
Hernandez, P. R., Woodcock, A., Estrada, M., & Schultz, P. W. (2017) · BioScience · doi:10.1093/biosci/bix163 · Cited by 250 · see citing works
Can 14,737 women be wrong? A meta‐analysis of the LSI‐R and recidivism for female offenders*
Smith, P., Cullen, F. T., & Latessa, E. J. (2009) · Criminology & Public Policy · doi:10.1111/j.1745-9133.2009.00551.x · Cited by 247 · see citing works
Báez, S., Flichtentrei, D., Prats, M., Mastandueno, R., García, A. M., Cetkovich, M., & Ibáñez, A. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179336 · Cited by 246 · see citing works
Improving the Dependability of Research in Personality and Social Psychology
Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2013) · Personality and Social Psychology Review · doi:10.1177/1088868313507536 · Cited by 254 · see citing works
“As was mentioned earlier, when effects within a study are theoretically expected to be different from each other, examination of the CIs illustrates whether this expectation was met (Cumming & Finch, 2005).”
BackgroundEditors Can Lead Researchers to Confidence Intervals, but Can't Make Them Think
Fidler, F. M., Thomason, N. R., Cumming, G., Finch, S. J., & Leeman, J. (2004) · Psychological Science · doi:10.1111/j.0963-7214.2004.01502008.x · Cited by 244 · see citing works
“Williams, & Cumming, 2005; Cumming, Williams, & Fidler, 2004); • empirical and conceptual studies of how CIs can most effectively be presented and used to interpret research results (e.g., Cumming & Finch, 2005 ); • empirical studies of how to teach CIs, and the consequences of giving CIs a foundational role in statis…”
BackgroundSelf-citationMeta-analysis in Second Language Research: Choices and Challenges
Oswald, F. L., & Plonsky, L. (2010) · Annual Review of Applied Linguistics · doi:10.1017/s0267190510000115 · Cited by 237 · see citing works
“For this latter situation, there are some rules of thumb for interpreting overlapping CIs appropriately (Cumming & Finch, 2005), or one may conduct a formal statistical test of the difference.”
BackgroundWorking Memory Deficits and Social Problems in Children with ADHD
Kofler, M. J., Rapport, M. D., Bolden, J., Sarver, D. E., Raiker, J. S., & Alderson, R. M. (2011) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-011-9492-8 · Cited by 227 · see citing works
“This difference is compared to the pooled SE across variables, with gaps equal to or larger than the pooled SE indicating significant differences between models (Cumming and Finch 2005, rule 7).”
MethodLamb, S. H., & Kwok, K. C. S. (2015) · Applied Ergonomics · doi:10.1016/j.apergo.2015.07.010 · Cited by 228 · see citing works
Calculating and graphing within-subject confidence intervals for ANOVA
Baguley, T. S. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0123-7 · Cited by 217 · see citing works
“Cumming and Finch (2005) explored some of these barriers, providing guidance on how to calculate, report, and interpret CIs (with emphasis on the graphical presentation of means in a two-independent-group design).”
BackgroundMethodInfluentialAntipsychotic-Induced Extrapyramidal Side Effects in Bipolar Disorder and Schizophrenia
Gao, K., Kemp, D. E., Ganocy, S. J., Gajwani, P., Xia, G., & Calabrese, J. R. (2008) · Journal of Clinical Psychopharmacology · doi:10.1097/jcp.0b013e318166c4d5 · Cited by 216 · see citing works
Subjective socioeconomic status and health in cross-national comparison
Präg, P., Mills, M. C., & Wittek, R. P. M. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.11.044 · Cited by 216 · see citing works
“(Note that when two 95% CI's do not overlap, the respective point estimates are significantly different at least at the 99% level, not just the 95% level, see e.g. Cumming and Finch, 2005).”
BackgroundThompson, R. J., Parker, K. J., Hallmayer, J. F., Waugh, C. E., & Gotlib, I. H. (2011) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2010.07.003 · Cited by 211 · see citing works
Ecological behavior across the lifespan: Why environmentalism increases as people grow older
Otto, S., & Kaiser, F. G. (2014) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2014.08.004 · Cited by 221 · see citing works
Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing
Hubbard, R., & Lindsay, R. (2008) · Theory & Psychology · doi:10.1177/0959354307086923 · Cited by 244 · see citing works
“…overlapping CIs suggest tenable estimates of the same population parameter, and we applaud the very useful recent work in this area (e.g., Cumming & Finch, 2001, 2005; Fidler, Thomason, Cumming, Finch, & Leeman, 2005; Goldstein & Healy, 1995; Schenker & Gentleman, 2001; Schmidt, 1996;…”
MethodDemir‐Lira, Ö. E., Applebaum, L. R., Susan Goldin-meadow, A., & Levine, S. C. (2018) · Developmental Science · doi:10.1111/desc.12764 · Cited by 211 · see citing works
A Meta-Analysis of Perfectionism and Academic Achievement
Madigan, D. J. (2019) · Educational Psychology Review · doi:10.1007/s10648-019-09484-2 · Cited by 219 · see citing works
Effect-Size Measures and Meta-Analytic Thinking in Counseling Psychology Research
Henson, R. K. (2006) · The Counseling Psychologist · doi:10.1177/0011000005283558 · Cited by 200 · see citing works
“Many researchers have advanced the field by placing research focus on effect sizes and CIs and by emphasizing meta-analytic thinking (Cumming & Finch, 2005; Kline, 2004; Quintana & Minami, in press [TCP special issue, part 2]; Smithson, 2000; Thompson, 2002; Tryon, 2001).”
BackgroundInfluentialAltering expectancy dampens neural response to aversive taste in primary taste cortex
Nitschke, J. B., Dixon, G. E., Sarinopoulos, I., Short, S. J., Cohen, J. D., Smith, E. E., Kosslyn, S. M., Rose, R. M., & Davidson, R. J. (2006) · Nature Neuroscience · doi:10.1038/nn1645 · Cited by 199 · see citing works
Considerations for Visualizing Comparison
Gleicher, M. L. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744199 · Cited by 199 · see citing works
(2008) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 194 · see citing works
Hullman, J., Resnick, P., & Adar, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142444 · Cited by 194 · see citing works
“For example, 92 error bars representing confidence intervals can be superimposed on bar charts [12].”
BackgroundIndividual differences and fitting methods for the two-choice diffusion model of decision making.
Ratcliff, R., & Childers, R. (2015) · Decision · doi:10.1037/dec0000030 · Cited by 192 · see citing works
“To identify an individual as different from our student population, his or her value for any of the parameters would have to lie at least one SD outside the 2 SD confidence interval for the students (e.g., Cumming & Finch, 2005).”
Confidence intervals in repeated-measures designs: The number of observations principle.
Jarmasz, J., & Hollands, J. G. (2009) · Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale · doi:10.1037/a0014164 · Cited by 191 · see citing works
“Rather than weigh in on the debate, Cumming and Finch (2005) take a “decisionwise” approach when using CIs, which limits the number of visual comparisons made with CIs, similar to the”
BackgroundSchubert, E. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00837 · Cited by 183 · see citing works
Lo, S., Scolyer, R. A., & Thompson, J. F. (2018) · Annals of Surgical Oncology · doi:10.1245/s10434-017-6325-1 · Cited by 181 · see citing works
“This difference was highly significant (p B 0.01), with no overlap of the CIs.16 This finding of a ‘‘natural’’ cut point at 0.8 mm was consistent with the Cox proportional hazard regression analyses.”
InfluentialGoedhart, J. (2021) · Molecular Biology of the Cell · doi:10.1091/mbc.e20-09-0583 · Cited by 181 · see citing works
“Super-PlotsOfData has an option to display the mean and 95% CI to enable inference by eye (Cumming and Finch, 2005; Cumming, 2009).”
Substitutes for procedural fairness: Prototypical leaders are endorsed whether they are fair or not.
Ullrich, J., Christ, O., & van Dick, R. (2009) · Journal of Applied Psychology · doi:10.1037/a0012936 · Cited by 176 · see citing works
Closed-form confidence intervals for functions of the normal mean and standard deviation
Donner, A. P., & Zou, G. (2010) · Statistical Methods in Medical Research · doi:10.1177/0962280210383082 · Cited by 172 · see citing works
“Thus, there is little need to conduct hypothesis testing by examining the overlap of these limits using, for example, an 83% confidence intervals.(18,19,35,36).”
Confidence intervals for correlations when data are not normal
Bishara, A. J., & Hittner, J. B. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0702-8 · Cited by 173 · see citing works
“…Department of Psychology, College of Charleston, 66 George Street, Charleston, SC 29424, USA hypothesis significance testing (see, e.g., Cohen, 1994; Cumming & Finch, 2005; Loftus, 1996; Tryon, 2001), and this suspicion has been supported by recent studies in statistical cognition.”
BackgroundConfidence intervals for two sample means: Calculation, interpretation, and a few simple rules
Pfister, R., & Janczyk, M. (2013) · Advances in Cognitive Psychology · doi:10.5709/acp-0133-x · Cited by 173 · see citing works
Welfare regimes, population health and health inequalities: a research synthesis
Brennenstuhl, S., Quesnel‐Vallée, A., & McDonough, P. (2011) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2011-200277 · Cited by 172 · see citing works
Conley, C. S., Shapiro, J. B., Huguenel, B. M., & Kirsch, A. C. (2020) · Emerging Adulthood · doi:10.1177/2167696818791603 · Cited by 171 · see citing works
“We followed the guidelines of Cumming and Finch (2005), who emphasized that interpreting significance only when the CIs of means do not overlap is too conservative.”
MethodSubstance and artifact in the higher-order factors of the Big Five.
McCrae, R. R., Yamagata, S., Jang, K. L., Riemann, R., Ando, J., Ono, Y., Angleitner, A., & Spinath, F. M. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.95.2.442 · Cited by 169 · see citing works
“It is important to note, however, that overlapping confidence intervals does not necessarily imply that the difference between the models is nonsignificant (Cumming & Finch, 2005).”
ResultDetecting and Correcting the Lies That Data Tell
Schmidt, F. (2010) · Perspectives on Psychological Science · doi:10.1177/1745691610369339 · Cited by 163 · see citing works
“CIs are just as objective as significance tests, and they provide far more information (Borenstein, 1994; Cumming & Finch, 2005).”
BackgroundThe New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known
Calin-Jageman, R. J., & Cumming, G. (2019) · The American Statistician · doi:10.1080/00031305.2018.1518266 · Cited by 170 · see citing works
“Thus, the natural inclination to use a “no overlap” rule is actually overly cautious, in many cases equivalent to requiring p 0.01 to judge results inconsistent (see Cumming and Finch 2005 for an explanation and important caveats).”
ResultInfluentialSelf-citationConfidence intervals for two sample means: Calculation, interpretation, and a few simple rules.
Pfister, R., & Janczyk, M. (2013) · Europe PMC (PubMed Central) · doi:10.2478/v10053-008-0133-x · Cited by 166 · see citing works
- Authors
- Cumming, G., & Finch, S.
- Affiliations
- La Trobe University (AU)
- Venue
- American Psychologist
- Published
- 2005
- Volume
- 60
- Issue
- 2
- Pages
- 170-180
- ISSN
- 0003-066X (ISSN-L)
- 1935-990X
- OpenAlex
- W2005534705
- Cited by
- 1,449
- References
- 59
- Citations by year
- 2026: 62 · 2025: 66 · 2024: 74 · 2023: 92 · 2022: 102 · 2021: 110 · 2020: 92 · 2019: 86 · 2018: 80 · 2017: 82 · 2016: 76 · 2015: 89 · 2014: 63 · 2013: 59 · 2012: 48
- FWCI
- 39.36
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Psychologist (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- confidence intervals, error bars, statistical significance testing, interval estimation, scholarly communication
- MeSH terms
- Humans, Confidence Intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
