Inference by Eye: Confidence Intervals and How to Read Pictures of Data.

“For excellent examples, see ESCI (Cumming and Finch, 2005), confidence interval software by Smithson (2001), and G ∗ Power (Faul et al., 2009).”

“Multiple papers evaluating the same intervention but containing different outcome data at post or follow-up for the same cohort were combined into a single study.”

“The confidence intervals are computed with a bootstrap algorithm, allowing them to generalize over many different statistics, and the default level allows the user to perform “inference by eye” (Cumming & Finch, 2005).”

“The best way to think of SE bars on a mean is usually to double the whole length—so the bars extend 2 SE above and 2 SE below the mean—and interpret them as being, approximately, the 95% CI, as Figure 2 illustrates (Cumming & Finch, 2005).”

“Thus, even if confidence intervals overlap, values can still be significantly different from one another (for interpretation of confidence intervals, see Cumming & Finch, 2005).”

“Substantive interpretations can be based on values anywhere in the interval (Cumming & Finch, 2005).”

“To compare the effect sizes between regulation types, we relied on the confidence interval method proposed by Cumming and Finch (2005).”

“When the observations are uncorrelated, two values can be considered to differ at the p .01 level if their CIs do not overlap (Cumming & Finch, 2005).”

“, from n knock-out mice and n wild-type mice, you can compare the SE bars or the 95% CIs on the two means (6).”

“Cumming and Finch (2005) showed that if 95% CIs for independent means do not overlap, this is equivalent to a significant difference between the two respective means at .01 (see also Cumming, 2011).”

“If the key parameter (e.g. a correlation) of the replication study falls within the CI of the initial study (or if the two CIs overlap substantially, Cumming & Finch, 2005), one can argue more strongly that the result is replicated.”

“Please note that inferences on significance are somewhat limited if the sizes of the two CIs to be compared differ by more than a factor of 2 (Cumming and Finch 2005).”

“An informal adjustment towards conservatism, to protect against inflated Type 1 error rates [ 5 ], may be to use p 5 ] recommended a range of ways to think about CIs without invoking p values.”

“For example graphs and interpre-tative strategies, see Cumming and Finch (2005), Jarmasz and Hollands (2009), and Masson and Loftus (2003).”

“Cumming and Finch’s (2005) procedure was used to detect significant differences between group means by examining the extent of any overlap in the two groups’ mean confidence intervals.”

“…data by making dichotomous statements (Neyman 1937, 1941), eventually becoming a method that many believe is the best way to reason from data (e.g. Cumming & Finch, 2005; Cumming & Fidler, 2009) and a way to avoid dichotomous statements (e.g. Cumming, 2014; Hoekstra, Finch, Kiers, & Johnson,…”

“Many advocates of statistical reform, including Kline (2004, Chapter 3) and Cumming and Finch (2005), have presented reasons why psychologists should use CIs wherever possible.”

“IF in the ‘ knowledge ’ outcome was the only moderator variable for which there was no observed CI overlap between conditions and thus a signifi cant difference (Cumming & Finch, 2005).”

“When two CIs overlapped by no more than one quarter of the average length of the two intervals, then these mean effects were also interpreted as representing two different population parameters and were statistically different at approximately p ≤ .05 (Cumming & Finch, 2005).”

“The standard error of the estimate was .10 (CI = ± .196), confi dence intervals of the items did not overlap in 14 out of the 20 items when comparing regular and combined forms, so statistically signifi cant differences were found in these 14 items (α = .05; Cumming & Finch, 2006).”

“The confidence interval reflects the extent to which sampling error remains in the effect size estimate (Whitener, 1990) and suggests the range of plausible values for the true effect size (Cumming & Finch, 2005).”

“In these analyses, similar estimated effects and overlapping confidence interval would indicate that published and unpublished samples are similar (Cumming & Finch, 2005) and therefore do not present evidence for publication bias.”

“The misuse of NHST has perhaps been most hotly debated in psychology (Loftus, 1996; Cumming & Finch, 2005).”

“More recently, LeDoux (2002) described aversive stimuli and warning signals that precede them as being served by a single brain system (pp.”

“In accordance with Cumming and Finch (2005), non-overlapping CIs indicated differences bet-ween values at a probability approximately equal to .01, and CIs which overlapped less than 50 % were considered indicative of differences in values of approximately p .05.”

“In fact, when sample sizes are similar and not small, and CI widths are similar, then if 95% CIs on independent means just touch, the two-tail p value is about .006 ( Cumming & Finch, 2005; Payton, Greenstone, & Schenker, 2003), not .05 as many believe.”

“Based on credibility intervals, this approach is stronger than the many recommendations for using confidence intervals (e.g., American Psychological Association, 2010; Cumming & Finch, 2005).”

“A first management implication of our study relates to the multidimensional notion of innovation in services.”

“Our software program also graphs CIs, keeping in mind that the statistical differences between statistics may still be significant even when their respective CIs overlap (Cumming & Finch, 2005).”

“Cumming and Finch(16) list four advantages of using CIs: they give point interval estimates in easily comprehensible units; p values can be easily estimated using CIs, with 14 overlap of 95% CI width being approximately equivalent to p<.05 (for independent groups); CIs help combine evidence from multiple studies, thus…”

“For a discussion of the different ways the CI can be interpreted, see Cumming and Finch (2005). It should be noted that the CI can be interpreted as a significance test; that is, a CI that contains 0 is not statistically significant.”

“…placed in the context of additional information, such as confidence intervals and effect size (AERA, 2006; Ander- son et al., 2000; APA, 2001, 2010; Cumming & Finch, 2002; Fan, 2001; Kirk, 1996, 2001; McLean & Ernest, 1998; Snyder & Lawson, 1993; Thompson, 1996, 1997, 2002b; Vacha-Haase &…”

“Effect Size Comparisons We followed the guidelines of Cumming and Finch (2005), wherein mean ESs whose CIs do not include zero are considered statistically significant, pairs of CIs with no overlap are statistically different at the p<.”

“…the RTs of each individual and ascertained whether the individual data fall outside of the 95% confidence interval of the control subjects, following the recommendations for single case study analysis (Crawford & Garthwaite, 2004) and other statistical recommendations (Cumming & Finch, 2005).”

“If needed, a p-value can always be roughly inferred from a confidence interval by looking at how far it is from zero (Cumming and Finch, 2005; Cumming, 2012, pp.106–108).”

“However, the converse is not true: overlapping con dence intervals do not necessarily mean that the means are not di erent (see Figure 4 in [17] for explanation).”

“As described by Cumming and Finch (2005), this approach does not require a specialised understanding of meta-analytic techniques, and instead assesses how closely related two confidence intervals are, such as those presented in Figures 1–3 (see ‘results’).”

“The error bars represent 95% confidence intervals (according to Cumming and Finch, 2005, two samples differ significantly at, for example, p \ 0.01 if the corresponding intervals do not overlap).”

“As was mentioned earlier, when effects within a study are theoretically expected to be different from each other, examination of the CIs illustrates whether this expectation was met (Cumming & Finch, 2005).”

“Williams, & Cumming, 2005; Cumming, Williams, & Fidler, 2004); • empirical and conceptual studies of how CIs can most effectively be presented and used to interpret research results (e.g., Cumming & Finch, 2005 ); • empirical studies of how to teach CIs, and the consequences of giving CIs a foundational role in statis…”

“For this latter situation, there are some rules of thumb for interpreting overlapping CIs appropriately (Cumming & Finch, 2005), or one may conduct a formal statistical test of the difference.”

“This difference is compared to the pooled SE across variables, with gaps equal to or larger than the pooled SE indicating significant differences between models (Cumming and Finch 2005, rule 7).”

“Cumming and Finch (2005) explored some of these barriers, providing guidance on how to calculate, report, and interpret CIs (with emphasis on the graphical presentation of means in a two-independent-group design).”

“(Note that when two 95% CI's do not overlap, the respective point estimates are significantly different at least at the 99% level, not just the 95% level, see e.g. Cumming and Finch, 2005).”

“…overlapping CIs suggest tenable estimates of the same population parameter, and we applaud the very useful recent work in this area (e.g., Cumming & Finch, 2001, 2005; Fidler, Thomason, Cumming, Finch, & Leeman, 2005; Goldstein & Healy, 1995; Schenker & Gentleman, 2001; Schmidt, 1996;…”

“Many researchers have advanced the field by placing research focus on effect sizes and CIs and by emphasizing meta-analytic thinking (Cumming & Finch, 2005; Kline, 2004; Quintana & Minami, in press [TCP special issue, part 2]; Smithson, 2000; Thompson, 2002; Tryon, 2001).”

“For example, 92 error bars representing confidence intervals can be superimposed on bar charts [12].”

“To identify an individual as different from our student population, his or her value for any of the parameters would have to lie at least one SD outside the 2 SD confidence interval for the students (e.g., Cumming & Finch, 2005).”

“Rather than weigh in on the debate, Cumming and Finch (2005) take a “decisionwise” approach when using CIs, which limits the number of visual comparisons made with CIs, similar to the”

“This difference was highly significant (p B 0.01), with no overlap of the CIs.16 This finding of a ‘‘natural’’ cut point at 0.8 mm was consistent with the Cox proportional hazard regression analyses.”

“Super-PlotsOfData has an option to display the mean and 95% CI to enable inference by eye (Cumming and Finch, 2005; Cumming, 2009).”

“Thus, there is little need to conduct hypothesis testing by examining the overlap of these limits using, for example, an 83% confidence intervals.(18,19,35,36).”

“…Department of Psychology, College of Charleston, 66 George Street, Charleston, SC 29424, USA hypothesis significance testing (see, e.g., Cohen, 1994; Cumming & Finch, 2005; Loftus, 1996; Tryon, 2001), and this suspicion has been supported by recent studies in statistical cognition.”

“We followed the guidelines of Cumming and Finch (2005), who emphasized that interpreting significance only when the CIs of means do not overlap is too conservative.”

“It is important to note, however, that overlapping confidence intervals does not necessarily imply that the difference between the models is nonsignificant (Cumming & Finch, 2005).”

“CIs are just as objective as significance tests, and they provide far more information (Borenstein, 1994; Cumming & Finch, 2005).”

“Thus, the natural inclination to use a “no overlap” rule is actually overly cautious, in many cases equivalent to requiring p 0.01 to judge results inconsistent (see Cumming and Finch 2005 for an explanation and important caveats).”