CoChat
Sign Up
Cited by 71Data Visualization and AnalyticsVisual and Cognitive Learning ProcessesStatistics Education and Methodologies

Illusion of Causality in Visualized Data

Open access
Xiong, Cindy, Shapiro, Joel, Hullman, Jessica, Franconeri, Steven·2020·IEEE Transactions on Visualization and Computer Graphics
DOI 10.1109/tvcg.2019.2934399
Open this paper in CoChat — free accountOpen PDF

Abstract

Students who eat breakfast more frequently tend to have a higher grade point average. From this data, many people might confidently state that a before-school breakfast program would lead to higher grades. This is a reasoning error, because correlation does not necessarily indicate causation - X and Y can be correlated without one directly causing the other. While this error is pervasive, its prevalence might be amplified or mitigated by the way that the data is presented to a viewer. Across three crowdsourced experiments, we examined whether how simple data relations are presented would mitigate this reasoning error. The first experiment tested examples similar to the breakfast-GPA relation, varying in the plausibility of the causal link. We asked participants to rate their level of agreement that the relation was correlated, which they rated appropriately as high. However, participants also expressed high agreement with a causal interpretation of the data. Levels of support for the causal interpretation were not equally strong across visualization types: causality ratings were highest for text descriptions and bar graphs, but weaker for scatter plots. But is this effect driven by bar graphs aggregating data into two groups or by the visual encoding type? We isolated data aggregation versus visual encoding type and examined their individual effect on perceived causality. Overall, different visualization designs afford different cognitive reasoning affordances across the same data. High levels of data aggregation by graphs tend to be associated with higher perceived causality in data. Participants perceived line and dot visual encodings as more causal than bar encodings. Our results demonstrate how some visualization designs trigger stronger causal links while choosing others can help mitigate unwarranted perceptions of causality.

AI snapshot

Written from the paper’s full text.

TL;DR

Across three crowdsourced experiments, the paper finds that visualization design changes how strongly viewers infer causation from correlational data, while correlation judgments remain largely stable. Higher data aggregation and bar/text formats increased perceived causality, whereas scatter plots, line graphs, and less aggregation reduced it.

Key findings

  • In Experiment 1, causation ratings were highest for bar graphs (M=76.59) and text (M=71.26), lower for line graphs (M=68.43) and scatter plots (M=67.29). Context plausibility had a large effect (χ²=162.70, η²partial=0.274), while visualization design had a small effect (χ²=11.65, η²partial=0.026). Correlation ratings were predicted mainly by correlation statement plausibility (χ²=71.02, η²partial=0.141), with negligible visualization design effect (χ²=1.98, η²partial=0.005).
  • Experiment 1 qualitative coding found causal conclusions in 39.0% of text trials, 33.8% of bar graph trials, 20.6% of scatter plot trials, and 18.4% of line graph trials. Correlation conclusions were most frequent for line graphs (75.7%) and scatter plots (69.1%), versus bar graphs (52.9%) and text (50.0%).
  • Experiment 2 found a large main effect of aggregation level on perceived causation (χ²=117.05, η²partial=0.29): aggregation level 2 had M=84.76, level 8 M=82.95, and level 16 M=63.74. Visual encoding marks had a small effect (χ²=5.97, η²partial=0.020): dot encodings M=79.38, line M=77.78, bar M=74.32, with an interaction between encoding and aggregation (χ²=28.10, η²partial=0.089).
  • Experiment 3, using non-aggregated displays, found dot encodings most causal (M=55.49), followed by line (M=52.02) and bar (M=43.21); encoding type effect was χ²=15.44, η²partial=0.10. A post-hoc comparison reported a large effect of data manipulation (χ²=93.38, η²partial=0.17, p<0.001), described as aggregated data being perceived as more causal than non-aggregated data.

Methods

  • A pilot experiment surveyed 21 MTurk participants on 19 variable pairs, collecting correlation and causation plausibility ratings and free-response interpretations coded into a six-dimension taxonomy. Experiment 1 used a 4×4 Graeco Latin Square with 136 participants, four variable contexts, and four designs (bar, line, scatter, text), plus judgment and generative tasks. Experiment 2 recruited 129 participants in a 3×3 design crossing encoding marks (bar, line, dot) and aggregation levels (2, 8, 16 bins) with abstract labels and judgment tasks only. Experiment 3 used a within-subject design with a target sample of 62, testing non-aggregated bar, line, and scatter displays.
  • Main analyses used mixed-effect linear models for correlation and causation ratings, with fixed effects including visualization design, plausibility, trial order, and demographics, and random intercepts for participants. Qualitative responses were analyzed with chi-square tests of independence, and power analyses were based on pilot effect sizes.

Limitations

  • The paper states it is too early to provide concrete design guidelines; it notes unresolved special-case effects of two-bar bar graphs, untested aggregation in concrete contexts, untested complex displays such as dashboards, reliance on abstract variable names in Experiments 2 and 3, use of a single positive upward-trend dataset, ambiguity in the generative-task taxonomy, limited statement choices, fixed question order, and no test of alternative interventions such as verbal annotation.
  • The paper states that explanations for visual encoding effects are speculative—line encodings may suggest continuous trends, dot encodings may allow mentally drawn lines, and bar encodings may be affected by vertical asymmetry—and that these hypotheses need empirical testing.
  • Evident from the text: samples were MTurk-based with exclusion criteria, Experiments 2 and 3 used non-conventional or abstract stimuli that may limit ecological validity, and the post-hoc comparison text appears to reverse the labels for aggregated versus non-aggregated means.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tvcg.2019.2934399

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 32 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § 4.8 Discussion of Experiment 1 · p. 6

    Context had a relatively large effect on perceived causality.

    “Context also had a relatively large effect on perceived causality, but the effect of visualization design on perceived causality was not context dependent.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 4.7 Qualitative Results from Generative Task · p. 6

    There was an overall effect of visualization design on the number of group-wise comparisons made (χ2=15.57, p<0.001).

    “We observed an overall effect of visualization design on the number of group-wise comparisons made ( c 2 =15.57, p &lt;0.001).”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4.5 Causation Judgment Results · p. 4

    Causation statement plausibility (context) had a relatively large effect on participants' causation ratings (χ2=162.70, η2_partial=0.274, p<0.001).

    “The regression model indicated a relatively large effect of causation statement plausibility (context), c 2 =162.70, h 2 partial =0.274, p &lt;0.001”
    Quote in the paperSubstantial claim
  • Findingown data
    § 3.1 Picking Statements · p. 3

    The four selected contexts differed significantly in their perceived correlation and causation ratings, based on an analysis of variances.

    “These four context differed significantly in their perceived correlation and causation ratings, based on an analysis of variances, as shown in Figure 4.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4.5 Causation Judgment Results · p. 4

    The interaction between causation statement plausibility (context) and visualization design was negligible (χ2=0.97, η2_partial=0.002, p=0.81).

    “negligible interaction effect between causation statement plausibility (context) and visualization design ( c 2 =0.97, h 2 partial =0.002, p =0.81)”
    Quote in the paperSubstantial claim
  • May combine two claims. 31
  • Classified as probably not a substantial claim. 38
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data
    § 5.2 Causation Judgment Results · p. 7

    For bar encodings, aggregation level eight was perceived as most causal (M=89.42, 95% CI [84.85, 93.98]), followed by level two (M=72.77, 95% CI [63.62, 81.92]), and level 16 least causal (M=60.77, 95% CI [53.33, 68.20]).

    “For bar encodings, aggregation level eight was perceived as being the most causal ( M =89.42, CI 95%=[84.85,93.98]), followed by aggregation level two ( M =72.77, CI 95%=[63.62, 81.92]), and aggregation level 16 the least causal ( M =60.77, CI 95%=[53.33, 68.20]).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.6 Correlation Judgment Results · p. 5

    Participants with higher levels of education gave lower correlation ratings (tiny effect).

    “a tiny effect of education ( c 2 =2.99, h 2 partial =0.007, p =0.08), such that participants with higher levels of education gave lower correlation ratings.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.5 Causation Judgment Results · p. 5

    There was a negligible effect of the order in which visualizations were presented on causation ratings, indicating no learning effect.

    “There is negligible effect of the order the visualizations were presented ( c 2 =0.11, h 2 partial =0.002, p =0.74), which means perceived causation does not depend on what was presented to them previously nor was there a learning effect.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.8 Discussion of Experiment 1 · p. 6

    Quantitative and qualitative results told similar stories: people perceived causality as more likely in bar graphs and less likely in scatter graphs.

    “In general, the quantitative and qualitative results told similar stories of how, when given specific pairs of common variables, people perceived causality as more likely in bar graphs and less likely in scatter graphs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Causation Judgment Results · p. 7

    For line encodings, perceived causality significantly decreased as the number of bins increased, with aggregation level two most causal (M=94.37, 95% CI [91.76, 96.98]), level eight next (M=84.91, 95% CI [78.55, 91.26]), and level 16 least causal (M=54.05, 95% CI [46.43, 61.67]).

    “For line encodings, perceived causality significantly decreased as the number bins increased, such that aggregation level two ( M =94.37, CI 95%=[91.76,96.98]) was perceived the most causal, followed by aggregation level eight ( M =84.91, CI 95%=[78.55,91.26]), and aggregation level 16 was perceived the least causal ( M =54.05, CI 95%=[46.43,61.67]).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.5 Causation Judgment Results · p. 5

    Participants with higher levels of education gave lower causation ratings (small effect).

    “and education ( c 2 =0.4.53, h 2 partial =0.011, p =0.033), such that participants with higher levels of educating gave lower causation ratings.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.7 Qualitative Results from Generative Task · p. 5

    On average, participants drew correlation conclusions in 75.7% of trials from line graphs, 69.1% from scatterplots, 52.9% from bar graphs, and 50.0% from text.

    “On average, in 75.7% of the trials participants drew correlation conclusion from line graphs ( CI 95%=[68.7, 82.9]), 69.1% from scatterplots ( CI 95%=[61.4, 76.9]), 52.9% from bar graphs ( CI 95%=[44.6, 61.3]), and 50.0% from text ( CI 95%=[41.6, 58.4]).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Causation Judgment Results · p. 8

    There is a negligible effect of the order the visualizations were presented (χ2=0.14, η2partial=0.002, p=0.71) as well as participant age, political orientation, gender and education.

    “There is a negligible effect of the order the visualizations were presented ( c 2 =0.14, h 2 partial =0.002, p =0.71) as well as participant age, political orientation, gender and education.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • May combine two claims.

    Findingown data
    § 4.7 Qualitative Results from Generative Task · p. 6

    Two types of no-conclusion responses were observed: one in which participants inferred causality but decided to draw no conclusion because it went against their intuition, and one in which participants made a conscious decision not to.

    “We observed two types of no conclusion responses, one in which participants inferred causality from the visualization but decided to draw no conclusion because it went against their intuition, and the other in which participants made a conscious decision not to.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.7 Qualitative Results from Generative Task · p. 6

    On average, participants mentioned third variables in 30.9% of trials in scatter plots, 30.9% in text, 30.2% in line graphs, and 24.3% in bar graphs.

    “On average, in 30.9% of the trials participants mentioned third variables in scatter plots ( CI 95%=[23.1, 38.7]), 30.9% in text ( CI 95%=[23.1, 38.7]), 30.2% in line graphs ( CI 95%=[22.4, 37.9]), and 24.3% in bar graphs ( CI 95%=[17.1, 31.5]).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.7 Qualitative Results from Generative Task · p. 6

    Participants made direct observations in 11.0% of trials in bar graphs, 6.6% in scatter plots, 5.9% in text, and 4.4% in line graphs.

    “in 11.0% of the trials participants made direct observations in bar graphs ( CI 95%=[5.8, 16.3]), 6.6% in scatter plots ( CI 95%=[2.4, 10.8]), 5.9% in text ( CI 95%=[1.9, 9.8]), and 4.4% line graphs ( CI 95%=[0.96, 7.9]).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.5 Causation Judgment Results · p. 4

    Participants rated bar graphs as the most causal (M=76.59, 95% CI [71.51, 81.76]) and text as the second most causal (M=71.26, 95% CI [65.30, 77.23]).

    “participants rated bar graphs to be the most causal ( M =76.59, CI 95%=[71.51, 81.76]) and text the second most causal ( M =71.26, CI 95%=[65.30, 77.23]).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.5 Causation Judgment Results · p. 5

    Male participants gave higher causation ratings than female participants (small effect).

    “Results also indicated a comparatively small effect of gender ( c 2 =4.23, h 2 partial =0.007, p =0.040), such that male participants gave higher causation ratings”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.6 Correlation Judgment Results · p. 5

    There were negligible effects of order, age and gender on correlation ratings.

    “There were negligible effects of order, age and gender (details included in the supplementary materials).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.7 Qualitative Results from Generative Task · p. 5

    Participants drew causal conclusions in 39.0% of trials from text, 33.8% from bar graphs, 20.6% from scatter plots, and 18.4% from line graphs.

    “in 39.0% of the trials participants drew causal conclusion from text ( CI 95%=[30.8, 47.2]), in 33.8% from bar graphs ( CI 95%=[25.9, 41.8]), in 20.6% from scatter plots ( CI 95%=[13.8, 27.4]), and 18.4% from line graphs ( CI 95%=[11.9, 24.9]).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.7 Qualitative Results from Generative Task · p. 6

    Participants made group-wise comparisons in 27.9% of trials in bar graphs, 16.2% in text, 16.2% in scatter plots, and 9.6% in line graphs.

    “In 27.9% of the trials participants made group-wise comparisons of variables in bar graphs ( CI 95%=[20.4, 35.5]), 16.2% in text ( CI 95%=[9.99, 22.4]), 16.2% in scatter plots ( CI 95%=[9.99, 22.4]), and 9.6% in line graphs ( CI 95%=[4.6, 14.5]).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.7 Qualitative Results from Generative Task · p. 6

    Few participants stopped and thought of counter examples, questioned the validity of the data, or showed clear signs of understanding that correlation is not causation.

    “Few stopped and thought of”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.6 Correlation Judgment Results · p. 5

    There was a small interaction between correlation statement plausibility and visualization design on perceived correlations.

    “a small interaction between the two ( c 2 =6.15, h 2 partial =0.012, p =0.10)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 5.5 Discussion of Experiment 2 · p. 8

    Bar visual encoding marks received the lowest causal ratings, followed by line, and dot encodings received the highest causal ratings.

    “Bar visual encoding marks received the lowest causal ratings, followed by line, and dot encodings received the highest causal ratings.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.2 Visual Mark Encoding Types · p. 8

    There is an effect of age (χ2=3.43, η2partial=0.014), such that older participants rated causation less on average.

    “age ( c 2 =3.43, h 2 partial =0.014), such that older participants rated causation less on average”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • May combine two claims.

    Findingown data
    § 5.4 Correlation Judgment Results · p. 8

    For line encodings, perceived correlations significantly decreased as the number of bins increased, with level two most correlational (M=96.42, 95% CI [94.49, 98.35]), level eight next (M=93.37, 95% CI [91.03, 95.72]), and level 16 least correlational (M=64.28, 95% CI [56.88, 71.68]).

    “For line encodings, perceived correlations significantly decreased as the number bins increased, such that aggregation level two ( M =96.42, CI 95%=94.49,98.35]) was perceived to be the most correlational, followed by aggregation level eight ( M =93.37, CI 95%=[91.03,95.72]), and aggregation level 16 was perceived to be the least correlational ( M =64.28, CI 95%=[56.88,71.68]).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.2 Visual Mark Encoding Types · p. 8

    There is an effect of education (χ2=4.84, η2partial=0.035), such that participants with higher levels of education gave higher causation ratings.

    “and education ( c 2 =4.84, h 2 partial =0.035), such that participants with higher levels of education gave higher causation ratings.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 5.4 Correlation Judgment Results · p. 8

    There was a relatively small effect of education (χ2=3.5, η2partial=0.019, p=0.84), such that participants with higher levels of education gave higher correlation ratings.

    “and education ( c 2 =3 .5, h 2 partial =0.019, p =0.84), such that participants with higher levels of education gave higher correlation ratings.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • May combine two claims.

    Findingown data
    § 8 LIMITATIONS AND FUTURE DIRECTIONS · p. 9

    There was no significant effect of context in Experiment 1, and no significant difference between causation ratings of the two-bar bar graph from Experiment 1 (with context) and that from Experiment 2 (no context).

    “We found no significant effect of context in Experiment 1, and no significant difference between causation ratings of the two-bar bar graph from Experiment 1 (with context) with that from Experiment 2 (no context).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 5.4 Correlation Judgment Results · p. 8

    There was negligible effect of age and gender, but a relatively small effect of political orientation (χ2=1.85, η2partial=0.013, p=0.17), such that more liberal participants gave higher correlation ratings overall.

    “There was negligible effect of age and gender, but a relatively small effect of political orientation ( c 2 =1.85, h 2 partial =0.013, p =0.17), such that more liberal participants gave higher correlation ratings overall”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.2 Visual Mark Encoding Types · p. 9

    There is a relatively small effect of order (χ2=7.68, η2partial=0.055) suggesting a relatively small learning effect.

    “There is a relatively small effect of order ( c 2 =7.68, h 2 partial =0.055) suggesting a relatively small learning effect”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • May combine two claims.

    Findingown data
    § 5.4 Correlation Judgment Results · p. 8

    For bar encodings, aggregation level eight was perceived as most correlational (M=95.30, 95% CI [93.18, 97.43]), followed by level two (M=88.77, 95% CI [82.74, 94.80]), and level 16 least correlational (M=62.23, 95% CI [55.39, 69.07]).

    “For bar encodings, aggregation level eight was perceived to be the most correlational ( M =95.30, CI 95%=[93.18,97.43]), followed by aggregation level two ( M =88.77, CI 95%=[82.74, 94.80]), and aggregation level 16 the least correlational ( M =62.23, CI 95%=[55.39, 69.07]).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2 RELATED WORK · p. 2

    Bar graphs facilitate finding clusters, line graphs facilitate finding correlations and scatter plots facilitate finding outliers.

    “Bar graphs facilitate finding clusters, line graphs facilitate finding correlations and scatter plots facilitate finding outliers [41, 50].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [41] [41] B. Saket, A. Endert, and C. Demiralp. Task-based effectiveness of basic visualizations. IEEE transactions on visualization and computer graphics , 2018. - source

      Not checked in the free run.

    • cites [50] [50] J. Zacks and B. Tversky. Bars and lines: A study of graphic communication. Memory &amp; Cognition , 27(6):1073-1079, 1999. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 RELATED WORK · p. 2

    People rely on visual salience of attributes to interpret data.

    “People also rely on visual salience of attributes to interpret data [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] [21] S. L. Jarvenpaa. Graphic displays in decision making-the visual salience effect. Journal of Behavioral Decision Making , 3(4):247-262, 1990. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § 1 INTRODUCTION · p. 1

    Even trained scientists can sometimes struggle with correlation and causation because establishing causal inference is complex.

    “Because establishing causal inference is complex, even trained scientists can sometimes struggle with correlation and causation [15].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] [15] D. F. Halpern. Teaching critical thinking for transfer across domains: Disposition, skills, structure training, and metacognitive monitoring. American psychologist , 53(4):449, 1998. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 2 RELATED WORK · p. 2

    In perceptual analytic tasks, choosing appropriate visualization designs can improve the accuracy and efficiency of information extraction.

    “In perceptual analytic tasks, choosing the appropriate visualization designs can improve the accuracy and efficiency of information extraction.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 RELATED WORK · p. 2

    People interpret climate data differently depending on whether the visualization presented percentile information versus showing the range.

    “People interpret climate data differently depending on whether the visualization presented percentile information versus showing the range [8].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] [8] J. D. Daron, S. Lorenz, P. Wolski, R. C. Blamey, and C. Jack. Interpreting climate data visualisations to inform adaptation decisions. Climate Risk Management , 10:17-26, 2015. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2 RELATED WORK · p. 2

    People can be more easily persuaded by tabular designs of data when they hold strong attitudes against the depicted topic, but more easily persuaded by bar graphs when they have no strong attitudes.

    “People can be more easily persuaded by tabular designs of data when they hold strong attitudes against the depicted topic, but more easily persuaded by bar graphs when they have no strong attitudes [34].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] [34] A. V. Pandey, A. Manivannan, O. Nov, M. Satterthwaite, and E. Bertini. The persuasive power of data visualization. IEEE transactions on visualization and computer graphics , 20(12):2211-2220, 2014. - source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 2 RELATED WORK · p. 2

    In bar graphs depicting average values, people judge data values that fall within the bar as being more likely to be part of the data set than points outside the bar, despite being equidistant from the mean.

    “In bar graphs depicting average values, people judge data values that fall within the bar as being more likely to be part of the data set than points outside the bar, despite being equidistant from the mean [30, 6].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [30] [30] G. E. Newman and B. J. Scholl. Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias. Psychonomic bulletin &amp;review , 19(4):601-607, 2012. - source

      Not checked in the free run.

    • cites [6] [6] M. Correll and M. Gleicher. Error bars considered harmful: Exploring alternate encodings for mean and error. IEEE transactions on visualization and computer graphics , 20(12):2142-2151, 2014. - source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 2 RELATED WORK · p. 2

    People often perceive high causality when they judge the evidence as sound and stop thinking through other possible explanations.

    “People often perceive high causality when they judge the evidence as sound and stop thinking through other possible explanations [22].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [22] [22] D. Kahneman and P. Egan. Thinking, fast and slow. Chapter 16, Causes Trump Statistics , volume 1. Farrar, Straus and Giroux New York, 2011. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 RELATED WORK · p. 2

    Bar graphs, line graphs and scatter plots are common ways to depict correlated data in media, alongside text.

    “Bar graphs, line graphs and scatter plots are common ways to depict correlated data in media [31, 14], alongside text, as shown in Figure 2 and Figure 3.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [31] [31] B. News. Heatwave: Is there more crime in hot weather?, 2018. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [14] [14] J. Guo. Researchers have debunked one of our most basic assumptions about how the world works, 2016. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1 INTRODUCTION · p. 1

    A large portion of work in economics, education, epidemiology, psychology and public health involves analyzing correlations in observed data, which cannot definitively establish causation.

    “A large portion of work in economics, education, epidemiology, psychology and public health involves analyzing correlations in observed data, which cannot definitively establish causation [39].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] [39] J. M. Robins, M. A. Hernan, and B. Brumback. Marginal structural models and causal inference in epidemiology, 2000. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 RELATED WORK · p. 2

    Visual marks, such as rectangular bars, lines or dots, can support different inferences about data relations based on their geometric properties.

    “Visual marks, such as rectangular bars, lines or dots, can support different inferences about data relations based on their geometric properties.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [17] [17] J. Heiser and B. Tversky. Arrows in comprehending and producing mechanical diagrams. Cognitive science , 30(3):581-592, 2006. - source

      Not checked in the free run.

    • cites [12] [12] M. Friendly and D. Denis. The early origins and development of the scatterplot. Journal of the History of the Behavioral Sciences , 41(2):103130, 2005. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § Illusion of Causality in Visualized Data · p. 1

    Correlation does not necessarily indicate causation; two variables can be correlated without one directly causing the other.

    “This is a reasoning error, because correlation does not necessarily indicate causation - X and Y can be correlated without one directly causing the other.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 1 INTRODUCTION · p. 1

    Researchers and journalists can sometimes exaggerate causal implications from correlational results, making it more difficult for people to decide what conclusions are sound.

    “Researchers and journalists can sometimes exaggerate causal implications from these results, making it even more difficult for people to decide what kind of conclusions are sound [45, 48].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [45] [45] R. M. Shiffrin. Drawing causal inference from big data. Proceedings of the National Academy of Sciences , 113(27):7308-7309, 2016. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [48] [48] P. Sumner, S. Vivian-Griffiths, J. Boivin, A. Williams, C. A. Venetis, A. Davies, J. Ogden, L. Whelan, B. Hughes, B. Dalton, et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. Bmj , 349:g7015, 2014. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2 RELATED WORK · p. 2

    Spatially upward trends are conventionally associated with increasing values, even when the axes are reverse labelled.

    “Spatially upward trends are conventionally associated with increasing values, even when the axes are reverse labelled [33].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] [33] L. M. Padilla. A case for cognitive models in visualization research: Position paper. In 2018 IEEE Evaluation and Beyond-Methodological Approaches for Visualization (BELIV) , pages 69-77. IEEE, 2018. -

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 2 RELATED WORK · p. 2

    When the evidence presented aligns with people's prior experience, emotional response or beliefs, they become more likely to judge the evidence as sound.

    “When the evidence presented aligns with people's prior experience, emotional response or beliefs, they become more likely to judge the evidence as sound [43].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [43] [43] P. Shah, A. Michal, A. Ibrahim, R. Rhodes, and F. Rodriguez. What makes everyday scientific reasoning so challenging? In Psychology of Learning and Motivation , volume 66, pages 251-299. Elsevier, 2017. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data
    § 4.7 Qualitative Results from Generative Task · p. 6

    Some participants only memorized the phrase 'correlation is not causation' without truly understanding the concept, and dismissed observable correlation in the data.

    “It is also apparent when a participant only memorized the phrase”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 4.3 Materials · p. 4

    To ensure all participants viewed the same visualized data across all conditions, the axis was relabeled to fit the context without changing the underlying dataset.

    “To ensure all participants viewed the same visualized data across all conditions, we relabeled the axis to fit the context without changing the underlying dataset.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 5.3 Comparing Experiment 1 and Experiment 2 Bars · p. 8

    One explanation may be that many participants associate aggregation level 2 bar graphs with controlled experiments, which can be a valid way to establish causal relationships.

    “one explanation may be that many participants associate aggregation level 2 bar graphs with controlled experiments, which can be a valid way to establish causal relationships.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 4.2 Design · p. 4

    Four researchers blind to both the study design and the condition manipulations coded the generative task responses based on the interpretive taxonomy.

    “Four researchers blind to both the study design and the condition manipulations coded the response in the generative task based on the interpretive taxonomy, and the participant count in each category (e.g., direct observation) was one dependent variable.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 2 RELATED WORK · p. 2

    Persuasiveness of visualized data depends on both context and visualization designs.

    “Prior work suggests that persuasiveness of visualized data depends on both context (does the topic align with the viewers' prior beliefs?) and visualization designs (tabular design or bar graphs) [25, 26, 34].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [25] [25] Y.-S. Kim, K. Reinecke, and J. Hullman. Data through others' eyes: The impact of visualizing others' expectations on visualization interpretation. IEEE transactions on visualization and computer graphics , 24(1):760769, 2017. - source

      Not checked in the free run.

    • cites [26] [26] Y.-S. Kim, L. A. Walls, P. Krafft, and J. Hullman. A bayesian cognition approach to improve data visualization. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems , CHI '19, pages 682:1-682:14, New York, NY, USA, 2019. ACM. - source

      Not checked in the free run.

    • cites [34] [34] A. V. Pandey, A. Manivannan, O. Nov, M. Satterthwaite, and E. Bertini. The persuasive power of data visualization. IEEE transactions on visualization and computer graphics , 20(12):2211-2220, 2014. - source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 4.2 Design · p. 4

    Each condition was replicated 34 times with different participants to increase reliability.

    “We replicated each condition 34 times with different participants to increase the reliability in our measures.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 5.1 Design · p. 7

    Participants who failed the comprehension checks were excluded from analysis because they did not appear to have understood the data.

    “Similar to Experiment 1, participants who failed the comprehension checks were excluded from analysis as they did not appear to have understood the data”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data
    § 4.7 Qualitative Results from Generative Task · p. 6

    There may exist a conventional interpretation of causality in data for the x-axis variable to be seen as the cause and the y-axis variable to be seen as the effect.

    “This suggests that there may exist a conventional interpretation of causality in data for the x-axis variable to be seen as the cause and the y-axis variable to be seen as the cause.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 4.1 Participants · p. 3

    Participants were excluded if they were not based in the United States, had an approval rate below 95%, failed attention checks, entered nonsensical answers for free response questions, or failed graph reading comprehension checks.

    “We excluded workers who are not based in the United States, have an approval rate below 95%, failed the attention checks, entered nonsensical answers for the free response questions or failed the graph reading comprehension checks (details of these checks are included in the supplementary materials).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § Illusion of Causality in Visualized Data · p. 1

    Some visualization designs trigger stronger causal links while choosing others can help mitigate unwarranted perceptions of causality.

    “Our results demonstrate how some visualization designs trigger stronger causal links while choosing others can help mitigate unwarranted perceptions of causality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 2 RELATED WORK · p. 2

    The task people perform when viewing the visualizations may influence the conclusions they draw.

    “The task people perform when viewing the visualizations may also influence the conclusions they draw.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 4.1 Participants · p. 3

    An omnibus power analysis based on pilot effect sizes suggested a target sample of 136 participants would give 95% power to detect an overall difference between visualization designs at alpha level of 0.05.

    “An omnibus power analysis based on pilot effect sizes suggested a target sample of 136 participants would give us 95% power to detect an overall difference between visualization designs at alpha level of 0.05.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisereasoning
    § 2 RELATED WORK · p. 2

    Generative tasks may shed more insights on how participants interpreted data and arrived at possible correlational/causal conclusions, but because they are open-ended, they tend to be more difficult to formally evaluate.

    “Generative tasks may shed more insights on how participants interpreted data and arrived at possible correlational/causal conclusions, but because they are open-ended, they tend to be more difficult to formally evaluate.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data
    § 8 LIMITATIONS AND FUTURE DIRECTIONS · p. 9

    Some participant responses suggested that two-bar bar graphs could be associated with controlled experiments.

    “Some participant responses suggested that two-bar bar graphs could be associated with controlled experiments.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 8 LIMITATIONS AND FUTURE DIRECTIONS · p. 9

    The work took an initial step toward showing that visualizations can be designed to mitigate misinterpretation of correlation and causation.

    “Our work took an initial step toward showing that visualizations can be designed to mitigate misinterpretation of correlation and causation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.
    • Rests on an assumption: The simple and common data displays used in the experiments are representative enough for conclusions about visualization design effects to generalize to real-world visualizations.
    • Rests on an assumption: The results from abstract variable pairings extrapolate to visualizations embedded in context or with explanatory text.

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 8 LIMITATIONS AND FUTURE DIRECTIONS · p. 9

    Dot encoding types, although conventionally associated with non-aggregated raw data, still depict apparent trends in data as participants could mentally draw lines connecting each point.

    “Dot encoding types, although conventionally associated with non-aggregated raw data, still depict apparent trends in data as participants could mentally draw lines connecting each points.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 8 LIMITATIONS AND FUTURE DIRECTIONS · p. 9

    Line encodings were most likely to be associated with causality because line encodings are likely associated with continuous trends in data, which could have made the line encoding marks appear more correlational, and thus more causal.

    “We suspect that line encodings were most likely to be associated with causality because line encodings are likely associated with continuous trends in data, which could have made the line encoding marks appear more correlational, and thus more causal.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 6.1 Design and Procedure · p. 8

    An omnibus power analysis, based on pilot effect sizes, suggested a target sample of 62 would yield enough power to detect an overall difference between visualization designs.

    “An omnibus power analysis, based on pilot effect sizes, suggested a target sample of 62 would yield enough power to detect an overall difference between visualization designs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 6.1 Design and Procedure · p. 8

    The same exclusion criteria as those in Experiment 2 were applied in Experiment 3.

    “We applied the same exclusion criteria as those in Experiment 2.”
    Quote in the paperNot a substantial claim
61 of 61

  • The paper relies on this without support.

    People routinely see causal relationships in data.

    “Yet, people routinely see causal relationships in data.”

  • The paper relies on this without support.

    Confusing correlation with causation is a ubiquitous decision-making error.

    “Confusing correlation with causation is a ubiquitous decisionmaking error.”

  • The paper relies on this without support.

    The judgment task reflects the judgment tasks people would perform in real life.

    “This task reflects the judgment tasks people would perform in real life.”

  • The paper relies on this without support.

    Participants' agreement ratings with correlation and causation statements measure their perceived correlation and causality in the visualized data.

    “The dependent variables are correlation ratings and causation ratings, similar to Experiment 1.”

  • The paper relies on this without support.

    The simple and common data displays used in the experiments are representative enough for conclusions about visualization design effects to generalize to real-world visualizations.

    “The present study relied on simple and common data displays, but future work could test more complex displays like dashboards with multiple displays.”

  • The paper relies on this without support.

    The results from abstract variable pairings extrapolate to visualizations embedded in context or with explanatory text.

    “Some of our studies also relied on displays that were free of context (abstract variable names), and future work should confirm that the results extrapolate to visualizations embedded in context or with explanatory text.”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Discussion denies Experiment 1's significant context effect

    Major

    The limitations section says Experiment 1 found no significant effect of context, whereas its results report a relatively large effect with p < 0.001.

  • Experiment 2's 16-bin mean differs between sections

    Major

    Section 6.3 says its comparison uses the 16-bin condition from Experiment 2 and gives the Experiment 2 mean as 77.16. Section 5.2 reports 63.74 as the average causality rating for that 16-bin condition.

  • Experiment 3 comparison reverses the aggregation labels and direction

    Minor

    The comparison first identifies Experiment 3 as non-aggregated and Experiment 2 as aggregated, then labels them the other way around. It also says the group rated 50.24 was perceived as more causal than the group rated 77.16.

  • Y-axis variable is called both effect and cause

    Minor

    Adjacent statements describe the y-axis variable as the effect and then as the cause under the same X-to-Y interpretation.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. 2018, Heatwave Is there more crime in hot weather?
  2. 2011, Money buys happiness
  3. 2013, I Qualtrics
  4. bates, 2005, Fitting linear mixed models in r, R News, 5, 27
  5. ibrahim, 2016, Using graphs to debias misinformation
  6. kahneman, 2011, Thinking fast and slow Chapter 16 Causes Trump Statistics, 1
  7. guo, 2016, Researchers have debunked one of our most basic assumptions about how the world works

Cited by

Sorted by citations

Related

Authors
Xiong, C., Shapiro, J., Hullman, J., & Franconeri, S.
Affiliations
Northwestern University (US)
ORCID
Venue
IEEE Transactions on Visualization and Computer Graphics
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2020-01
Volume
26
Issue
1
Pages
853-862
ISSN
  • 1077-2626 (ISSN-L)
  • 1941-0506
  • 2160-9306
DOI
10.1109/tvcg.2019.2934399
OpenAlex
W2969737323
Semantic Scholar
46b08e12dce8294690eda3f1f8940a800baa0e33
Cited by
71
References
47
Citations by year
2026: 5 · 2025: 11 · 2024: 13 · 2023: 15 · 2022: 8 · 2021: 9 · 2020: 8 · 2019: 2
FWCI
3.02
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
  • Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
Keywords
correlation versus causation, data visualization, perceived causality, data aggregation, bar charts, scatter plots, reasoning errors, crowdsourced experiments

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review