CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

The Rational Agent Benchmark for Data Visualization

Wu, Yifan, Guo, Ziyang, Mamakos, Michails, Hartline, Jason, Hullman, Jessica·2023·arXiv
DOI 10.48550/arxiv.2304.03432arXiv 2304.03432
Open this paper in CoChat — free account

Abstract

Understanding how helpful a visualization is from experimental results is difficult because the observed performance is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. We develop a rational agent framework for designing and interpreting visualization experiments. Our framework conceives two experiments with the same setup: one with behavioral agents (human subjects), and the other one with a hypothetical rational agent. A visualization is evaluated by comparing the expected performance of behavioral agents to that of a rational agent under different assumptions. Using recent visualization decision studies from the literature, we demonstrate how the framework can be used to pre-experimentally evaluate the experiment design by bounding the expected improvement in performance from having access to visualizations, and post-experimentally to deconfound errors of information extraction from errors of optimization, among other analyses.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.03432

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The paper's formal definition of AI reliance is grounded in statistical decision theory.

    “Following previous work on generating benchmarks for studies of information displays [34], our approach is grounded in statistical decision theory.”

    From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [34]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. Alex Kale, Matthew Kay, and Jessica Hullman. Visual reasoning strategies for effect size judgments and decisions. IEEE Transactions on Visualization and Computer Graphics, 27 (2):272-282, 2021. doi: 10.1109/TVCG.2020.3030335. · DOI · Paper page
  2. Michael Fernandes, Logan Walls, Sean Munson, Jessica Hullman, and Matthew Kay. Un- certainty displays using quantile dotplots or cdfs improve transit decision-making. In Pro- ceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18, page 1-12, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450356206. doi: 10.1145/3173574.3173718. URL https://doi.org/10.1145/ 3173574.3173718. · DOI · Paper page
  3. Tobias Isenberg, Petra Isenberg, Jian Chen, Michael Sedlmair, and Torsten Möller. A system- atic review on the practice of evaluating visualization. IEEE Transactions on Visualization and Computer Graphics, 19(12):2818-2827, 2013. doi: 10.1109/TVCG.2013.126. · DOI
  4. Heidi Lam, Enrico Bertini, Petra Isenberg, Catherine Plaisant, and Sheelagh Carpendale. Em- pirical studies in information visualization: Seven scenarios. IEEE Transactions on Visual- ization and Computer Graphics, 18(9):1520-1536, 2012. doi: 10.1109/TVCG.2011.279. · DOI
  5. Torre Zuk and Sheelagh Carpendale. Theoretical analysis of uncertainty visualizations. In Robert F. Erbacher, Jonathan C. Roberts, Matti T. Gröhn, and Katy Börner, editors, Visu- alization and Data Analysis 2006, volume 6060, page 606007. International Society for Optics and Photonics, SPIE, 2006. doi: 10.1117/12.643631. URL https://doi.org/10. 1117/12.643631. · DOI
  6. Petra Isenberg, Torre Zuk, Christopher Collins, and Sheelagh Carpendale. Grounded evalua- tion of information visualizations. In Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization, page 6. ACM, 2008. doi: 10.1145/1377966.1377974. · DOI
  7. Tamara Munzner. A nested model for visualization design and validation. IEEE Transactions on Visualization and Computer Graphics, 15(6):921-928, 2009. doi: 10.1109/TVCG.2009. 111. · DOI
  8. Ben Shneiderman and Catherine Plaisant. Strategies for evaluating information visualiza- tion tools: Multi-dimensional in-depth long-term case studies. In Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Infor- mation Visualization, BELIV '06, page 1-7, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595935622. doi: 10.1145/1168149.1168158. URL https://doi.org/10.1145/1168149.1168158. · DOI
  9. Jessica Hullman and Andrew Gelman. Designing for interactive exploratory data analysis requires theories of graphical inference. Harvard Data Science Review, 3(3), 2021. doi: 10.1162/99608f92.3ab8a587. · DOI
  10. Evanthia Dimara and John Stasko. A critical reflection on visualization research: Where do decision making tasks hide? IEEE Transactions on Visualization and Computer Graphics, 28(1):1128-1138, 2022. doi: 10.1109/TVCG.2021.3114813. · DOI · Paper page
  11. Jessica Hullman, Xiaoli Qiao, Michael Correll, Alex Kale, and Matthew Kay. In pursuit of error: A survey of uncertainty visualization evaluation. IEEE Transactions on Visualization and Computer Graphics, 25(1):903-913, 2019. doi: 10.1109/TVCG.2018.2864889. · DOI · Paper page
  12. Christoph Kinkeldey, Alan M. MacEachren, and Jochen Schiewe. How to assess visual com- munication of uncertainty? a systematic review of geospatial uncertainty visualisation user studies. The Cartographic Journal, 51(4):372-386, 2014. doi: 10.1179/1743277414Y. 0000000099. URL https://doi.org/10.1179/1743277414Y.0000000099. · DOI
  13. Alex Kale, Yifan Wu, and Jessica Hullman. Causal support: Modeling causal inferences with visualizations. IEEE Transactions on Visualization and Computer Graphics, 28(1):1150- 1160, 2022. doi: 10.1109/TVCG.2021.3114824. · DOI
  14. Yea-Seul Kim, Paula Kayongo, Madeleine Grunde-McLaughlin, and Jessica Hullman. Bayesian-assisted inference from visualized data. IEEE Transactions on Visualization and Computer Graphics, 27(2):989-999, 2021. doi: 10.1109/TVCG.2020.3028984. · DOI
  15. Alex Kale, Ziyang Guo, Xiaoli Qiao, Jeffrey Heer, and Jessica Hullman. Evm: Incorporating model checking into exploratory visual analysis. IEEE Transactions on Visualization and Computer Graphics (forthcoming), 2023.
  16. David Knill and Richards Whitman. Perception as Bayesian Inference, chapter 7, pages 825- 837. MIT Press, 1996.
  17. Jake M Hofman, Duncan J Watts, Susan Athey, Filiz Garip, Thomas L Griffiths, Jon Kleinberg, Helen Margetts, Sendhil Mullainathan, Matthew J Salganik, Simine Vazire, et al. Integrating explanation and prediction in computational social science. Nature, 595(7866):181-188, 2021. doi: 10.1038/s41586-021-03659-0. · DOI · Paper page
  18. Mayank Agrawal, Joshua C. Peterson, and Thomas L. Griffiths. Scaling up psychology via scientific regret minimization. Proceedings of the National Academy of Sciences, 117(16): 8825-8835, 2020. doi: 10.1073/pnas.1915841117. URL https://www.pnas.org/doi/ abs/10.1073/pnas.1915841117. · DOI
  19. Drew Fudenberg, Jon M. Kleinberg, Annie Liang, and Sendhil Mullainathan. Measuring the completeness of economic models. Journal of Political Economy, 130:956 -990, 2022. doi: 10.1086/718371. · DOI
  20. Sonia Savelli and Susan Joslyn. The advantages of predictive interval forecasts for non-expert users and the impact of visualizations. Applied Cognitive Psychology, 27(4):527-541, 2013. doi: https://doi.org/10.1002/acp.2932. URL https://onlinelibrary.wiley. com/doi/abs/10.1002/acp.2932. · DOI
  21. Jessica Hullman, Paul Resnick, and Eytan Adar. Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PloS one, 10(11): e0142444, 2015. doi: 10.1371/journal.pone.0142444. · DOI · Paper page
  22. William S. Cleveland and Robert McGill. Graphical perception: Theory, experimentation, and application to the development of graphical methods. Journal of the American Statis- tical Association, 79(387):531-554, 1984. doi: 10.1080/01621459.1984.10478080. URL https://www.tandfonline.com/doi/abs/10.1080/01621459.1984.10478080. · DOI
  23. Cleotilde Gonzalez and Varun Dutt. Instance-based learning: integrating sampling and re- peated decisions from experience. Psychological review, 118(4):523, 2011. doi: 10.1037/ a0024558. · DOI · Paper page
  24. Jessica Hullman. Why authors don't visualize uncertainty. IEEE transactions on visualization and computer graphics, 26(1):130-139, 2019. doi: 10.1109/TVCG.2019.2934287. · DOI
  25. Katherine Button, John Ioannidis, Claire Mokrysz, Brian Nosek, Jonathan Flint, Emma Robin- son, and Marcus Munafò. Power failure: Why small sample size undermines the reliability of neuroscience. Nature reviews. Neuroscience, 14, 04 2013. doi: 10.1038/nrn3475. · DOI · Paper page
  26. Tal Yarkoni. The generalizability crisis. Behavioral and Brain Sciences, 45:e1, 2022. doi: 10.1017/S0140525X20001685. · DOI · Paper page
  27. Robert Coe. It's the effect size, stupid. In British Educational Research Association Annual Conference, volume 12, page 14, 2002.
  28. Matthew Kay, Tara Kola, Jessica R. Hullman, and Sean A. Munson. When (ish) is my bus? user-centered visualizations of uncertainty in everyday, mobile predictive systems. In Pro- ceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16, page 5092-5103, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450333627. doi: 10.1145/2858036.2858558. URL https://doi.org/10.1145/ 2858036.2858558. · DOI · Paper page
  29. Daniel Kahneman and Amos Tversky. Prospect Theory: An Analysis of Decision Un- der Risk, chapter 6, pages 99-127. [Wiley, Econometric Society], 2013. doi: 10. 1142/9789814417358 0006. URL https://www.worldscientific.com/doi/abs/10. 1142/9789814417358_0006. · DOI · Paper page
  30. Christian Heine. Towards modeling visualization processes as dynamic bayesian networks. IEEE Transactions on Visualization and Computer Graphics, 27(2):1000-1010, 2020. doi: 10.1109/TVCG.2020.3030395. · DOI
  31. J.J. van Wijk. The value of visualization. In VIS 05. IEEE Visualization, 2005., pages 79-86, 2005. doi: 10.1109/VISUAL.2005.1532781. · DOI
  32. Yingkai Li, Jason D. Hartline, Liren Shan, and Yifan Wu. Optimization of scoring rules. In Proceedings of the 23rd ACM Conference on Economics and Computation, EC '22, page 988-989, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450391504. doi: 10.1145/3490486.3538338. URL https://doi.org/10.1145/ 3490486.3538338. · DOI
  33. Winter Mason and Duncan J. Watts. Financial incentives and the "performance of crowds". In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '09, page 77-85, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605586724. doi: 10.1145/1600150.1600175. URL https://doi.org/10.1145/ 1600150.1600175. · DOI

Cited by

Sorted by citations

Metadata

Authors
Wu, Y., Guo, Z., Mamakos, M., Hartline, J., & Hullman, J.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2304.03432
arXiv
2304.03432
References
0
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review