Uncertainty Displays Using Quantile Dotplots or CDFs Improve Transit Decision-Making
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Everyday predictive systems typically present point predictions, making it hard for people to account for uncertainty when making decisions. Evaluations of uncertainty displays for transit prediction have assessed people's ability to extract probabilities, but not the quality of their decisions. In a controlled, incentivized experiment, we had subjects decide when to catch a bus using displays with textual uncertainty, uncertainty visualizations, or no-uncertainty (control). Frequency-based visualizations previously shown to allow people to better extract probabilities (quantile dotplots) yielded better decisions. Decisions with quantile dotplots with 50 outcomes were(1) better on average, having expected payoffs 97% of optimal(95% CI: [95%,98%]), 5 percentage points more than control (95% CI: [2,8]); and (2) more consistent, having within-subject standard deviation of 3 percentage points (95% CI:[2,4]), 4 percentage points less than control (95% CI: [2,6]).Cumulative distribution function plots performed nearly as well, and both outperformed textual uncertainty, which was sensitive to the probability interval communicated. We discuss implications for real time transit predictions and possible generalization to other domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“displaying uncertainty in predictions can help users can make more optimal decisions [19, 22, 25, 37, 60]”
From Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021) · cites this paper as [22]
Citations
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · cited 1×
“A large body of research in judgment under uncertainty and promoted most famously by Tversky and Kahneman [32] shows, for example, how people have trouble understanding statistical principles that govern uncertainty like the relationship between sample size and variance [31].”
BackgroundCribari-Neto, F., & Zeileis, A. (2010) · Journal of Statistical Software · doi:10.18637/jss.v034.i02 · Cited by 2,505 · cited 1×
“Because some values of expected/optimal payoff may be exactly 1 (when the participant makes the optimal choice), and beta regression requires outcomes on (0,1) exclusive, we use a correction for values equal to 1 [3]: we replace those values with expected payoff/(optimal expected payoff +1).”
MethodHow to improve Bayesian reasoning without instruction: Frequency formats.
Gigerenzer, G., & Hoffrage, U. (1995) · Psychological Review · doi:10.1037//0033-295x.102.4.684 · Cited by 2,127
Cleveland, W. S., & McGill, R. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478080 · Cited by 1,788
A better lemon squeezer? maximum-likelihood regression with beta-distributed dependent variables.
Smithson, M., & Verkuilen, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.1.54 · Cited by 1,788
Davis, D. D., & Holt, C. A. (1993) · Princeton University Press eBooks · doi:10.1515/9780691233376 · Cited by 971
Zhang, H., & Maloney, L. T. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00001 · Cited by 679
Design Features of Graphs in Health Risk Communication: A Systematic Review
Ancker, J. S., Senathirajah, Y., Kukafka, R., & Starren, J. B. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2115 · Cited by 548
Understanding Contemporary Society: Theories of the Present
Browning, G., Halcli, A., & Webster, F. (2000) · SAGE Publications Ltd eBooks · doi:10.4135/9781446218310 · Cited by 284
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239 · cited 11×
“[20] developed and evaluated visualizations depicting uncertainty in bus arrival time predictions.”
BackgroundMethodInfluentialSelf-citationUsing the Box-Cox t distribution in GAMLSS to model skewness and kurtosis
Rigby, R. A., & Stasinopoulos, D. M. (2006) · Statistical Modelling · doi:10.1191/1471082x06st122oa · Cited by 212
OneBusAway: results from providing real-time arrival information for public transit
Ferris, B., Watkins, K., & Borning, A. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753597 · Cited by 204
Do Icon Arrays Help Reduce Denominator Neglect?
Garcia-Retamero, R., Galesic, M., & Gigerenzer, G. (2010) · Medical Decision Making · doi:10.1177/0272989x10369000 · Cited by 204
Hullman, J., Resnick, P., & Adar, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142444 · Cited by 194 · cited 2×
“Perhaps due to the advantages of frequency-based representations for reasoning about uncertainty [9, 12], quantile dotplots with 50 outcomes led to better decisions than CDFs, if slightly.”
BackgroundMethodSelf-citationUncertainty forecasts improve weather-related decisions and attenuate the effects of forecast error.
Joslyn, S. L., & LeClerc, J. E. (2012) · Journal of Experimental Psychology: Applied · doi:10.1037/a0025185 · Cited by 180
Communicating forecast uncertainty: public perception of weather forecast uncertainty
Joslyn, S., & Savelli, S. (2010) · Meteorological Applications · doi:10.1002/met.190 · Cited by 175
Graphical Communication of Uncertain Quantities to Nontechnical People
Ibrekk, H., & Morgan, M. G. (1987) · Risk Analysis · doi:10.1111/j.1539-6924.1987.tb00488.x · Cited by 159
Guidelines for the Display of Attribute Certainty
Leitner, M., & Buttenfield, B. P. (2000) · Cartography and Geographic Information Science · doi:10.1559/152304000783548037 · Cited by 124
Dynamic display of spatial data-reliability: Does it benefit the map user?
Evans, B. J. (1997) · Computers & Geosciences · doi:10.1016/s0098-3004(97)00011-3 · Cited by 120
Kay, M., Nelson, G. L., & Hekler, E. B. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858465 · Cited by 113 · cited 2×
“After Kay et al. 2016 [22], we use a mixed-effects Bayesian regression model with weakly-informed priors.”
MethodSelf-citationJung, M. F., Sirkin, D., Gür, T. M., & Steinert, M. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702479 · Cited by 112
Decisions With Uncertainty: The Glass Half Full
Joslyn, S., & LeClerc, J. (2013) · Current Directions in Psychological Science · doi:10.1177/0963721413481473 · Cited by 109
A Laboratory Study of the Benefits of Including Uncertainty Information in Weather Forecasts
Roulston, M. S., Bolton, G. E., Kleit, A. N., & Sears-Collins, A. L. (2006) · Weather and Forecasting · doi:10.1175/waf887.1 · Cited by 97
Uncertainty Forecasts Improve Decision Making Among Nonexperts
Nadav-Greenberg, L., & Joslyn, S. L. (2009) · Journal of Cognitive Engineering and Decision Making · doi:10.1518/155534309x474460 · Cited by 87
Savelli, S., & Joslyn, S. (2013) · Applied Cognitive Psychology · doi:10.1002/acp.2932 · Cited by 75
Windschitl, P. D., & Weber, E. U. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1304758 · Cited by 73
Natural Language Generation enhances human decision-making with uncertain information
Gkatzia, D., Lemon, O., & Rieser, V. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2043 · Cited by 59
There's no such thing as gaining a pound: reconsidering the bathroom scale user interface
Kay, M., Morris, D., schraefel, M., & Kientz, J. A. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493456 · Cited by 54
Self-citationWu, H., Lindell, M. K., & Prater, C. S. (2015) · Natural Hazards · doi:10.1007/s11069-015-1846-z · Cited by 44
Ishak, E. W., & Feiner, S. (2006) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1166253.1166277 · Cited by 36
Measuring the vague meanings of probability terms.
Wallsten, T. S., & Al, E. (1986) · Journal of Experimental Psychology General · doi:10.1037//0096-3445.115.4.348 · Cited by 28
Visualization of Delay Uncertainty and its Impact on Train Trip Planning: A Design Study
Wunderlich, M., Ballweg, K., Fuchs, G., & von Landesberger, T. (2017) · Computer Graphics Forum · doi:10.1111/cgf.13190 · Cited by 26
Understanding clinical and non-clinical decisions under uncertainty: a scenario-based survey
The Writing Group for CERTAIN-CHOICES, Simianu, V. V., Grounds, M. A., Joslyn, S. L., LeClerc, J. E., Ehlers, A. P., Agrawal, N., Alfonso‐Cristancho, R., Flaxman, A. D., & Flum, D. R. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0391-3 · Cited by 24
Investigating Representation Alternatives for Communicating Uncertainty to Non-experts
Greis, M., Ohler, T., Henze, N., & Schmidt, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22723-8_21 · Cited by 19
Statistical Challenges in Modern Astronomy
Coakley, K. J., Feigelson, E. D., & Babu, G. J. (1994) · Technometrics · doi:10.1080/00401706.1994.10485823 · Cited by 12
Statistical Challenges in Modern Astronomy
Coakley, K. J., Feigelson, E. D., & Babu, G. J. (1994) · Technometrics · doi:10.2307/1269390 · Cited by 6
Research works printed but not linked: 4
- 10.1037/0033-295X.102.4.684 · doi:10.1037/0033-295x.102.4.684 · Paper page
- James J. Chudley Jesmond J. Allen. 2012. Smashing UX Design: Foundations for Designing Online User Experiences. Vol. 1st edition. Wiley.
- Leland Wilkinson. 1999. Dot Plots. The American Statistician 53, 3 (August 1999), 276--281.
- Paul Windschitl and Elke Weber. 1999. The Interpretation of 'Likely' Depends on the Context, But 70Perceived Certainty. 25 (12 1999), 1514--33.
Cited by
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“Second, displaying uncertainty in predictions can help users can make more optimal decisions [19, 22, 25, 37, 60].”
BackgroundThe Science of Visual Data Communication: What Works
Franconeri, S. L., Padilla, L. M., Shah, P., Zacks, J. M., & Hullman, J. (2021) · Psychological Science in the Public Interest · doi:10.1177/15291006211051956 · Cited by 360
Self-citationKocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277
In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation
Hullman, J., Qiao, X., Correll, M., Kale, A., & Kay, M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864889 · Cited by 187
“Studies that measured decision quality against utility theory were most likely to provide feedback on the uncertain outcomes they asked participants about [23, 48, 68], though some studies that used simpler error measures also incorporated feedback for realism and calibration [9].”
BackgroundInfluentialSelf-citation
Metadata
- Authors
- Fernandes, M., Walls, L., Munson, S., Hullman, J., & Kay, M.
- Affiliations
- University of Washington (US)
- University of Michigan (US)
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-19
- Pages
- 1-12
- OpenAlex
- W2799246381
- Cited by
- 175
- References
- 37
- Citations by year
- 2026: 12 · 2025: 18 · 2024: 23 · 2023: 22 · 2022: 31 · 2021: 29 · 2020: 21 · 2019: 12 · 2018: 7
- FWCI
- 5.27
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- Google (Faculty Research Award)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- uncertainty visualization, decision making under uncertainty, bus arrival prediction, expected payoff, decision consistency, real-time transit information, probability communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.