The DOI matches its Crossref record; 97 of the 97 references it cites were found in a registry or an index.
Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies
Open accessAbstract
AI systems are adopted in numerous domains due to their increasingly strong predictive performance. However, in high-stakes domains such as criminal justice and healthcare, full automation is often not desirable due to safety, ethical, and legal concerns, yet fully manual approaches can be inaccurate and time-consuming. As a result, there is growing interest in the research community to augment human decision making with AI assistance. Besides developing AI technologies for this purpose, the emerging field of human-AI decision making must embrace empirical approaches to form a foundational understanding of how humans interact and work with AI to make decisions. To invite and help structure research efforts towards a science of understanding and improving human-AI decision making, we survey recent literature of empirical human-subject studies on this topic. We summarize the study design choices made in over 100 papers in three important aspects: (1) decision tasks, (2) AI assistance elements, and (3) evaluation metrics. For each aspect, we summarize current trends, discuss gaps in current practices of the field, and make a list of recommendations for future research. Our work highlights the need to develop common frameworks to account for the design and research spaces of human-AI decision making, so that researchers can make rigorous choices in study design, and the research community can build on each other’s work and produce generalizable scientific knowledge. We also hope this work will serve as a bridge for HCI and AI communities to work together to mutually shape the empirical science and computational technologies for human-AI decision making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there is a limited understanding around how the important task characteristics of complexity and uncertainty shape human decision-making and human-AI team performance.”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [68]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Both of these have been studied in prior work on AI-assisted decision making [54], as well as specifically for studying the effect of uncertainty expression [102].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“Recently, there are growing interest in providing documentation or an “About Me” page to present such global information e.g., Model cards [78], Fact-Sheets [7].”
BackgroundAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 6×
“The most commonly used metric is forward simulation [1, 2, 15, 21, 24, 36, 48, 80, 81, 83, 87, 105, 111], by asking participants to simulate a model’s predictions on unseen instances. and what design impacts the perception is an active research area.”
BackgroundMethodInfluentialAlgorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
He, S., & Lau, R. W. H. (2021) · Computer Graphics Forum · doi:10.1111/(issn)1467-8659 · Cited by 1,236
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · cited 9×
“…mturk [1, 9, 11, 15, 21, 23, 24, 26, 29, 30, 32, 35, 38, 41, 43–45, 47, 60–62, 71, 73, 74, 79, 82, 83, 86, 91, 94, 110, 112], university [24, 30, 48, 57, 61, 63, 68, 72, 75, 81, 82, 99, 106, 111], prolific [2, 36], company [8, 38, 56, 73, 75, 108], qualitrics [84], local community [12, 58],…”
BackgroundMethodInfluentialAmazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044 · cited 1×
“This misalignment is analogous to the discrepancies between the recent burst of COVID-related machine learning papers and clinical practices [88].”
Result
Research works printed but not linked: 16
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica. See https://www. propublica. org/article/machine-bias-risk-assessments-in-criminal-sentencing (2016).
- Suresh Kumar Annappindi. 2014. System and method for predicting consumer credit risk using income risk based credit score. US Patent 8,799,150.
- Lee J Cronbach and Paul E Meehl. 1955. Construct validity in psychological tests.Psychological bulletin 52, 4 (1955), 281.
- Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114.
- Steven E Dilsizian and Eliot L Siegel. 2014. Artificial intelligence in medicine and cardiac imaging: harnessing big data and advanced computing to provide personalized medical diagnosis and treatment. Current cardiology reports 16, 1 (2014), 441.
- Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 (2018), eaao5580.
- Ben Green. 2021. The Flaws of Policies Requiring Human Oversight of Government Algorithms. Available at SSRN (2021).
- 10.2139/ssrn.3465622 · DOI · Paper page
- Charles H Lawshe. 1975. A quantitative approach to content validity. Personnel psychology 28, 4 (1975), 563–575.
- John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
- Piyawat Lertvittayakumjorn and Francesca Toni. 2021. Explanation-Based Human Debugging of NLP Models: A Survey. arXiv preprint arXiv:2104.15135 (2021). · arXiv
- Han Liu, Vivian Lai, and Chenhao Tan. 2021. Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making. arXiv preprint arXiv:2101.05303 (2021). · arXiv
- Mahsan Nourani, Chiradeep Roy, Jeremy E Block, Donald R Honeycutt, Tahrima Rahman, Eric Ragan, and Vibhav Gogate. 2021. Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems. In 26th International Conference on Intelligent User Interfaces. 340–350.
- Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (2018). · arXiv
- Peter Schmidt and Ann D Witte. 1988. Predicting recidivism in north carolina, 1978 and 1980. Inter-university Consortium for Political and Social Research.
- Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making. arXiv preprint arXiv:2001.02114 (2020). · arXiv
Cited by
Sorted by citations
When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“This result complements the qualitative literature reviews on human–AI collaboration 27–29 , which highlight some of the surprising challenges that arise when integrating human intelligence and AI.”
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
Exploring automation bias in human–AI collaboration: a review and implications for explainable AI
Romeo, G., & Conti, D. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02422-7 · Cited by 209
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Self-citation
Metadata
- Authors
- Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C.
- Affiliations
- University of Colorado Boulder (US)
- University of Chicago (US)
- Dataminr Inc., USA
- Microsoft (Canada) (CA)
- Venue
- 2023 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2023-06-12
- Pages
- 1369-1385
- OpenAlex
- W4380365974
- Cited by
- 204
- References
- 77
- Citations by year
- 2026: 83 · 2025: 71 · 2024: 45 · 2023: 4
- FWCI
- 101.59
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Funders
- National Science Foundation (2126602, 2040989)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- human-AI decision-making, AI assistance, research design, decision tasks, evaluation metrics, human-AI interaction, human-in-the-loop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
