Cited by 206Human-Automation Interaction and SafetyExplainable Artificial Intelligence (XAI)Usability and User Interface Design

The DOI matches its Crossref record; 97 of the 97 references it cites were found in a registry or an index.

Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies

Open access

Lai, Vivian, Chen, Chacha, Smith-Renner, Alison, Liao, Q. Vera, Tan, Chenhao·2023·2023 ACM Conference on Fairness Accountability and Transparency

DOI 10.1145/3593013.3594087

Open this paper in CoChat — free account Open PDF

Abstract

AI systems are adopted in numerous domains due to their increasingly strong predictive performance. However, in high-stakes domains such as criminal justice and healthcare, full automation is often not desirable due to safety, ethical, and legal concerns, yet fully manual approaches can be inaccurate and time-consuming. As a result, there is growing interest in the research community to augment human decision making with AI assistance. Besides developing AI technologies for this purpose, the emerging field of human-AI decision making must embrace empirical approaches to form a foundational understanding of how humans interact and work with AI to make decisions. To invite and help structure research efforts towards a science of understanding and improving human-AI decision making, we survey recent literature of empirical human-subject studies on this topic. We summarize the study design choices made in over 100 papers in three important aspects: (1) decision tasks, (2) AI assistance elements, and (3) evaluation metrics. For each aspect, we summarize current trends, discuss gaps in current practices of the field, and make a list of recommendations for future research. Our work highlights the need to develop common frameworks to account for the design and research spaces of human-AI decision making, so that researchers can make rigorous choices in study design, and the research community can build on each other’s work and produce generalizable scientific knowledge. We also hope this work will serve as a bridge for HCI and AI communities to work together to mutually shape the empirical science and computational technologies for human-AI decision making.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1145/3593013.3594087

Cites

97 of 97 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“Recently, there are growing interest in providing documentation or an “About Me” page to present such global information e.g., Model cards [78], Fact-Sheets [7].”
Background
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 6×
“The most commonly used metric is forward simulation [1, 2, 15, 21, 24, 36, 48, 80, 81, 83, 87, 105, 111], by asking participants to simulate a model’s predictions on unseen instances. and what design impacts the perception is an active research area.”
BackgroundMethodInfluential
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Computer Graphics Forum
He, S., & Lau, R. W. H. (2021) · Computer Graphics Forum · doi:10.1111/(issn)1467-8659 · Cited by 1,236
Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · cited 9×
“…mturk [1, 9, 11, 15, 21, 23, 24, 26, 29, 30, 32, 35, 38, 41, 43–45, 47, 60–62, 71, 73, 74, 79, 82, 83, 86, 91, 94, 110, 112], university [24, 30, 48, 57, 61, 63, 68, 72, 75, 81, 82, 99, 106, 111], prolific [2, 36], company [8, 38, 56, 73, 75, 108], qualitrics [84], local community [12, 58],…”
BackgroundMethodInfluential
Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044 · cited 1×
“This misalignment is analogous to the discrepancies between the recent burst of COVID-related machine learning papers and clinical practices [88].”
Result

Show all 77 references

Research works printed but not linked: 16

Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica. See https://www. propublica. org/article/machine-bias-risk-assessments-in-criminal-sentencing (2016).
Suresh Kumar Annappindi. 2014. System and method for predicting consumer credit risk using income risk based credit score. US Patent 8,799,150.
Lee J Cronbach and Paul E Meehl. 1955. Construct validity in psychological tests.Psychological bulletin 52, 4 (1955), 281.
Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114.
Steven E Dilsizian and Eliot L Siegel. 2014. Artificial intelligence in medicine and cardiac imaging: harnessing big data and advanced computing to provide personalized medical diagnosis and treatment. Current cardiology reports 16, 1 (2014), 441.
Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 (2018), eaao5580.
Ben Green. 2021. The Flaws of Policies Requiring Human Oversight of Government Algorithms. Available at SSRN (2021).
10.2139/ssrn.3465622 · DOI · Paper page
Charles H Lawshe. 1975. A quantitative approach to content validity. Personnel psychology 28, 4 (1975), 563–575.
John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
Piyawat Lertvittayakumjorn and Francesca Toni. 2021. Explanation-Based Human Debugging of NLP Models: A Survey. arXiv preprint arXiv:2104.15135 (2021). · arXiv
Han Liu, Vivian Lai, and Chenhao Tan. 2021. Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making. arXiv preprint arXiv:2101.05303 (2021). · arXiv
Mahsan Nourani, Chiradeep Roy, Jeremy E Block, Donald R Honeycutt, Tahrima Rahman, Eric Ragan, and Vibhav Gogate. 2021. Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems. In 26th International Conference on Intelligent User Interfaces. 340–350.
Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (2018). · arXiv
Peter Schmidt and Ann D Witte. 1988. Predicting recidivism in north carolina, 1978 and 1980. Inter-university Consortium for Political and Social Research.
Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making. arXiv preprint arXiv:2001.02114 (2020). · arXiv

Sorted by citations

When combinations of humans and AI are useful: A systematic review and meta-analysis
Vaccaro, M., Almaatouq, A., & Malone, T. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02024-1 · Cited by 570
“This result complements the qualitative literature reviews on human–AI collaboration 27–29 , which highlight some of the surprising challenges that arise when integrating human intelligence and AI.”
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
Exploring automation bias in human–AI collaboration: a review and implications for explainable AI
Romeo, G., & Conti, D. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02422-7 · Cited by 209
"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Self-citation
Complementarity in human-AI collaboration: concept, sources, and evidence
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2475962 · Cited by 185
Human-AI collaboration is not very collaborative yet: a taxonomy of interaction patterns in AI-assisted decision making from a systematic review
Gómez, C., Cho, S. M., Ke, S., Huang, C., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 207
AI auditing: The Broken Bus on the Road to AI Accountability
Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00037 · Cited by 158
On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives
Sterz, S., Baum, K., Biewer, S., Hermanns, H., Lauber-Rönsberg, A., Meinel, P., & Langer, M. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659051 · Cited by 91
Where are we so far? Understanding data storytelling tools from the perspective of human-AI collaboration
Li, H., Wang, Y., & Qu, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642726 · Cited by 77
Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making
Schoeffer, J., De‐Arteaga, M., & Kühl, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642621 · Cited by 105
Untitled work
(2024) · doi:10.1111/cgf.15107 · Cited by 53
Towards Responsible Urban Geospatial AI: Insights From the White and Grey Literatures
Marasinghe, R., Yiğitcanlar, T., Mayere, S., Washington, T. L., & Limb, M. (2024) · Journal of Geovisualization and Spatial Analysis · doi:10.1007/s41651-024-00184-2 · Cited by 64
The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive Visualization
Hoque, M. N., Mashiat, T., Ghai, B., Shelton, C. D., Chevalier, F., Kraus, K., & Elmqvist, N. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641895 · Cited by 48
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 87
Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“For instance, cognitive [35, 68, 96] or meta-cognitive biases [51] have the potential to influence how individuals comprehend and appraise the outcomes generated by AI systems which in turn can affect their reliance on AI.”
BackgroundResultInfluential
One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations
Lee, Y., Son, K., Kim, T. S., Kim, J., Chung, J. J. Y., Adar, E., & Kim, J. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3662681 · Cited by 36
As Confidence Aligns: Understanding the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making
Li, J., Yang, Y., Liao, Q. V., Zhang, J., & Lee, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713336 · Cited by 63
Self-citation
Trust in AI-assisted Decision Making: Perspectives from Those Behind the System and Those for Whom the Decision is Made
Vereschak, O., Alizadeh, F., Bailly, G., & Caramiaux, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642018 · Cited by 50
Trusting emotional support from generative artificial intelligence: a conceptual review
Volpato, R., DeBruine, L. M., & Stumpf, S. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100195 · Cited by 52
Whither bias goes, I will go: An integrative, systematic review of algorithmic bias mitigation.
Hickman, L., Huynh, C., Gass, J., Booth, B. M., Kuruzovich, J., & Tay, L. (2024) · Journal of Applied Psychology · doi:10.1037/apl0001255 · Cited by 36
AI, Help Me Think—but for Myself: Assisting People in Complex Decision-Making by Providing Different Kinds of Cognitive Support
Reicherts, L., Zhang, Z. T., von Oswald, E., Liu, Y., Rogers, Y., & Hassib, M. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713295 · Cited by 48
Interaction Design With Multi-Objective Bayesian Optimization
Liao, Y.-C., Dudley, J. J., Mo, G. B., Cheng, C.-L., Chan, L., Oulasvirta, A., & Kristensson, P. O. (2023) · IEEE Pervasive Computing · doi:10.1109/mprv.2022.3230597 · Cited by 25
Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
He, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23
A Human–AI interaction paradigm and its application to rhinocytology
Desolda, G., Dimauro, G., Esposito, A., Lanzilotti, R., Matera, M., & Zancanaro, M. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102933 · Cited by 25
To Search or To Gen? Exploring the Synergy between Generative AI and Web Search in Programming
Yen, R., Sultanum, N., & Zhao, J. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650867 · Cited by 26
To Trust or Distrust AI: A Questionnaire Validation Study
Scharowski, N., Perrig, S. A. C., von Felten, N., Aeschbach, L. F., Opwis, K., Wintersberger, P., & Brühlmann, F. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732025 · Cited by 25
Measuring the effect of mental workload and explanations on appropriate AI reliance using EEG
Zhang, Z. T., Argın, S. K., Bilen, M. B., Urgun, D., Deniz, S. M., Liu, Y., & Hassib, M. E. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2431055 · Cited by 22
Confronting verbalized uncertainty: Understanding how LLM's verbalized uncertainty influences users in AI-assisted decision-making
Xu, Z., Song, T., & Lee, Y.-C. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103455 · Cited by 18
Conversational Agents as Catalysts for Critical Thinking: Challenging Social Influence in Group Decision-making
Lee, S., Hwang, S., Kim, D.-J., & Lee, K. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719792 · Cited by 30
Do Expressions Change Decisions? Exploring the Impact of AI's Explanation Tone on Decision-Making
Okoso, A., Yang, M., & Baba, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713744 · Cited by 26
Stereotypical bias amplification and reversal in an experimental model of human interaction with generative artificial intelligence
Allan, K., Azcona, J., Sripada, S. G., Leontidis, G., Sutherland, C. A. M., Phillips, L. H., & Martin, D. S. (2025) · Royal Society Open Science · doi:10.1098/rsos.241472 · Cited by 19
Generating textual explanations for scheduling systems leveraging the reasoning capabilities of large language models
Powell, C., & Riccardi, A. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00940-w · Cited by 19
A Survey of AI Reliance
Eckhardt, S., Kühl, N., Dolata, M., & Schwabe, G. (2025) · ACM Computing Surveys · doi:10.1145/3776528 · Cited by 38
When Are Combinations of Humans and AI Useful?Preprint
Vaccaro, M. A., Almaatouq, A., & Malone, T. W. (2024) · doi:10.31234/osf.io/mk46p · Cited by 16
Beyond Recommendations: From Backward to Forward AI Support of Pilots' Decision-Making Process
Zhang, Z. T., Feger, S. S., Dullenkopf, L., Liao, R., Süsslin, L. E., Liu, Y., & Butz, A. M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687024 · Cited by 22
The role of user feedback in enhancing understanding and trust in counterfactual explanations for explainable AI
Suffian, M., Kuhl, U., Bogliolo, A., & Alonso-Moral, J. M. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103484 · Cited by 18
Applicants’ Fairness Perception of Human and AI Collaboration in Resume Screening
Ling, B., Dong, B., & Cai, F. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2437235 · Cited by 23
Applying Design Thinking to Develop AI-Based Multi-Actor Decision-Support Systems: A Case Study on Human Capital Investments
Marocco, S., Talamo, A., & Quintiliani, F. (2024) · Applied Sciences · doi:10.3390/app14135613 · Cited by 14
The Explanation That Hits Home: The Characteristics of Verbal Explanations That Affect Human Perception in Subjective Decision-Making
Ferguson, S. A., Aoyagui, P. A., Rizvi, R., Kim, Y., & Kuzminykh, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687056 · Cited by 17
Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on their Long-Term Collaboration with an AI Planning System
Kahr, P. K., Rooks, G., Snijders, C. C. P., & Willemsen, M. C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713099 · Cited by 19
Algorithmic loafing and mitigation strategies in Human-AI teams
Inuwa-Dutse, I., Toniolo, A., Weller, A., & Bhatt, U. (2023) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2023.100024 · Cited by 17
LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems
Li, C., Zhang, Z., Saugstad, M., Safranchik, E., Kulkarni, M., Huang, X., Patel, S., Iyer, V., Althoff, T., & Froehlich, J. E. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642089 · Cited by 15
How Do People Develop Folk Theories of Generative AI Text-to-Image Models? A Qualitative Study on How People Strive to Explain and Make Sense of GenAI
Lodovico, C. D., Torrielli, F., Di, L., & Rapp, A. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2491009 · Cited by 13
Fair Machine Guidance to Enhance Fair Decision Making in Biased People
Yang, M., Arai, H., Yamashita, N., & Baba, Y. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642627 · Cited by 16
When in Doubt! Understanding the Role of Task Characteristics on Peer Decision-Making with AI Assistance
Salimzadeh, S., & Gadiraju, U. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3627043.3659567 · Cited by 14
Towards Understanding AI Delegation: The Role of Self-Efficacy and Visual Processing Ability
Westphal, M., Hemmer, P., Vössing, M., Schemmer, M., Vetter, S., & Satzger, G. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3696423 · Cited by 13
Diagnostic Decision-Making Variability Between Novice and Expert Optometrists for Glaucoma: Comparative Analysis to Inform AI System Design
Ghaffar, F., Furtado, N. M., Ali, I., & Burns, C. M. (2024) · JMIR Medical Informatics · doi:10.2196/63109 · Cited by 9
A study on Gen-AI technology development trends to enhance small-medium sized enterprise digital competence and management quality
Cho, Y.-J., Park, J., Yoo, J., Kim, S., & Park, H. (2025) · Journal of Entrepreneurship in Emerging Economies · doi:10.1108/jeee-10-2024-0485 · Cited by 10
The Changing Nature of Human-AI Relations: A Scoping Review on Terminology and Evolvement in the Scientific Literature
Breckner, K., Neumayr, T., Mara, M., Streit, M., & Augstein, M. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2482742 · Cited by 14
Using AI Uncertainty Quantification to Improve Human Decision-MakingPreprint
Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcıoğlu, M. (2023) · doi:10.31234/osf.io/6syfv · Cited by 13
The Sound of Support: Gendered Voice Agent as Support to Minority Teammates in Gender-Imbalanced Team
Hwang, A. H., & Won, A. S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642202 · Cited by 9
Algorithm, expert, or both? Evaluating the role of feature selection methods on user preferences and reliance
Kornowicz, J., & Thommes, K. (2025) · PLoS ONE · doi:10.1371/journal.pone.0318874 · Cited by 14
The Amplifying Effect of Explainability in AI-assisted Decision-making in Groups
Duarte, R. D. B., Abreu, M. C., Campos, J., & Paiva, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713534 · Cited by 13
Designing job characteristics for effective hybrid intelligence: insights from job design and human–AI collaboration
Yang, T., & Jiang, J. (2025) · Personnel Review · doi:10.1108/pr-03-2024-0229 · Cited by 10
On the Complexities of Testing for Compliance with Human Oversight Requirements in AI Regulation
Langer, M., Lazar, V., & Baum, K. L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07132-3_11 · Cited by 12
The Influence of Curiosity Traits and On-Demand Explanations in AI-Assisted Decision-Making
Cau, F. M., & Spano, L. D. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712165 · Cited by 13
Negotiating the Shared Agency between Humans & AI in the Recommender System
Wu, M., Liu, W., Wang, Y., & Yao, M. Z. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719900 · Cited by 14
Factors influencing the effectiveness of artificial intelligence-assisted decision-making in medicine: a scoping review
Jackson, N. J., Brown, K. E., Rn, M., Murrow, M., Cauley, M. R., Collins, B. X., Novak, L. L., Benda, N. C., & Ancker, J. S. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag002 · Cited by 13
Beyond Individual UX: Defining Group Experience(GX) as a New Paradigm for Group-centered AI
Lee, S., Hwang, S., & Lee, K. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715668.3736348 · Cited by 7
IMPACT OF ARTIFICIAL INTELLIGENCE ON MANAGEMENT DECISION-MAKING
Ashritha, P., & Reddy, P. S. (2024) · International Journal of Advances in Business and Management Research · doi:10.62674/ijabmr.2024.v1i02.002 · Cited by 16
You Can Only Verify When You Know the Answer: Feature-Based Explanations Reduce Overreliance on AI for Easy Decisions, but Not for Hard Ones
Zhang, Z. T., Buchner, F., Liu, Y., & Butz, A. (2024) · Proceedings of Mensch und Computer 2024 · doi:10.1145/3670653.3670660 · Cited by 6
Do people appropriately rely on AI-advice? An analytical review of HCI research on human-AI decision-making.
Raees, M., Khan, V.-J., Lykourentzou, I., & Papangelis, K. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791467 · Cited by 5
Machine learning systems as mentors in human learning: A user study on machine bias transmission in medical training
Vicente, L., Matute, H., Fregosi, C., & Cabitza, F. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103474 · Cited by 14
Who is Responsible, the Advisor or the AI? Understanding the Effects of Advisors Disclosing Their AI Use on Their Perceived Responsibility and AI Reliance
Mendel, T., Mandal, S., Nov, O., & Wiesenfeld, B. M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757524 · Cited by 9
'The AI is uncertain, so am I. What now?': Navigating Shortcomings of Uncertainty Representations in Human-AI Collaboration with Capability-focused Guidance
Schäfer, U., Sipos, L., & Müller-Birn, C. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757451 · Cited by 10
In Search of Clarity: Discerning Between Human Replacement and Augmentation
Alfieri, C., Sanctis, M. D., Donati, D., & Inverardi, P. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240180 · Cited by 6
Bridging Expertise Gaps: The Role of LLMs in Human-AI Collaboration for CybersecurityPreprint
Tariq, S., Singh, R., Chhetri, M. B., Nepal‬, ‪., & Paris, C. L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5462737 · Cited by 6
Balancing Human Agency and AI Autonomy in Human-AI Idea Selection
Young-Ng, M., Zhu, Q., Liao, J., & Wang, H. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719880 · Cited by 8
Exploring the impact of explainable AI and cognitive capabilities on users’ decisions
Cau, F. M., & Spano, L. D. (2025) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-025-09438-0 · Cited by 16
Human–AI Collaboration Across Decision Support, Autonomous Systems, and LLM Agents: A Systematic Review and Collaboration Convergence Framework
Dong, A., Li, P., Chen, Y., Gibson, S., Zhao, L., & He, M. (2026) · Sustainability · doi:10.3390/su18115313 · Cited by 5
Comparing human evaluations of eyewitness statements to a machine learning classifier under pristine and suboptimal lineup administration procedures
Grabman, J. H., Dobbins, I. G., & Dodson, C. S. (2024) · Cognition · doi:10.1016/j.cognition.2024.105876 · Cited by 5
AI Support in Safety-Critical Tasks: How Perceived Autonomy and Meaningfulness Shape Human-AI Joint Performance
Faas, C., Uth, R., Sterz, S., Langer, M., & Feit, A. M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812265 · Cited by 25
Where Are We and Where Can We Go on the Road to Reliance-Aware Explainable User Interfaces?
Filho, J. C. D. S., Belloum, R., & de Oliveira, K. M. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00038 · Cited by 6
From Trust to Appropriate Reliance: Measurement Constructs in Human-AI Decision-Making
Raees, M., & Papangelis, K. (2026) · arXiv · doi:10.48550/arxiv.2604.23896 · Cited by 5
A copula-based human–artificial intelligence collaborative decision-making approach for multi-hazard risk assessment
Li, A., Guo, C., Zhang, J., & Zhao, Q. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.111532 · Cited by 4
Understanding Collaboration between Professional Designers and Decision-making AI: A Case Study in the Workplace
Ogawa, N., Okafuji, Y., Hatada, Y., & Baba, J. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757686 · Cited by 8
Evaluating human–machine collaboration through a comparative analysis of experts, machine learning, and hybrid approaches in real estate valuation
Kmen, C., Navratil, G., Kattenbeck, M., & Giannopoulos, I. (2026) · Scientific Reports · doi:10.1038/s41598-025-34099-9 · Cited by 4
Trust to Reliance: Measurement Constructs for Human-AI Appropriate Reliance
Raees, M., & Papagelis, K. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798835 · Cited by 8
TutorCraftEase: Enhancing Pedagogical Question Creation with Large Language Models
Kang, W., Zhang, L., Peng, X., Zhang, H., Li, A., Wang, M., Huang, J., Tian, F., & Dai, G. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713731 · Cited by 10
Mysterious and manipulative black boxes: A qualitative analysis of perceptions on recommender systems
Ruohonen, J. (2024) · First Monday · doi:10.5210/fm.v29i6.13357 · Cited by 6
Understanding the Effects of Explaining Predictive but Unintuitive Features in Human-XAI Interaction
Qu, J., Arguello, J., & Wang, Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732021 · Cited by 4
Effect of AI Performance, Risk Perception, and Trust on Human Dependence in Deepfake Detection AI System
Zhou, Y., Chen, E., Pisipati, M., Xiong, A., & Rajtmajer, S. M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757588 · Cited by 4
Towards Using Prompt Engineering in Large Language Models to Assist Decision Making
Phillips-Wren, G., & Håkansson, A. (2025) · International Conference on Knowledge-Based Intelligent Information & Engineering Systems · doi:10.1016/j.procs.2025.09.650 · Cited by 5
Design Considerations for Human Oversight of AI: Insights from Co-Design Workshops and Work Design Theory
Faas, C., Kerstan, S., Uth, R., Langer, M., & Feit, A. M. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789100 · Cited by 11
Mapping Human–AI Relationships: Intellectual Structure and Conceptual Insights
Gómez-Cruz, N. A., Castro, D. Y. R., Rey-Sarmiento, C. F., Torres, R. A. Z., & Niño, Á. M. (2026) · Technologies · doi:10.3390/technologies14020083 · Cited by 5
Human-AI Teaming: Following the IMOI Framework
Kleanthous, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60611-3_27 · Cited by 4
AI-assistance to decision-makers: evaluating usability, induced cognitive load, and trust's impact
Souchet, A. D., Amokrane, K., & Burkhardt, J. (2024) · European Conference on Cognitive Ergonomics · doi:10.1145/3673805.3673845 · Cited by 3
Underspecified Human Decision Experiments Considered Harmful
Hullman, J. R., Kale, A. M., & Hartline, J. D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714063 · Cited by 12
Understanding the Effects of AI-based Credibility Indicators When People Are Influenced By Both Peers and Experts
Lu, Z., Li, P., Wang, W., & Yin, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713871 · Cited by 5
Investigating the Capabilities and Limitations of Machine Learning for Identifying Bias in English Language Data with Information and Heritage Professionals
Havens, L., Bach, B., Terras, M., & Alex, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713217 · Cited by 6
Towards Interactive AI-assisted Material Selection for Sustainable Building Design
Zhong, S., Aseniero, B. A., Groom, A., Harsuvanakit, A., Lee, B. J., Zhao, D., & Benjamin, D. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715668.3736354 · Cited by 4
Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-MakingPreprint
Schoeffer, J., De‐Arteaga, M., & Kuehl, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.11812 · Cited by 3
Leveraging Actionable Explanations to Improve People’s Reactions to AI-Based Decisions
Langer, M., & Valera, I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73741-1_18 · Cited by 2
What Lies Beneath? Exploring the Impact of Underlying AI Model Updates in AI-Infused Systems
Mohanty, V., Lim, J., & Luther, K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713751 · Cited by 9
The Enterprising and Elusive Prospects of Human-AI Collaboration
Gadiraju, U., & Balayn, A. (2025) · Springer eBooks · doi:10.1007/978-3-032-01940-0_7 · Cited by 5
An Empirical Examination of the Evaluative AI Framework
Kornowicz, J. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2581260 · Cited by 2
Trusting the Middle in the TNO Trust Task: How Medium Risk and Reliability Shape AI‐Assisted Advice‐Taking
Sun, Z., & Ding, J. (2025) · Human Behavior and Emerging Technologies · doi:10.1155/hbe2/6901297 · Cited by 2
Unpacking human–AI collaboration: conceptualisations and emerging research streams
Wang, Y., de Paula, D. F. O., & Heavin, C. (2026) · Journal of Decision System · doi:10.1080/12460125.2026.2653697 · Cited by 4
A Human-Centric Assessment of the Usefulness of Attribution Methods in Computer Vision
Rim, W. B., Shaker, A., Xu, Z., Gashteovski, K., Kotnis, B., Lawrence, C., Quittek, J. W., & Saralajew, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70362-1_2 · Cited by 2

Authors: Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C.
Affiliations: University of Colorado Boulder (US)
University of Chicago (US)
Dataminr Inc., USA
Microsoft (Canada) (CA)
ORCID: Lai, Vivian
Chen, Chacha
Smith-Renner, Alison
Liao, Q. Vera
Tan, Chenhao
Venue: 2023 ACM Conference on Fairness Accountability and Transparency
Publisher: ACM
Published: 2023-06-12
Pages: 1369-1385
DOI: 10.1145/3593013.3594087
OpenAlex: W4380365974
Cited by: 204
References: 77
Citations by year: 2026: 83 · 2025: 71 · 2024: 45 · 2023: 4
FWCI: 101.59
Type: conference paper
Language: en
Open access: bronze
doi.org
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
Funders: National Science Foundation (2126602, 2040989)
Topics: Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords: human-AI decision-making, AI assistance, research design, decision tasks, evaluation metrics, human-AI interaction, human-in-the-loop

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks