CoChat
Sign Up
Cited by 49Human-Automation Interaction and SafetyExplainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and Education

Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure

Open access
Swaroop, Siddharth, Buçinca, Zana, Gajos, Krzysztof Z., Doshi-Velez, Finale·2024·Proceedings of the 29th International Conference on Intelligent User Interfaces
DOI 10.1145/3640543.3645206
Open this paper in CoChat — free accountOpen PDF

Abstract

In settings where users both need high accuracy and are time-pressured, such as doctors working in emergency rooms, we want to provide AI assistance that both increases decision accuracy and reduces decision-making time. Current literature focusses on how users interact with AI assistance when there is no time pressure, finding that different AI assistances have different benefits: some can reduce time taken while increasing overreliance on AI, while others do the opposite. The precise benefit can depend on both the user and task. In time-pressured scenarios, adapting when we show AI assistance is especially important: relying on the AI assistance can save time, and can therefore be beneficial when the AI is likely to be right. We would ideally adapt what AI assistance we show depending on various properties (of the task and of the user) in order to best trade off accuracy and time. We introduce a study where users have to answer a series of logic puzzles. We find that time pressure affects how users use different AI assistances, making some assistances more beneficial than others when compared to no-time-pressure settings. We also find that a user’s overreliance rate is a key predictor of their behaviour: overreliers and not-overreliers use different AI assistance types differently. We find marginal correlations between a user’s overreliance rate (which is related to the user’s trust in AI recommendations) and their personality traits (Big Five Personality traits). Overall, our work suggests that AI assistances have different accuracy-time tradeoffs when people are under time pressure compared to no time pressure, and we explore how we might adapt AI assistances in this setting.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3640543.3645206

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI-after conditions reduce overreliance on hard tasks at a modest time cost.

    “consistent with experimental evidence that 'AI-after' conditions reduce overreliance on hard tasks at a modest time cost (Swaroop et al. 2024).”

    From From Oracles to Productive Friction: Designing for Meaningful Human Oversight in Clinical AI (2026) · cites this paper as Swaroop et al. 2024

    Section 5.2 reports that, under time pressure, AI-after improved overreliers’ accuracy on hard questions from 55% to 70% by slowing them down. The paper’s results tables show lower overreliance with AI-after overall, but the cited passages do not establish a reduction in overreliance specifically on hard questions or characterize the additional time as modest. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Withholding AI assistance until after the user forms an independent judgment reduces overreliance compared to presenting it upfront, though at the cost of decision time.

    “Withholding AI assistance until after the user forms an independent judgment reduces overreliance compared to presenting it upfront, though at the cost of decision time(Swaroop et al. 2024).”

    From From Oracles to Productive Friction: Designing for Meaningful Human Oversight in Clinical AI (2026) · cites this paper as Swaroop et al. 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. Saleema Amershi, Daniel S. Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi T. Iqbal, Paul N. Bennett, Kori Inkpen Quinn, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019).
  2. Syed Z Arshad, Jianlong Zhou, Constant Bridon, Fang Chen, and Yang Wang. 2015. Investigating user confidence for uncertainty presentation in predictive decision making. In Proceedings of the annual meeting of the Australian special interest group for computer human interaction. 352-360.
  3. Umang Bhatt, Valerie Chen, Katherine M Collins, Parameswaran Kamalaruban, Emma Kallina, Adrian Weller, and Ameet Talwalkar. 2023. Learning Personalized Decision Support Policies. arXiv preprint arXiv:2304.06701 (2023). · arXiv
  4. Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces (IUI '20). ACM, New York, NY, USA.
  5. Kaileigh A Byrne, Crina D Silasi-Mansat, and Darrell A Worthy. 2015. Who chokes under pressure? The Big Five personality traits and decision-making under pressure. Personality and individual differences 74 (2015), 22-28.
  6. John T. Cacioppo and Richard E. Petty. 1982. The need for cognition. Journal of Personality and Social Psychology 42, 1 (1982), 116-131. https://doi.org/10.1037/ 0022-3514.42.1.116 · DOI · Paper page
  7. J T Cacioppo, R E Petty, and C F Kao. 1984. The efficient assessment of need for cognition. Journal of personality assessment 48, 3 (1984), 306-307. https: //doi.org/10.1207/s15327752jpa4803_13 · DOI · Paper page
  8. Shiye Cao, Catalina Gomez, and Chien-Ming Huang. 2023. How Time Pressure in Different Phases of Decision-Making Influences Human-AI Collaboration. Proc. ACM Hum.-Comput. Interact. 7, CSCW2, Article 277 (oct 2023), 26 pages. https://doi.org/10.1145/3610068 · DOI · Paper page
  9. Gretchen B Chapman and Eric J Johnson. 1999. Anchoring, activation, and the construction of values. Organizational behavior and human decision processes 79, 2 (1999), 115-153.
  10. Mats Danielsson and Kjell Ohlsson. 1999. Decision making in emergency man- agement: A survey study. International Journal of Cognitive Ergonomics 3, 2 (1999), 91-99.
  11. Adele Diederich, Marc Wyszynski, and Stefan Traub. 2020. Need, frames, and time constraints in risky decision-making. Theory and Decision 89, 1 (2020), 1-37.
  12. Nicholas Epley, Boaz Keysar, Leaf Van Boven, and Thomas Gilovich. 2004. Per- spective taking as egocentric anchoring and adjustment. Journal of personality and social psychology 87, 3 (2004), 327.
  13. Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Paul Fisher, Diane Wilson, Michael Fitzke, Mark Parkinson, Eric Horvitz, Kori Inkpen, and Besmira Nushi. 2022. Who goes first? Influences of human-AI workflow on decision making in clinical imaging. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 1362-1374.
  14. Raymond Fok and Daniel S. Weld. 2023. In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making. arXiv preprint arXiv:2305.07722 (2023). · arXiv
  15. Amy Franklin, Ying Liu, Zhe Li, Vickie Nguyen, Todd R. Johnson, David Robinson, Nnaemeka Okafor, Brent King, Vimla L. Patel, and Jiajie Zhang. 2011. Oppor- tunistic decision making and complexity in emergency care. Journal of Biomed- ical Informatics 44, 3 (2011), 469-476. https://doi.org/10.1016/j.jbi.2011.04.001 Biomedical Complexity and Error. · DOI
  16. Krzysztof Z. Gajos and Krysta Chauncey. 2017. The Influence of Personality Traits and Cognitive Load on the Use of Adaptive User Interfaces. In Proceedings of the 22Nd International Conference on Intelligent User Interfaces (Limassol, Cyprus) (IUI '17). ACM, New York, NY, USA, 301-306. https://doi.org/10.1145/3025171.3025192 · DOI · Paper page
  17. Susanne Gaube, Harini Suresh, Martina Raue, Alexander Merritt, Seth J Berkowitz, Eva Lermer, Joseph F Coughlin, John V Guttag, Errol Colak, and Marzyeh Ghas- semi. 2021. Do as AI say: susceptibility in deployment of clinical decision-aids. NPJ digital medicine 4, 1 (2021), 1-8.
  18. Samuel D Gosling, Peter J Rentfrow, and William B Swann Jr. 2003. A very brief measure of the Big-Five personality domains. Journal of Research in personality 37, 6 (2003), 504-528.
  19. Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scan- dinavian Journal of Statistics 6, 2 (1979), 65-70.
  20. Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 467-474.
  21. Miloš Kankaraš. 2017. Personality matters: Relevance and assessment of personality characteristics. OECD Education Working Papers 157. OECD Publishing. https: //doi.org/10.1787/8a294376-en · DOI
  22. Gary Klein. 1997. The recognition-primed decision (RPD) model: Looking back, looking forward. Naturalistic decision making 285 (1997), 292.
  23. Martin G Kocher and Matthias Sutter. 2006. Time is money-Time pressure, incentives, and the quality of decision-making. Journal of Economic Behavior & Organization 61, 3 (2006), 375-392.
  24. Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006 (2019). · arXiv
  25. Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. 2016. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1675-1684.
  26. Shuai Ma, Ying Lei, Xinru Wang, Chengbo Zheng, Chuhan Shi, Ming Yin, and Xiaojuan Ma. 2023. Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 759, 19 pages. https://doi.org/10.1145/ 3544548.3581058 · DOI · Paper page
  27. Mark A. Musen, Blackford Middleton, and Robert Greenes. 2014. Clinical decision- support systems. Springer London, United Kingdom, 643-674. https://doi.org/10. 1007/978-1-4471-4474-8_22 · DOI
  28. Gali Noti and Yiling Chen. 2022. Learning When to Advise Human Decision Makers. In International Joint Conference on Artificial Intelligence. https://api. semanticscholar.org/CorpusID:252545048
  29. Mahsan Nourani, Chiradeep Roy, Jeremy E. Block, Donald R. Honeycutt, Tahrima Rahman, Eric D. Ragan, and Vibhav Gogate. 2021. Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems. 26th International Conference on Intelligent User Interfaces (2021).
  30. Vimla Patel, Jiajie Zhang, Nicole Yoskowitz, Robert Green, and Osman Sayan. 2008. Translational Cognition for Decision Support in Critical Care Environments: A Review. Journal of biomedical informatics 41 (07 2008), 413-31. https://doi.org/ 10.1016/j.jbi.2008.01.013 · DOI
  31. Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wort- man Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (2018). · arXiv
  32. Charvi Rastogi, Yunfeng Zhang, Dennis Wei, Kush R Varshney, Amit Dhurandhar, and Richard Tomsett. 2022. Deciding fast and slow: The role of cognitive biases in ai-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1-22.
  33. René Riedl. 2022. Is trust in artificial intelligence systems related to user person- ality? Review of empirical evidence and future research directions. Electronic Markets 32, 4 (2022), 2021-2051.
  34. Ian D Roberts, Yi Yang Teoh, and Cendri A Hutcherson. 2022. Time to pay attention? Information search explains amplified framing effects under time pressure. Psychological Science 33, 1 (2022), 90-104.
  35. Henner Gimpel Sarah Bayer and Moritz Markgraf. 2022. The role of domain expertise in trusting and following explainable AI decision support systems. Jour- nal of Decision Systems 32, 1 (2022), 110-138. https://doi.org/10.1080/12460125. 2021.1958505 · DOI · Paper page
  36. Nadine B. Sarter and Beth Schroeder. 2001. Supporting Decision Making and Action Selection under Time Pressure and Uncertainty: The Case of In-Flight Icing. Human Factors 43, 4 (2001), 573-583. https://doi.org/10.1518/001872001775870403 · DOI
  37. Falk Scholer, Diane Kelly, Wan-Ching Wu, Hanseul S. Lee, and William Webber. 2013. The effect of threshold priming and need for cognition on relevance calibration and assessment. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (Dublin, Ireland) (SIGIR '13). Association for Computing Machinery, New York, NY, USA, 623-632. https://doi.org/10.1145/2484028.2484090 · DOI
  38. Irena Stepanikova. 2012. Racial-ethnic biases, time pressure, and medical deci- sions. Journal of health and social behavior 53, 3 (2012), 329-343.
  39. Rachel F Sussman and Robert Sekuler. 2022. Feeling rushed? Perceived time pressure impacts executive function and stress. Acta psychologica 229 (2022), 103702.
  40. Michelle Yik, Kin Fai Ellick Wong, and Kevin J Zeng. 2019. Anchoring-and- adjustment during affect inferences. Frontiers in psychology 9 (2019), 2567.

Cited by

Sorted by citations

Authors
Swaroop, S., Buçinca, Z., Gajos, K. Z., & Doshi-Velez, F.
Affiliations
  • Harvard University (US)
  • SEAS, Harvard University, United States
ORCID
Venue
Proceedings of the 29th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2024-03-18
Pages
138-154
DOI
10.1145/3640543.3645206
OpenAlex
W4393970536
Cited by
49
References
15
Citations by year
2026: 24 · 2025: 21 · 2024: 4
FWCI
28.92
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
NSF (National Science Foundation) (IIS-2107391)
Topics
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
AI-assisted decision-making, time pressure, overreliance on AI, trust in AI recommendations, logic puzzles, Big Five personality traits, human-AI interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review