The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
Open accessAbstract
The increased use of algorithmic predictions in sensitive domains has been accompanied by both enthusiasm and concern. To understand the opportunities and risks of these technologies, it is key to study how experts alter their decisions when using such tools. In this paper, we study the adoption of an algorithmic tool used to assist child maltreatment hotline screening decisions. We focus on the question: Are humans capable of identifying cases in which the machine is wrong, and of overriding those recommendations? We first show that humans do alter their behavior when the tool is deployed. Then, we show that humans are less likely to adhere to the machine's recommendation when the score displayed is an incorrect estimate of risk, even when overriding the recommendation requires supervisory approval. These results highlight the risks of full automation and the importance of designing decision pipelines that provide humans with autonomy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“In some real-world applications of HAIC1,21, an ML model is trained to perform the same classification task as the human expert.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior research has also established that an irreducible degree of uncertainty exists with respect to the outcomes in the public sector [21,34,57] with both humans and algorithms likely to make mistakes.”
From A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare (2021) · cites this paper as [34]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Diagnostic tasks entail circumstances where participants are given access to well-defned and comprehensive information about the current task, allowing them to make precise judgments [29].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prognostic tasks, on the other hand, involve scenarios in which participants are presented with restricted or unclear data and need to generate predictions regarding future outcomes [29].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With the presence of uncertainty in a task, individuals may lack sufcient capability to verify the correctness of AI advice and therefore rely more heavily on the AI systems [29], leading to reduced appropriate reliance on AI systems.”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [29]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The AFST is a risk assessment that predicts the likelihood that children will be removed from their home due to child neglect and abuse in the next two years.”
From The flaws of policies requiring human oversight of government algorithms (2022) · cites this paper as De-Arteaga et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By calling teachers' attention to discrepancies between their own judgments and those of an AIEd system, the system may promote productive teacher reflection and learning”
From Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education? (2021) · cites this paper as De-Arteaga et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has been shown that the expert’s performance is significantly impacted by seeing the ML model’s score along with the rest of the information pertaining to each instance1,21,22.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This creates a realistic dataset of expert predictions resulting from collecting a single prediction per instance, as is often the case in real-world systems1,19.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finally, we observe that current L2D research overlooks the limitations of human work capacity, despite these being a challenge in real-world systems1,19, where typically only one expert makes a decision per instance.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finally, we observe that current L2D research overlooks the limitations of human work capacity, despite these being a challenge in real-world systems1,19, where typically only one expert makes a decision per instance.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As such, algorithmic decision-making tools are now being used in high-stakes CWS situations, including making risk assessments of child abuse [34] and determining placement stability [86].”
From A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare (2021) · cites this paper as [34]
The cited source says what the claim attributes to it.
In the citing paper
“These predictions are presented to child welfare workers to inform their decisions about which cases to investigate( De-Arteaga et al., 2020 ;Eubanks, 2018 ).”
From The flaws of policies requiring human oversight of government algorithms (2022) · cites this paper as De-Arteaga et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The negative impact of hallucinations on data quality is well-documented [33] —incorrect AI recommendations have been shown to harm users in decision-making contexts [13].”
From Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations (2025) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“There are relevant examples where human operators successfully identified errors made by automated systems [14, 54].”
From On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2024) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, a growing body of research has focused on human-AI collaboration (HAIC), with several authors arguing that humans and AI have complementary strengths and weaknesses1,2.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Collaborative systems have shown that humans can correct model predictions in specific instances1, and that humans, in collaboration with a Machine Learning (ML) model, can achieve higher performance than either the expert or the model alone3.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Prior work has argued that algorithms in the public sector is a domain in its own right [34,62,107,109,121] and must be characterized differently as compared to algorithms in the private sector where the decision outcomes are well-defined.”
From A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare (2021) · cites this paper as [34]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Clinical versus statistical prediction: A theoretical analysis and a review of the evidence.
Meehl, P. E. (1954) · University of Minnesota Press eBooks · doi:10.1037/11281-000 · Cited by 2,227
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · cited 2×
“Meanwhile, studies focused on the causes of algorithm aversion have found that repeatedly seeing the algorithm make the same mistake leads to decreased reliance of the agent on the system [13], while giving some control over the algorithm can counter this phenomenon [14].”
BackgroundAutomation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“Studies analyzing factors contributing to automation bias have found that complex tasks and time pressure may increase over-reliance on decision support [41, 17].”
BackgroundÆgisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., et al. (2006) · The Counseling Psychologist · doi:10.1177/0011000005285875 · Cited by 737
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 2×
“Users’ reliance on the system is known to vary with the observed [51, 52] and stated accuracy [50] of the system.”
Background
Flagged first, then by citations
Research works printed but not linked: 13
- Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. 134--148.
- Robyn M Dawes, David Faust, and Paul E Meehl. 1989. Clinical versus actuarial judgment. Science 243, 4899 (1989), 1668--1674.
- Matthew DeMichele, Peter Baumgartner, Kelle Barrick, Megan Comfort, Samuel Scaggs, and Shilpi Misra. 2018. What do criminal justice professionals think about risk assessment at pretrial? Available at SSRN 3168490 (2018).
- Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press.
- William M Grove, David H Zald, Boyd S Lebow, Beth E Snitz, and Chad Nelson. 2000. Clinical versus mechanical prediction: a meta-analysis. Psychological assessment 12, 1 (2000), 19.
- Sophie Hilgard, Nir Rosenfeld, Mahzarin R Banaji, Jack Cao, and David C Parkes. 2019. Learning Representations by Humans, for Humans. arXiv preprint arXiv:1905.12686 (2019). · arXiv
- John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50--80.
- David Madras Toni Pitassi and Richard Zemel. 2018. Predict responsibly: improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems. 6147--6157.
- Katharina Marten, Tobias Seyfarth, Florian Auer, Edzard Wiener, Andreas Grillhösl, Silvia Obenauer, Ernst J Rummeny, and Christoph Engelke. 2004. Computer-assisted detection of pulmonary nodules: performance evaluation of an expert knowledge-based detection system in consensus reading with experienced and inexperienced chest radiologists. European radiology 14, 10 (2004), 1930--1938.
- Nadine B. Sarter and Beth Schroeder. 2001. Supporting decision making and action selection under time pressure and uncertainty: The case of in-flight icing. Human factors 43, 4 (2001), 573--583.
- Jennifer L. Skeem, Nicholas Scurich, and John Monahan. 2019. Impact of Risk Assessment on Judges. Fairness in Sentencing Relatively Poor Defendants. Virginia Public Law and Legal Theory Research Paper 2019-02 (2019).
- Linda J. Skitka, Kathleen L. Mosier, Mark Burdick, and Bonnie Rosenblatt. 2000. Automation bias and errors: are crews better than individuals? The International journal of aviation psychology 10, 1 (2000), 85--97.
- Michael Yeomans, Anuj Shah, Sendhil Mullainathan, and Jon Kleinberg. 2017. Making sense of recommendations. Journal of Behavioral Decision Making (2017).
Cited by
Sorted by citations
Items 101–200 of 221
Learning When to Defer to Humans for Short Answer Grading
Li, Z., Zhang, C., Jin, Y., Cang, X., Puntambekar, S., & Passonneau, R. J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_34 · Cited by 10
Counterfactual Predictions under Runtime ConfoundingPreprint
Coston, A., Kennedy, E. H., & Chouldechova, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16916 · Cited by 12
Self-citationAshktorab, Z., Desmond, M., Pan, Q., Johnson, J., Brachman, M., Dugan, C., Danilevsky, M., & Geyer, W. (2025) · Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work · doi:10.1145/3729176.3729179 · Cited by 10
A Human-Centric Perspective on Fairness and Transparency in Algorithmic Decision-Making
Schoeffer, J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503811 · Cited by 9
Wang, X., Tian, Y., Huang, K., & Liang, B. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104113 · Cited by 12
Nurturing Capabilities: Unpacking the Gap in Human-Centered Evaluations of AI-Based Systems
Khullar, A., Nalin, N., Prasad, A., Mampilli, A. J., & Kumar, N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713278 · Cited by 14
Leading AI Adoption in Organizations: Introducing a Behavioral Human-Centered Approach
Schweitzer, S., Narayanan, D., McGuire, J., & Cremer, D. D. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2531287 · Cited by 11
Koenecke, A., Giannella, E., Willer, R., & Goel, S. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22163 · Cited by 11
Empowering Social Service with AI: Insights from a Participatory Design Study with Practitioners
Tan, Y., Soh, K. X., Zhang, R., Lee, J., Meng, H., Sen, B., & Lee, Y. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719736 · Cited by 19
Enhancing AI-Assisted Purchase Decisions: The Role of the Sense of Autonomy
Hou, J., Yang, S., Xiong, G., & Pavlou, P. A. (2025) · MIS Quarterly · doi:10.25300/misq/2025/17607 · Cited by 8
Workshop on Trust and Reliance in AI-Human Teams (TRAIT)
Bansal, G., Buçinca, Z., Holstein, K., Hullman, J., Smith-Renner, A. M., Stumpf, S., & Wu, S. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3573831 · Cited by 8
Recommender Systems: Legal and Ethical Issues
Genovesi, S., Kaesling, K., & Robbins, S. (2023) · The International library of ethics, law and technology · doi:10.1007/978-3-031-34804-4 · Cited by 10
Die Tragik der Algorithm Aversion
Filiz, I., Judek, J. R., Lorenz, M., & Spiwoks, M. (2021) · doi:10.46850/sofia.9783941627888 · Cited by 8
Digital administrative burdens and clawback logics: the Australian Robodebt scheme
Moynihan, D. P., Hybschmann, M., Gimborys, K., Loudin, S., & McClellan, W. (2025) · Perspectives on Public Management and Governance · doi:10.1093/ppmgov/gvaf011 · Cited by 7
Sun, C., Major, L., Daltry, R., Moustafa, N., & Friedberg, A. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3664662 · Cited by 10
Valdivia, A., Hyde-Vaamonde, C., & Marcos, J. G. (2024) · AI & Society · doi:10.1007/s00146-024-02016-9 · Cited by 22
Give and Take: Perceptions of a Conversational Coach Agent in Fitness Trackers MHCI014
Loerakker, M. B., Stefanidi, E., Niess, J., Mildner, T., & Woźniak, P. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3743718 · Cited by 6
Passi, S., Dhanorkar, S., & Vorvoreanu, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_98 · Cited by 9
Planning to Fairly Allocate: Probabilistic Fairness in the Restless Bandit Setting
Herlihy, C. P., Prins, A., Srinivasan, A., & Dickerson, J. P. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599467 · Cited by 6
You Are What You Tweet: Profiling Users by Past Tweets to Improve Hate Speech Detection
Chaudhry, P., & Lease, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-96960-8_13 · Cited by 6
Arnanz, A. S. (2023) · Documentación Administrativa · doi:10.24965/da.11148 · Cited by 6
Doing Responsibilities with Automated Grading Systems: An Empirical Multi-Stakeholder Exploration
Figueras, C., Rossitto, C., & Pargman, T. C. (2024) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3679318.3685334 · Cited by 5
The ethics of algorithms from the perspective of the cultural history of consciousness: first look
Martínez, C. A. S., & Montoya, O. (2022) · Ai & Society · doi:10.1007/s00146-022-01475-2 · Cited by 7
Overriding (in)justice: pretrial risk assessment administration on the frontlines
Riley, S. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658920 · Cited by 5
A Survey on Human Factors in Cyberspace: A New Dimension of Privacy Threats
Ishaq, M., Kifayat, K., & Zafar, M. (2023) · International Conference on Communication, Computing and Digital Systems (C-CODE) · doi:10.1109/c-code58145.2023.10139904 · Cited by 5
Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making
Lee, M. H., & Tok, M. Z. Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732155 · Cited by 8
Social eXplainable AI (Social XAI): Towards Expanding the Social Benefits of XAI
Naiseh, M. (2024) · Studies in neuroscience, psychology and behavioral economics · doi:10.1007/978-3-031-70355-3_13 · Cited by 4
A Data-Driven Simulation of the New York State Foster Care System
Du, Y., Ionescu, S., Sage, M., & Joseph, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533165 · Cited by 4
Stroke Home Rehabilitation Approach Using Mobile Application Based on PostNet Machine Learning Model
Das, U. C., Le, N. T., Benjapolakul, W., Vitoonpong, T., & Pluempitiwiriyawej, C. (2023) · International Conference on Medical and Health Informatics (ICMHI) · doi:10.1145/3608298.3608339 · Cited by 5
When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-Making
Donahue, K., Gollapudi, S., & Kollias, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28866 · Cited by 9
The Societal Impacts of Algorithmic Decision-Making
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195 · Cited by 8
Learning to predict with supporting evidence
Raghu, A., Guttag, J. V., Young, K., Pomerantsev, E. V., Dalca, A. V., & Stultz, C. M. (2021) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3450439.3451869 · Cited by 9
Correia, A., Schneider, D., Jameel, S., Paredes, H., & Fonseca, B. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-71187-0_117 · Cited by 4
Kucher, K., Sultanum, N., Daza, A., Simaki, V., Skeppstedt, M., Plank, B., Fekete, J., & Mahyar, N. (2022) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv57783.2022.00008 · Cited by 5
Learning Representations by Humans, for HumansPreprint
Hilgard, S., Rosenfeld, N., Banaji, M. R., Cao, J., & Parkes, D. C. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4nvts · Cited by 6
Proceedings of The ACM Collective Intelligence Conference
(2023) · doi:10.1145/3582269 · Cited by 4
Artificial Intelligence and Empathy in Social Work Practice
Arzroomchilar, E. (2025) · Journal of Social Work · doi:10.1177/14680173251390786 · Cited by 4
AI aversion? Effects of author disclosure on young people’s perceptions of mental health advice
Brandtzæg, P. B., Skjuve, M. B., & Følstad, A. (2026) · Cyberpsychology Journal of Psychosocial Research on Cyberspace · doi:10.5817/cp2026-2-1 · Cited by 4
Lost in the Loop: Who Is the ‘Human’ of the Human in the Loop?
Goldenfein, J. (2026) · Cambridge University Press eBooks · doi:10.1017/9781009425308.006 · Cited by 4
Human-AI Teaming: Following the IMOI Framework
Kleanthous, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60611-3_27 · Cited by 4
Antar, A. D., Molaei, S., Chen, Y.-Y., Lee, M. L., & Banović, N. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676323 · Cited by 9
Exploring the Effects of User Trust in Generative AI on Decision-Making in Semi-structured Problem
Kamada, K., Takahashi, H., Wang, T.-Y., & Yuizono, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93965-5_17 · Cited by 3
Khullar, A., Panjal, P., Pandey, R., Burnwal, A., Raj, P., Jha, A. A., Hitesh, P., Reddy, R. J., Himanshu, H., & Seth, A. (2021) · India HCI (IndiaHCI) · doi:10.1145/3506469.3506473 · Cited by 3
Review on the Application Areas of Decision-Making Algorithms in Smart Homes
Jin, L., & Boden, A. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230076 · Cited by 4
Algorithmic Decision Making in Education: Challenges and Opportunities
Levantis, N., & Sgora, A. (2024) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578645 · Cited by 6
Zhang, A., Liao, M., Kravchenko, E., Taylor, M., Haddad, A. J., Bhat, C. R., Watkins, S. C., & Lee, M. K. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757477 · Cited by 4
Co-Designing Islamic AI Ethics: Insights from the UK Muslim Community
Campbell-Esen, R., Mahoney, J., & Talhouk, R. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790413 · Cited by 6
Personnel selection system based on the selection algorithm
Quintanilla-Medina, M. D. C. V., & Ygnacio, M. A. C. (2024) · DYNA · doi:10.15446/dyna.v91n231.110412 · Cited by 2
Brown Hands Aren’t Terrorists: Challenges in Image Classification of Violent Extremist Content
Hall, M., & Haas, C. S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77820-0_15 · Cited by 2
Trucks Don’t Mean Trump: Diagnosing Human Error in Image Analysis
Zamfirescu-Pereira, J. D., Chen, J., Wen, E., Koenecke, A., Garg, N., & Pierson, E. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533145 · Cited by 3
Investigating the Relative Strengths of Humans and Machine Learning in Decision-Making
Rastogi, C. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604738 · Cited by 2
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603198 · Cited by 2
Algorithmic Monoculture and Social Welfare
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603211 · Cited by 2
Saving Lives With Algorithm-Enabled Process Innovation for Sepsis CarePreprint
Adjerid, I., Ayvaci, M., & Özer, Ö. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3456870 · Cited by 2
Learning to Predict with Supporting Evidence: Applications to Clinical Risk PredictionPreprint
Raghu, A., Guttag, J. V., Young, K., Pomerantsev, E. V., Dalca, A. V., & Stultz, C. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02768 · Cited by 2
A Bandit Model for Human-Machine Decision Making with Private Information and OpacityPreprint
Bordt, S., & Ulrike Von Luxburg, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04800 · Cited by 10
Saxena, D., & Guha, S. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4077245 · Cited by 2
Kellogg, K. C., Lifshitz‐Assaf, H., Randazzo, S., Mollick, E., Dell’Acqua, F., McFowland, E., Candelon, F., & Lakhani, K. R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4857373 · Cited by 2
AI Reliance and Decision Quality: Fundamentals, Interdependence, and the Effects of InterventionsPreprint
Schoeffer, J., Jakubik, J., Vössing, M., Kühl, N., & Satzger, G. (2024) · doi:10.31219/osf.io/cekm9 · Cited by 2
Law and Infrastructure: Reliability, Automation Transition, and Irregularities of “U‐Space”
Nawaz, S. A. (2025) · Regulation & Governance · doi:10.1111/rego.70046 · Cited by 2
Exploration and Optimization of Generative Variability in Future Work: A Mixed-Initiative Analysis
Muller, M. J., He, J., & Weisz, J. D. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729195 · Cited by 2
Application of the data quality framework to administrative data on child maltreatment
Thompson, Y. T., Li, Y., Kang, Z., Miller, M., Wurgler, R., & Silovsky, J. F. (2025) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2025.107646 · Cited by 3
Eliciting Multimodal Approaches for Machine Learning–Assisted Photobook Creation
Bittner, S.-J., Barz, M., & Sonntag, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05002-1_31 · Cited by 1
Interactive Social Agent Behaviors of Politeness and Compromise in Collaborative Decision-making
Jeong, J., & Hoffman, G. (2025) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3717511.3747073 · Cited by 1
A Systematic Review on Human Roles, Solutions, and Methodological Approaches to Address Bias in AI
Hashky, A., & Ragan, E. D. (2026) · ACM Computing Surveys · doi:10.1145/3793667 · Cited by 3
Thirukkumaran, M., Foo, E., Chang, E. C., Koh, B. Y., Cho, K., & Chong, Y. K. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-28274-3_14 · Cited by 1
Zhang, Y., Hiekata, K., & Nakashima, T. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2685232 · Cited by 1
Human-in-the-Loop Based Online Handwriting Mathematical Expressions Recognition
Kang, W., Huang, J., Tian, F., Fan, X., Liu, J., & Dai, G. (2021) · Journal of Computer-Aided Design & Computer Graphics · doi:10.3724/sp.j.1089.2021.18796 · Cited by 1
Ammeling, J., Manger, C., Kwaka, E., Krügel, S., Uhl, M., Kießig, A., Fritz, A., Ganz, J., Riener, A., Bertram, C., et al. (2023) · Mensch und Computer (MuC) · doi:10.1145/3603555.3608561 · Cited by 1
Mitigating Bias in Algorithmic Decision-making: Evaluating Claims and Practices
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603212 · Cited by 1
Harnessing Real-Time Data for Intelligent Decision-Making in Cyber-Physical Systems
Premavathi, T., Jain, R., Thorat, V. V., Parmar, K. J., Palaniappan, D., & Thorat, C. (2024) · Advances in computer and electrical engineering book series · doi:10.4018/979-8-3693-5728-6.ch010 · Cited by 1
Preposition Salad: Placing Humans & AI in/on/over/along/under ‘the-loop’
Singh, A. P., & Szajnfarber, Z. (2025) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v5i1.35571 · Cited by 1
López, J. M. C., Escobar, B. R. P., Paredes, O. M. A., Coaquira, E. H. B., & Carita-Choquecahua, A. (2026) · Athenea · doi:10.47460/athenea.v7i23.128 · Cited by 1
The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603204 · Cited by 1
A Labeling Task Design for Supporting Recent Algorithmic Needs
You, J., Park, D., Song, J.-Y., & Suh, B. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020415 · Cited by 2
On the Fairness of Machine-Assisted Human DecisionsPreprint
Gillis, T. B., McLaughlin, B., & Spiess, J. (2021) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2110.15310 · Cited by 14
Supporting Information Integration in Human-AI Augmentation via Reflection on UnobservablesPreprint
De‐Arteaga, M., & Holstein, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4884604 · Cited by 1
Self-citationAlgorithmic monoculture and social welfare
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603202
"A cold, technical decision-maker": Can AI provide explainability, negotiability, and humanity?Preprint
Woodruff, A., Anderson, Y. A., Armstrong, K. J., Gkiza, M., Jennings, J., Moessner, C., Viégas, F. B., WATTENBERG, M. P., Webb, A. L., Wrede, F., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00874 · Cited by 10
Robinson, K. P., Robert, L., & Eglash, R. (2021) · IEEE RO-MAN · doi:10.1109/ro-man50785.2021.9515372
Cognitive Pairwise Comparisons for Human Centric Design in Artificial Intelligence and Data SciencePreprint
Yuen, K. K. F. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4048692
Algorithmic Decision-Making, Economic Behavior and Predictability in Financial Markets
Judek, J. R. (2023) · eDiss (Georg-August-Universität Göttingen) · doi:10.53846/goediss-9868
Levina, O. A., & Mattern, S. (2023) · The International library of ethics, law and technology · doi:10.1007/978-3-031-34804-4_9
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603213
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603205
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603196
How Do Classifiers Induce Agents to Behave Strategically?
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603201
How Do Classifiers Induce Agents to Behave Strategically?
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603210
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603214
Ai Liability in Europe: The Problem of the Human in the LoopPreprint
Arcila, B. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4647568
Explanations for Robust DecisionsPreprint
Yang, K. H., Yoder, N., & Zentefis, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4723587
Recognizing Selected Features in the News Text
Kaczmarek, A., Tobolewski, F., Ganzha, M., Paprzycki, M., & Wróblewska, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-86193-2_17
Gabriels, A. E., Kuiper, C. H. Z., & Harder, A. T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5247190
DiSorbo, M. D., & Ferreira, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5253399
Racial Implications of Police-Algorithm Interactions: Evidence from Rearrest PredictionsPreprint
Lee, Y. S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5255282
Naming And Blaming Automated Decision-Making Systems: Analysis Of Contesting Algorithmic Injustice
Gules-Guctas, E. (2023) · Scholars Archive - University at Albany (University at Albany, State University of New York) · doi:10.54014/ybg8-z8sd
Improving AI-Assisted Decision-Making Through Calibrated CoarseningPreprint
Hoong, R., & Dreyfuss, B. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5286198
Error in the Loop: How Human Mistakes Can Improve Algorithmic Learning
Copus, R. W., Spackman, C., & Laqueur, H. S. (2025) · Journal of law & empirical analysis. · doi:10.1177/2755323x251357643
Understanding Human Detection of Algorithmic Bias Through Drift-Diffusion ModelingPreprint
Overbye-Thompson, H. E., Hamilton, K. A., & Fisher, J. C. (2025) · Research Square · doi:10.21203/rs.3.rs-7359286/v1
Aladwan, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3743733
Metadata
- Authors
- De-Arteaga, M., Fogliato, R., & Chouldechova, A.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-12
- OpenAlex
- W3007910004
- Cited by
- 221
- References
- 33
- Citations by year
- 2026: 24 · 2025: 35 · 2024: 34 · 2023: 59 · 2022: 40 · 2021: 24 · 2020: 3
- FWCI
- 32.68
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Hillman Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- human-in-the-loop, algorithmic decision support, child maltreatment screening, child protection services, human-machine decision-making, risk prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
