The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
Open accessAbstract
The increased use of algorithmic predictions in sensitive domains has been accompanied by both enthusiasm and concern. To understand the opportunities and risks of these technologies, it is key to study how experts alter their decisions when using such tools. In this paper, we study the adoption of an algorithmic tool used to assist child maltreatment hotline screening decisions. We focus on the question: Are humans capable of identifying cases in which the machine is wrong, and of overriding those recommendations? We first show that humans do alter their behavior when the tool is deployed. Then, we show that humans are less likely to adhere to the machine's recommendation when the score displayed is an incorrect estimate of risk, even when overriding the recommendation requires supervisory approval. These results highlight the risks of full automation and the importance of designing decision pipelines that provide humans with autonomy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“In some real-world applications of HAIC1,21, an ML model is trained to perform the same classification task as the human expert.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior research has also established that an irreducible degree of uncertainty exists with respect to the outcomes in the public sector [21,34,57] with both humans and algorithms likely to make mistakes.”
From A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare (2021) · cites this paper as [34]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Diagnostic tasks entail circumstances where participants are given access to well-defned and comprehensive information about the current task, allowing them to make precise judgments [29].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prognostic tasks, on the other hand, involve scenarios in which participants are presented with restricted or unclear data and need to generate predictions regarding future outcomes [29].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With the presence of uncertainty in a task, individuals may lack sufcient capability to verify the correctness of AI advice and therefore rely more heavily on the AI systems [29], leading to reduced appropriate reliance on AI systems.”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [29]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The AFST is a risk assessment that predicts the likelihood that children will be removed from their home due to child neglect and abuse in the next two years.”
From The flaws of policies requiring human oversight of government algorithms (2022) · cites this paper as De-Arteaga et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By calling teachers' attention to discrepancies between their own judgments and those of an AIEd system, the system may promote productive teacher reflection and learning”
From Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education? (2021) · cites this paper as De-Arteaga et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has been shown that the expert’s performance is significantly impacted by seeing the ML model’s score along with the rest of the information pertaining to each instance1,21,22.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This creates a realistic dataset of expert predictions resulting from collecting a single prediction per instance, as is often the case in real-world systems1,19.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finally, we observe that current L2D research overlooks the limitations of human work capacity, despite these being a challenge in real-world systems1,19, where typically only one expert makes a decision per instance.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Finally, we observe that current L2D research overlooks the limitations of human work capacity, despite these being a challenge in real-world systems1,19, where typically only one expert makes a decision per instance.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As such, algorithmic decision-making tools are now being used in high-stakes CWS situations, including making risk assessments of child abuse [34] and determining placement stability [86].”
From A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare (2021) · cites this paper as [34]
The cited source says what the claim attributes to it.
In the citing paper
“These predictions are presented to child welfare workers to inform their decisions about which cases to investigate( De-Arteaga et al., 2020 ;Eubanks, 2018 ).”
From The flaws of policies requiring human oversight of government algorithms (2022) · cites this paper as De-Arteaga et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The negative impact of hallucinations on data quality is well-documented [33] —incorrect AI recommendations have been shown to harm users in decision-making contexts [13].”
From Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations (2025) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“There are relevant examples where human operators successfully identified errors made by automated systems [14, 54].”
From On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives (2024) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Recently, a growing body of research has focused on human-AI collaboration (HAIC), with several authors arguing that humans and AI have complementary strengths and weaknesses1,2.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“Collaborative systems have shown that humans can correct model predictions in specific instances1, and that humans, in collaboration with a Machine Learning (ML) model, can achieve higher performance than either the expert or the model alone3.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [1]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Prior work has argued that algorithms in the public sector is a domain in its own right [34,62,107,109,121] and must be characterized differently as compared to algorithms in the private sector where the decision outcomes are well-defined.”
From A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare (2021) · cites this paper as [34]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Clinical versus statistical prediction: A theoretical analysis and a review of the evidence.
Meehl, P. E. (1954) · University of Minnesota Press eBooks · doi:10.1037/11281-000 · Cited by 2,227
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018) · Management Science · doi:10.1287/mnsc.2016.2643 · Cited by 1,230 · cited 2×
“Meanwhile, studies focused on the causes of algorithm aversion have found that repeatedly seeing the algorithm make the same mistake leads to decreased reliance of the agent on the system [13], while giving some control over the algorithm can counter this phenomenon [14].”
BackgroundAutomation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“Studies analyzing factors contributing to automation bias have found that complex tasks and time pressure may increase over-reliance on decision support [41, 17].”
BackgroundÆgisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., et al. (2006) · The Counseling Psychologist · doi:10.1177/0011000005285875 · Cited by 737
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 2×
“Users’ reliance on the system is known to vary with the observed [51, 52] and stated accuracy [50] of the system.”
Background
Research works printed but not linked: 13
- Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. 134--148.
- Robyn M Dawes, David Faust, and Paul E Meehl. 1989. Clinical versus actuarial judgment. Science 243, 4899 (1989), 1668--1674.
- Matthew DeMichele, Peter Baumgartner, Kelle Barrick, Megan Comfort, Samuel Scaggs, and Shilpi Misra. 2018. What do criminal justice professionals think about risk assessment at pretrial? Available at SSRN 3168490 (2018).
- Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press.
- William M Grove, David H Zald, Boyd S Lebow, Beth E Snitz, and Chad Nelson. 2000. Clinical versus mechanical prediction: a meta-analysis. Psychological assessment 12, 1 (2000), 19.
- Sophie Hilgard, Nir Rosenfeld, Mahzarin R Banaji, Jack Cao, and David C Parkes. 2019. Learning Representations by Humans, for Humans. arXiv preprint arXiv:1905.12686 (2019). · arXiv
- John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50--80.
- David Madras Toni Pitassi and Richard Zemel. 2018. Predict responsibly: improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems. 6147--6157.
- Katharina Marten, Tobias Seyfarth, Florian Auer, Edzard Wiener, Andreas Grillhösl, Silvia Obenauer, Ernst J Rummeny, and Christoph Engelke. 2004. Computer-assisted detection of pulmonary nodules: performance evaluation of an expert knowledge-based detection system in consensus reading with experienced and inexperienced chest radiologists. European radiology 14, 10 (2004), 1930--1938.
- Nadine B. Sarter and Beth Schroeder. 2001. Supporting decision making and action selection under time pressure and uncertainty: The case of in-flight icing. Human factors 43, 4 (2001), 573--583.
- Jennifer L. Skeem, Nicholas Scurich, and John Monahan. 2019. Impact of Risk Assessment on Judges. Fairness in Sentencing Relatively Poor Defendants. Virginia Public Law and Legal Theory Research Paper 2019-02 (2019).
- Linda J. Skitka, Kathleen L. Mosier, Mark Burdick, and Bonnie Rosenblatt. 2000. Automation bias and errors: are crews better than individuals? The International journal of aviation psychology 10, 1 (2000), 85--97.
- Michael Yeomans, Anuj Shah, Sendhil Mullainathan, and Jon Kleinberg. 2017. Making sense of recommendations. Journal of Behavioral Decision Making (2017).
Cited by
Sorted by citations
Inherent Tradeoffs in the Fair Determination of Risk Scores
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603197 · Cited by 350
Algorithmic bias: Senses, sources, solutions
Fazelpour, S., & Danks, D. J. (2021) · Philosophy Compass · doi:10.1111/phc3.12760 · Cited by 338
The flaws of policies requiring human oversight of government algorithms
Green, B. (2022) · Computer Law & Security Review · doi:10.1016/j.clsr.2022.105681 · Cited by 263
How transparency modulates trust in artificial intelligence
Zerilli, J., Bhatt, U., & Weller, A. (2022) · Patterns · doi:10.1016/j.patter.2022.100455 · Cited by 257
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
Metadata
- Authors
- De-Arteaga, M., Fogliato, R., & Chouldechova, A.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-12
- OpenAlex
- W3007910004
- Cited by
- 221
- References
- 33
- Citations by year
- 2026: 24 · 2025: 35 · 2024: 34 · 2023: 59 · 2022: 40 · 2021: 24 · 2020: 3
- FWCI
- 32.68
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Hillman Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- human-in-the-loop, algorithmic decision support, child maltreatment screening, child protection services, human-machine decision-making, risk prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
