Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
AI-based decision support tools (ADS) are increasingly used to augment human decision-making in high-stakes, social contexts. As public sector agencies begin to adopt ADS, it is critical that we understand workers’ experiences with these systems in practice. In this paper, we present findings from a series of interviews and contextual inquiries at a child welfare agency, to understand how they currently make AI-assisted child maltreatment screening decisions. Overall, we observe how workers’ reliance upon the ADS is guided by (1) their knowledge of rich, contextual information beyond what the AI model captures, (2) their beliefs about the ADS’s capabilities and limitations relative to their own, (3) organizational pressures and incentives around the use of the ADS, and (4) awareness of misalignments between algorithmic predictions and their own decision-making objectives. Drawing upon these findings, we discuss design implications towards supporting more effective human-AI decision-making.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“researchers have explored the feasibility of human-AI collaborative decision-making, in which an AI model provides humans new data-driven insights on a task for achieving complementary performance, outperforming neither of the AI or the human alone [5, 30, 42, 43].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [30]
In the citing paper
“Concerns about provider liability have been a long-recognized challenge in recommendation-driven AI assistance.”
From To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical Events (2025) · cites this paper as [37]
In the citing paper
“If there is any institutional friction to disagreeing with AI recommendations, such as increased chance of liability for a medical error resulting from overriding AI recommendation, the presence of the algorithmic recommendation can be seen as taking away some authority from the human decision-maker without assuming any of the responsibility for the decision [30, 37, 72].”
From To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical Events (2025) · cites this paper as [37]
In the citing paper
“Similar to findings from prior work, our participants raised concerns about the effects of AI-enabled tools on their authority and liability [30, 37, 72].”
From To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical Events (2025) · cites this paper as [37]
Citations
Cites
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2019) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,279 · cited 2×
“, reflecting biases in the administrative data on which they are trained and operate upon) [13, 17] or introducing absurd errors into decision processes, which humans would not otherwise make [1, 15, 28].”
BackgroundResultBuilding Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206 · cited 1×
“Following a shared open coding session to calibrate coding granularity, each participant’s datawere qualitatively coded by two to three researchers [42].”
MethodBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 8×
“Whereas prior work has often studied human reliance on ADS tools as a function of an individual’s trust [5, 9, 38, 50], this conceptualization overlooks the potential impacts that organizational pressures and incentive structures may have on workers’ reliance, independent of their trust in the technology itself.”
BackgroundResultInfluentialDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754 · cited 8×
“Whereas prior work has often studied human reliance on ADS tools as a function of an individual’s trust [5, 9, 38, 50], this conceptualization overlooks the potential impacts that organizational pressures and incentive structures may have on workers’ reliance, independent of their trust in the technology itself.”
BackgroundResultInfluentialManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 3×
“Prior work has often understood human reliance on algorithmic decision supports in terms of patterns of “over-reliance,” “underreliance,” or “appropriate reliance”, that are shaped according to how well-calibrated an individual’s trust is in a given algorithmic system [5, 9, 38, 50].”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 1×
“, where it is likely to err and what factors it is not able to take into account) [4, 10, 43].”
BackgroundThe What-If Tool: Interactive Probing of Machine Learning Models
Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Viegas, F., & Wilson, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934619 · Cited by 505 · cited 1×
“[65]) in order to learn how particular factors impact the AFST score in specific cases (C1, C6, C8, S1).”
BackgroundVeale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 1×
“At the same time, AI-based judgments are themselves likely to be imperfect and biased, even if in different ways than humans [15, 32, 39, 64].”
BackgroundCamillo, C. A. (2018) · Global Encyclopedia of Public Administration, Public Policy, and Governance · doi:10.1007/978-3-319-20928-9_654 · Cited by 474
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 2×
“In some studies, human and machine intelligence combined has been shown to outperform either humans or AI alone [5, 24, 48], while in others human–AI partnerships have failed to improve or have even harmed decision quality, for various reasons [19, 50].”
BackgroundCo-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity
Holstein, K., McLaren, B. M., & Aleven, V. (2019) · Journal of Learning Analytics · doi:10.18608/jla.2019.62.3 · Cited by 390
Self-citationTeam learning: building shared mental models
Van den Bossche, P., Gijselaers, W., Segers, M., Woltjer, G., & Kirschner, P. (2011) · Instructional Science · doi:10.1007/s11251-010-9128-3 · Cited by 379 · cited 1×
“, the internal representations or systems of beliefs that they build up through experience, which they use to understand, explain, and predict algorithmic behavior, and then act accordingly [4, 34, 35, 61]—and we discuss the impacts that these beliefs have on their decision-making and communicative practices around th…”
BackgroundWhen the implication is not to design (technology)
Baumer, E. P. S., & Silberman, M. S. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979275 · Cited by 350 · cited 1×
“We note that the following design implications are relevant only when there is good reason to expect benefits of having an ADS in the first place, which outweigh potential harms [6, 18, 49].”
BackgroundHuman–machine partnership with artificial intelligence for chest radiograph diagnosis
Patel, B. N., Rosenberg, L., Willcox, G., Baltaxe, D., Lyons, M., Irvin, J., Rajpurkar, P., Amrhein, T., Gupta, R., Halabi, S., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0189-7 · Cited by 254
Yang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247 · cited 2×
“AI-based decision support tools (ADS) are increasingly used to augment human workers in complex social contexts, including social work, education, healthcare, and criminal justice [15, 19, 21, 26, 39, 66].”
BackgroundStudent Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93843-1_12 · Cited by 235 · cited 1×
“In some studies, human and machine intelligence combined has been shown to outperform either humans or AI alone [5, 24, 48], while in others human–AI partnerships have failed to improve or have even harmed decision quality, for various reasons [19, 50].”
BackgroundSelf-citation(2018) · Cited by 230
Tell me more?: the effects of mental model soundness on personalizing an intelligent agent
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207678 · Cited by 225 · cited 1×
“, the internal representations or systems of beliefs that they build up through experience, which they use to understand, explain, and predict algorithmic behavior, and then act accordingly [4, 34, 35, 61]—and we discuss the impacts that these beliefs have on their decision-making and communicative practices around th…”
BackgroundInternational Joint Conference on Artificial Intelligence (IJCAI 2017)
Belle, V. (2017) · Cited by 223
Value-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
Self-citationBrown, A., Chouldechova, A., Putnam-Hornstein, E., Tobin, A., & Vaithianathan, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300271 · Cited by 207 · cited 7×
“important role in the future of child welfare decision-making [8, 13, 45, 51, 53].”
BackgroundInfluentialStreet-Level Algorithms: A Theory at the Gaps Between Policy and Decisions
Alkhatib, A., & Bernstein, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300760 · Cited by 188 · cited 3×
“Yet in many real-world decision-making settings, humans and AI systems may have access to complementary sources of information, opening potential for each to help overcome the other’s blindspots [2, 21, 33].”
BackgroundShen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171 · cited 1×
“(2021) have called “everyday algorithm auditing” : the ways that users detect, understand, and interrogate machine behaviors via their day-to-day interactions with algorithmic systems [55].”
BackgroundSelf-citationA Conceptual Framework for Human–AI Hybrid Adaptivity in Education
Holstein, K., Aleven, V., & Rummel, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-52237-7_20 · Cited by 168
Self-citationDesigning for human–AI complementarity in K‐12 education
Holstein, K., & Aleven, V. (2022) · AI Magazine · doi:10.1002/aaai.12058 · Cited by 156
Self-citationA Human-Centered Review of Algorithms used within the U.S. Child Welfare System
Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376229 · Cited by 116 · cited 3×
“The most widely-used tools resemble simple checklists, assisting social workers in assessing the risk of a given case based on a small set of manually-entered factors [52, 53].”
BackgroundMethodGetting Ourselves Together: Data-centered participatory design research & epistemic burden
Pierre, J., Crooks, R., Currie, M., Paris, B., & Pasquetto, I. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445103 · Cited by 112 · cited 2×
“We note that the following design implications are relevant only when there is good reason to expect benefits of having an ADS in the first place, which outweigh potential harms [6, 18, 49].”
BackgroundNissen, L. B. (2020) · Journal of Technology in Human Services · doi:10.1080/15228835.2020.1796892 · Cited by 108 · cited 2×
“important role in the future of child welfare decision-making [8, 13, 45, 51, 53].”
BackgroundAlgorithms and Decision-Making in the Public Sector
Levy, K., Chasalow, K. E., & Riley, S. (2021) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-041221-023808 · Cited by 108 · cited 4×
“At the same time, AI-based judgments are themselves likely to be imperfect and biased, even if in different ways than humans [15, 32, 39, 64].”
BackgroundResultInfluentialYang, Q., Zimmerman, J., Steinfeld, A., Carey, L., & Antaki, J. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858373 · Cited by 105 · cited 1×
“[66, 67] investigated why efforts to integrate AI-based decision support tools often fail in clinical settings.”
BackgroundShifting Concepts of Value: Designing Algorithmic Decision-Support Systems for Public Services
Holten Møller, N., Shklovski, I., & Hildebrandt, T. T. (2020) · Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society · doi:10.1145/3419249.3420149 · Cited by 97 · cited 7×
“[26] found that while data scientists were fixated on developing decision support tools that predicted individual’s risk of long-term unemployment, workers desired fundamentally different forms of decision support.”
BackgroundInfluentialStreet-Level Algorithms and AI in Bureaucratic Decision-Making: A Caseworker Perspective
Ammitzbøll Flügge, A., Hildebrandt, T., & Møller, N. H. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449114 · Cited by 83 · cited 2×
“, [3, 26, 52]) regarding the need for a broader re-consideration of how ADS should be designed, evaluated, and integrated into public sector contexts.”
BackgroundTo Live in Their Utopia: Why Algorithmic Systems Create Absurd Outcomes
Alkhatib, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445740 · Cited by 81 · cited 1×
“, reflecting biases in the administrative data on which they are trained and operate upon) [13, 17] or introducing absurd errors into decision processes, which humans would not otherwise make [1, 15, 28].”
BackgroundSmith, C. E., Yu, B., Srivastava, A., Halfaker, A., Terveen, L., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376783 · Cited by 77 · cited 2×
“For example, find methods to co-design an ADS’s predictive targets with decision-makers, affected community members, data scientists, and other domain experts, while considering how to mitigate disagreements [26, 56], especially when prominent power dynamics are at play.”
BackgroundSelf-citationZytek, A., Liu, D., Vaithianathan, R., & Veeramachaneni, K. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114864 · Cited by 62
Soliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems
Cheng, H.-F., Stapleton, L., Wang, R., Bullock, P., Chouldechova, A., Wu, Z. S. S., & Zhu, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445308 · Cited by 54 · cited 4×
“, [11, 13, 15, 59]), and public sector agencies are beginning to look to the AFST as an example of what AI-assisted decision-making can or should look like in child welfare and similar contexts [51].”
BackgroundMethodInfluentialSelf-citationTowards A Process Model for Co-Creating AI Experiences
Subramonyam, H., Seifert, C., & Adar, E. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462012 · Cited by 43 · cited 1×
“sign choices for an ADS, such as specific choices of predictive targets or performancemeasures [23, 57, 68].”
BackgroundKao, Y.-F., & Venkatachalam, R. (2021) · Computational Economics · doi:10.1007/s10614-018-9803-z · Cited by 42
Child welfare informatics: A proposed subspecialty for social work
Naccarato, T. (2010) · Children and Youth Services Review · doi:10.1016/j.childyouth.2010.07.016 · Cited by 36 · cited 1×
“They do not view themselves as investigators, nor has their professional training necessarily prepared them for the investigative work with which they are tasked [44].”
BackgroundUsing a Machine Learning Tool to Support High‐Stakes Decisions in Child Protection
Vaithianathan, R., Benavides‐Prado, D., Dalton, E., Chouldechova, A., & Putnam‐Hornstein, E. (2021) · AI Magazine · doi:10.1002/j.2371-9621.2021.tb00011.x · Cited by 34 · cited 2×
“Whereas workers generally focused on ensuring children’s immediate safety and considering short-term risk when making their decisions, the AFST was intentionally designed to complement workers’ near-term focus by predicting proxies for adverse outcomes over a much longer (two-year) timespan [14, 59, 60].”
BackgroundScherz, C. (2011) · PoLAR Political and Legal Anthropology Review · doi:10.1111/j.1555-2934.2011.01137.x · Cited by 33 · cited 1×
“The stakes of the decisions these workers make day-to-day cannot be overstated, as workers face a challenging balancing act between “erring on the side of child safety” versus “erring on the side of family preservation” [13, 54].”
BackgroundReplay Enactments: Exploring Possible Futures through Historical Data
Holstein, K., Harpstead, E., Gulotta, R., & Forlizzi, J. (2020) · Proceedings of the 2020 ACM Designing Interactive Systems Conference · doi:10.1145/3357236.3395427 · Cited by 19 · cited 1×
“sign choices for an ADS, such as specific choices of predictive targets or performancemeasures [23, 57, 68].”
BackgroundSelf-citationChild Welfare Financing: What Do We Fund, How, and What Could Be Improved?
Haskins, R. (2020) · The Annals of the American Academy of Political and Social Science · doi:10.1177/0002716220970909 · Cited by 15
Teaching Humans When to Defer to a Classifier via Exemplars
Mozannar, H., Satyanarayan, A., & Sontag, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20469 · Cited by 5
Research works printed but not linked: 22
- Gagan Bansal , Besmira Nushi , Ece Kamar , Walter S Lasecki , Daniel S Weld , and Eric Horvitz . 2019 . Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance . Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 7, 1(2019) , 19. www.aaai.org Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019. Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 7, 1(2019), 19. www.aaai.org
- Hugh Beyer and Karen Holtzblatt . 1999. Contextual design. interactions 6, 1 ( 1999 ), 32–42. Hugh Beyer and Karen Holtzblatt. 1999. Contextual design. interactions 6, 1 (1999), 32–42.
- 10.1145/3449287 · doi:10.1145/3449287 · Paper page
- Marc Cherna. [n.d.]. We will use all resources to keep children safe. Pittsburgh Post-Gazette([n. d.]). https://www.post-gazette.com/opinion/letters/2018/03/23/We-will-use-all-resources-to-keep-children-safe/stories/201803230094 Online; accessed 8-September-2021. Marc Cherna. [n.d.]. We will use all resources to keep children safe. Pittsburgh Post-Gazette([n. d.]). https://www.post-gazette.com/opinion/letters/2018/03/23/We-will-use-all-resources-to-keep-children-safe/stories/201803230094 Online; accessed 8-September-2021.
- Maria De-Arteaga , Artur Dubrawski , and Alexandra Chouldechova . 2021. Leveraging expert consistency to improve algorithmic decision support. arXiv ( 2021 ), 1–33. arxiv:2101.09648http://arxiv.org/abs/2101.09648 Maria De-Arteaga, Artur Dubrawski, and Alexandra Chouldechova. 2021. Leveraging expert consistency to improve algorithmic decision support. arXiv (2021), 1–33. arxiv:2101.09648http://arxiv.org/abs/2101.09648
- Maria De-Arteaga , Riccardo Fogliato , and Alexandra Chouldechova . 2020. A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. arXiv ( 2020 ), 1–12. Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020. A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. arXiv (2020), 1–12.
- Berkeley J Dietvorst , Joseph P Simmons , and Cade Massey . 2015 . Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology : General 144 , 1 (2015), 114 . Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114.
- Ben Green . 2021. The flaws of policies requiring human oversight of government algorithms. Available at SSRN ( 2021 ). Ben Green. 2021. The flaws of policies requiring human oversight of government algorithms. Available at SSRN (2021).
- D. Jackson and G. Marx. 2017. Data mining program designed to predict child abuse proves unreliable DCFS says.http://www.jstor.org/stable/10.7758/9781610447713 D. Jackson and G. Marx. 2017. Data mining program designed to predict child abuse proves unreliable DCFS says.http://www.jstor.org/stable/10.7758/9781610447713 · doi:10.7758/9781610447713
- Daniel Kahneman , Olivier Sibony , and Cass R Sunstein . 2021 . Noise: A flaw in human judgment. Little, Brown. Daniel Kahneman, Olivier Sibony, and Cass R Sunstein. 2021. Noise: A flaw in human judgment. Little, Brown.
- Emily Keddell . 2019. Algorithmic Justice in Child Protection: Statistical Fairness, Social Justice and the Implications for Practice. Social Sciences ( 2019 ). Emily Keddell. 2019. Algorithmic Justice in Child Protection: Statistical Fairness, Social Justice and the Implications for Practice. Social Sciences (2019).
- Vivian Lai Chacha Chen Q Vera Liao Alison Smith-Renner and Chenhao Tan. 2021. Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. arXiv preprint arXiv:2112.11471(2021). Vivian Lai Chacha Chen Q Vera Liao Alison Smith-Renner and Chenhao Tan. 2021. Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. arXiv preprint arXiv:2112.11471(2021). · arXiv
- John D Lee and Katrina A See . 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 ( 2004 ), 50–80. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
- Emma Lurie and Deirdre K Mulligan . 2020 . Crowdworkers are not judges: Rethinking crowdsourced vignette studies as a risk assessment evaluation technique . Proceedings of the Workshop on Fair and Responsible AI at CHI 2020 (2020). Emma Lurie and Deirdre K Mulligan. 2020. Crowdworkers are not judges: Rethinking crowdsourced vignette studies as a risk assessment evaluation technique. Proceedings of the Workshop on Fair and Responsible AI at CHI 2020 (2020).
- Allegheny County Department of Human Services. [n.d.]. The Allegheny Family Screening Tool. https://www.alleghenycounty.us/Human-Services/News-Events/Accomplishments/Allegheny-Family-Screening-Tool.aspx. Online ; accessed 7- September - 2021 . Allegheny County Department of Human Services. [n.d.]. The Allegheny Family Screening Tool. https://www.alleghenycounty.us/Human-Services/News-Events/Accomplishments/Allegheny-Family-Screening-Tool.aspx. Online; accessed 7-September-2021.
- Allegheny County Department of Human Services. [n.d.]. Allegheny Family Screening Tool , Frequently-Asked Questions | Updated August 2018 . https://www.alleghenycountyanalytics.us/wp-content/uploads/2018/10/17-ACDHS-11_AFST_102518.pdf. Online; accessed 8-September-2021. Allegheny County Department of Human Services. [n.d.]. Allegheny Family Screening Tool, Frequently-Asked Questions | Updated August 2018. https://www.alleghenycountyanalytics.us/wp-content/uploads/2018/10/17-ACDHS-11_AFST_102518.pdf. Online; accessed 8-September-2021.
- Anjana Samant , Aaron Horowitz , Kath Xu , and Sophie Beiers . 2021. Family surveillance by algorithm: The rapidly spreading tools few have heard of . American Civil Liberties Union (ACLU)( 2021 ). https://www.aclu.org/sites/default/files/field_document/2021.09.28a_family_surveillance_by_algorithm.pdf Anjana Samant, Aaron Horowitz, Kath Xu, and Sophie Beiers. 2021. Family surveillance by algorithm: The rapidly spreading tools few have heard of. American Civil Liberties Union (ACLU)(2021). https://www.aclu.org/sites/default/files/field_document/2021.09.28a_family_surveillance_by_algorithm.pdf
- Devansh Saxena , Karla Badillo-Urquiola , Pamela Wisniewski , and Shion Guha . 2021. A framework of high-stakes algorithmic decision-making for the public sector developed through a case study of child welfare. arXiv 5 , October ( 2021 ). arxiv:arXiv:2107.03487v2 Devansh Saxena, Karla Badillo-Urquiola, Pamela Wisniewski, and Shion Guha. 2021. A framework of high-stakes algorithmic decision-making for the public sector developed through a case study of child welfare. arXiv 5, October (2021). arxiv:arXiv:2107.03487v2 · arXiv
- Sarah Tan , Julius Adebayo , and Kori Inkpen . 2018. Investigating human + machine complementarity for recidivism predictions. arXiv ( 2018 ). arxiv:arXiv:1808.09123v2 Sarah Tan, Julius Adebayo, and Kori Inkpen. 2018. Investigating human + machine complementarity for recidivism predictions. arXiv (2018). arxiv:arXiv:1808.09123v2 · arXiv
- Rhema Vaithianathan , Emily Putnam-Hornstein , Nan Jiang , Parma Nand , and Tim Maloney . 2017. Developing predictive models to support child maltreatment hotline screening decisions: Allegheny County methodology and implementation . Center for Social data Analytics( 2017 ). Rhema Vaithianathan, Emily Putnam-Hornstein, Nan Jiang, Parma Nand, and Tim Maloney. 2017. Developing predictive models to support child maltreatment hotline screening decisions: Allegheny County methodology and implementation. Center for Social data Analytics(2017).
- Elmira van den Broek , Anastasia Sergeeva , and Marleen Huysman . 2020 . Hiring algorithms: An ethnography of fairness in practice . 40th International Conference on Information Systems, ICIS 2019 (2020). Elmira van den Broek, Anastasia Sergeeva, and Marleen Huysman. 2020. Hiring algorithms: An ethnography of fairness in practice. 40th International Conference on Information Systems, ICIS 2019 (2020).
- MW Van Someren , YF Barnard , and JAC Sandberg . 1994. The think aloud method: a practical approach to modelling cognitive . London : AcademicPress( 1994 ). MW Van Someren, YF Barnard, and JAC Sandberg. 1994. The think aloud method: a practical approach to modelling cognitive. London: AcademicPress(1994).
Cited by
Shi, Y., Gao, T., Jiao, X., & Cao, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610217 · Cited by 170
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220
“Different from a small but growing number of qualitative studies that aim to understand the experience of decision-makers retrospectively [14, 41, 43, 60, 71], we adopt a think-aloud protocol to investigate the real-time process as participants engage with AI output [39, 77], and use the data to understand why two typ…”
MethodSivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581075 · Cited by 141
“While expert consensus can serve as a useful proxy for the ground truth [40], the accuracy of a real-world decision is often fundamentally unknowable and contentious [47].”
BackgroundSelf-citationYang, Z., Xu, X., Yao, B., Rogers, E., Zhang, S., Intille, S., Shara, N., Gao, G. G., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3659625 · Cited by 120
Challenges of responsible AI in practice: scoping review and recommended actions
Sadek, M., Kallina, E., Bohné, T., Mougenot, C., Calvo, R. A., & Cave, S. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-01880-9 · Cited by 79
“…or not to detain criminal defend-ants (Dressel and Farid 2018); analysing which child protection requests seem credible (Chouldechova et al. 2018; Kawakami et al. 2022); in HR related topics (Mujtaba and Mahapatra 2019); facial recognition (Lohr 2018); detecting hate speech on social media…”
Metadata
- Authors
- Kawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K.
- Affiliations
- Carnegie Mellon University (US)
- University of Minnesota (US)
- University of California, Berkeley (US)
- ORCID
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-27
- Pages
- 1-18
- OpenAlex
- W4225001143
- Cited by
- 143
- References
- 42
- Citations by year
- 2026: 25 · 2025: 37 · 2024: 44 · 2023: 30 · 2022: 7
- FWCI
- 29.32
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (National Science Foundation) (1939606)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- human-AI collaboration, algorithmic decision support, child welfare, child maltreatment screening, AI-assisted decision-making, contextual information, trust in AI predictions, organizational incentives, public sector organizations, design implications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.