The Principles and Limits of Algorithm-in-the-Loop Decision Making
The DOI matches its Crossref record; 61 of the 64 references it cites were found in a registry or an index.
Abstract
The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an "algorithm-in-the-loop'' process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decisions. Society lacks both clear normative principles regarding how people should collaborate with algorithms as well as robust empirical evidence about how people do collaborate with algorithms. Given research suggesting that people struggle to interpret machine learning models and to incorporate them into their decisions---sometimes leading these models to produce unexpected outcomes---it is essential to consider how different ways of presenting models and structuring human-algorithm interactions affect the quality and type of decisions made. This paper contributes to such research in two ways. First, we posited three principles as essential to ethical and responsible algorithm-in-the-loop decision making. Second, through a controlled experimental study on Amazon Mechanical Turk, we evaluated whether people satisfy these principles when making predictions with the aid of a risk assessment. We studied human predictions in two contexts (pretrial release and financial lending) and under several conditions for risk assessment presentation and structure. Although these conditions did influence participant behaviors and in some cases improved performance, only one desideratum was consistently satisfied. Under all conditions, our study participants 1) were unable to effectively evaluate the accuracy of their own or the risk assessment's predictions, 2) did not calibrate their reliance on the risk assessment based on the risk assessment's performance, and 3) exhibited bias in their interactions with the risk assessment. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“there has been increasing recent research efforts [8, 15, 22, 41, 58, 65] that highlight the importance of involving stakeholders to understand their practices and needs [15, 41, 65] and socio-environmental factors [8] for the design and evaluation of a system.”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [22]
In the citing paper
“However, many empirical studies on human-AI decision-making did not observe complementary performance [6,16,37,52,58,61,68,82,84,87].”
From Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations (2023) · cites this paper as [37]
In the citing paper
“We note that these approaches correspond to the Update and Feedback methods of Green & Chen [27], which were effective, albeit not in the complementary zone.”
From Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021) · cites this paper as [27]
In the citing paper
“More interestingly, communication failure can also stem from a deeper inability to understand how AI makes decisions and how likely AI may err in specific conditions [61] .”
From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [61]
In the citing paper
“However, many studies have shown that human-AI teams under-perform the AI alone in tasks [3, 4, 6, 14, 18, 20, 21, 30].”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [14]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
What Should Be Done with Equity Theory?
Leventhal, G. S. (1980) · Social Exchange · doi:10.1007/978-1-4613-3087-5_2 · Cited by 3,010 · cited 2×
“For example, the principle of procedural justice requires that decisions be (among other things) accurate, fair, consistent, correctable, and ethical [55].”
BackgroundAlgorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
Lee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Research works printed but not linked: 12
- Alex Albright . 2019. If You Give a Judge a Risk Score: Evidence from Kentucky Bail Decisions . The John M. Olin Center for Law , Economics , and Business Fellows' Discussion Paper Series 85 ( 2019 ). Alex Albright. 2019. If You Give a Judge a Risk Score: Evidence from Kentucky Bail Decisions. The John M. Olin Center for Law, Economics, and Business Fellows' Discussion Paper Series 85 (2019).
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016. Machine Bias. ProPublica ( 2016 ). https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias. ProPublica (2016). https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
- Laura Bliss . 2018. Former Uber Backup Driver: 'We Saw This Coming'. CityLab ( 2018 ). https://www.citylab.com/transportation/2018/03/former-uber-backup-driver-we-saw-this-coming/556427/ Laura Bliss. 2018. Former Uber Backup Driver: 'We Saw This Coming'. CityLab (2018). https://www.citylab.com/transportation/2018/03/former-uber-backup-driver-we-saw-this-coming/556427/
- Darwin Bond-Graham and Ali Winston . 2013. All Tomorrow's Crimes: The Future of Policing Looks a Lot Like Good Branding. SF Weekly ( 2013 ). http://archives.sfweekly.com/sanfrancisco/all-tomorrows-crimes-the-future-of-policing-looks-a-lot-like-good-branding/Content?oid=2827968 Darwin Bond-Graham and Ali Winston. 2013. All Tomorrow's Crimes: The Future of Policing Looks a Lot Like Good Branding. SF Weekly (2013). http://archives.sfweekly.com/sanfrancisco/all-tomorrows-crimes-the-future-of-policing-looks-a-lot-like-good-branding/Content?oid=2827968
- Pamela M. Casey , Roger K. Warren , and Jennifer K . Elek . 2011 . Using Offender Risk and Needs Assessment Information at Sentencing : Guidance for Courts from a National Working Group .National Center for State Courts . Pamela M. Casey, Roger K. Warren, and Jennifer K. Elek. 2011. Using Offender Risk and Needs Assessment Information at Sentencing: Guidance for Courts from a National Working Group .National Center for State Courts.
- Thomas H. Cohen , Bailey Pendergast , and Scott W. VanBenschoten . 2016 . Examining overrides of risk classifications for offenders on federal supervision . Federal Probation , Vol. 80 , 1 (2016), 12 . Thomas H. Cohen, Bailey Pendergast, and Scott W. VanBenschoten. 2016. Examining overrides of risk classifications for offenders on federal supervision. Federal Probation, Vol. 80, 1 (2016), 12.
- Mary L. Cummings . 2006. Automation and Accountability in Decision Support System Interface Design. Journal of Technology Studies ( 2006 ). Mary L. Cummings. 2006. Automation and Accountability in Decision Support System Interface Design. Journal of Technology Studies (2006).
- William Dieterich , Christina Mendoza , and Tim Brennan . 2016. COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity . Northpointe Inc . ( 2016 ). William Dieterich, Christina Mendoza, and Tim Brennan. 2016. COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity. Northpointe Inc. (2016).
- Jeff Larson , Surya Mattu , Lauren Kirchner , and Julia Angwin . 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica ( 2016 ). https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. ProPublica (2016). https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
- 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
- Frank Main . 2016. Cook County judges not following bail recommendations: study. Chicago Sun-Times ( 2016 ). https://chicago.suntimes.com/chicago-news/cook-county-judges-not-following-bail-recommendations-study-find/ Frank Main. 2016. Cook County judges not following bail recommendations: study. Chicago Sun-Times (2016). https://chicago.suntimes.com/chicago-news/cook-county-judges-not-following-bail-recommendations-study-find/
- Megan T. Stevenson and Jennifer L . Doleac . 2018 . The Roadblock to Reform. The American Constitution Society ( 2018). https://www.acslaw.org/wp-content/uploads/2018/11/RoadblockToReformReport.pdf Megan T. Stevenson and Jennifer L. Doleac. 2018. The Roadblock to Reform. The American Constitution Society (2018). https://www.acslaw.org/wp-content/uploads/2018/11/RoadblockToReformReport.pdf
Cited by
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
“Prior studies in human-AI decision-making have shown the anchoring bias that occurs by presenting people with AI’s recommendation before allowing them to make a decision first [26].”
BackgroundResultInfluentialDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“’s initial answer or only having the AI present an explanation if it disagreed with the human’s choice. We note that these approaches correspond to the Update and Feedback methods of Green & Chen [21], which were effective, albeit not in the complementary zone. However, by delaying display of the AI’s input until…”
BackgroundMethodResultInfluentialAlon-Barkat, S., & Busuioc, M. (2023) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muac007 · Cited by 469
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
“Human-AI performance generally only exceeds human-only performance [8, 26, 30, 55, 65, 78, 97]; stark performance differences between humans and AIs make it such that one would achieve the highest performance”
BackgroundProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
“Although not universally effective [38], these strategies have shown promising results as they improve decision-making performance, both if the human is assisted [17, 34] or is not assisted by a DSS [31].”
BackgroundMethodResultInfluentialWang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320
“1An example of task that may not be applicable to some decision making context is to examine human decision maker’s ability in detecting fairness problems of the AI model or utilizing the AI model more fairly, in the presence of explanations [23, 30].”
MethodThe Flaws of Policies Requiring Human Oversight of Government Algorithms
Green, B. (2021) · doi:10.2139/ssrn.3921216 · Cited by 259
Self-citationJacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247
“The lack of change in con fi dence scores suggests that participants struggled to calibrate their own performance, which can lead to poor calibration of ML performance 11 .”
BackgroundHuman factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219
“…(HCI) have devoted effort to understanding which ML models can better conform to human understanding of prediction tasks [77], how people’s interaction with ML models affects the quality of decisions [20], and how data scientists use particular tools in conducting their interpretability tasks [32].”
Three Challenges for AI-Assisted Decision-Making
Steyvers, M., & Kumar, A. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231181102 · Cited by 217
“…human in isolation (Bansal, Wu, et al., 2021; Tejeda et al., 2022), other studies have shown that the human DM does not contribute to such performance and that the AI acting by itself leads to better performance (Feng & Boyd-Graber, 2019; Green & Chen, 2019; Lai & Tan, 2019; Y. Zhang et al., 2020).”
The flaws of policies requiring human oversight of government algorithms
Green, B. (2022) · Computer Law & Security Review · doi:10.1016/j.clsr.2022.105681 · Cited by 263
“On the other hand, the use of algorithms by governments has been a source of numerous injustices ( Angwin et al., 2016 ; Calo & Citron, 2021 ; Eubanks, 2018 ; Green, 2019 ; Henley & Booth, 2020 ).”
BackgroundMethodInfluentialSelf-citationDeciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making
Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512930 · Cited by 216
“Research has shown that people do not calibrate their reliance on the AI based on its accuracy (Green and Chen, 2019).”
BackgroundMethodLai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
“…$10 [1, 16, 23, 30, 36, 45, 46, 68, 73, 82, 84, 101, 112], less than equals to $15 [2, 24, 38, 41, 47, 48, 72, 94], more than $15 [5, 9, 11, 15, 43, 44, 50, 57, 70, 72, 99], bonus available [2, 16, 21, 23, 29, 30, 41, 43, 61, 62, 71, 91, 105] Table 3: Types of tasks grouped by field in human-AI…”
BackgroundMethodInfluentialRapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186
“72,73 Providing uncertainty estimates along with explanations may also improve trust calibration, but research remains to be done in this area.”
BackgroundJacobs, M., He, J., F. Pradier, M., Lam, B., Ahn, A. C., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445385 · Cited by 177
“We have seen several examples in recent years of AI predictions leading to biased or unfair behaviors [20, 45].”
BackgroundMiller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
“(2) Control: Evaluative AI explicitly hands control of which hypotheses are investigated and prioritised to the decision maker, resulting a machine-in-the-loop paradigm [7, 8], rather than a human-in-the-loop paradigm.”
BackgroundInfluentialNaiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102941 · Cited by 170
Steiger, M., Bharucha, T. J., Venkatagiri, S., Riedl, M. J., & Lease, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445092 · Cited by 222
Algorithmic realism: expanding the boundaries of algorithmic thought
Green, B., & Viljoen, S. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372840 · Cited by 161
“Yet such assessments typically overlook the ways in which these “fair” algorithms can lead to unfair social impacts, whether through biased uses by practitioners [2, 29, 55, 56], entrenching unjust policies [54], distorting deliberative processes [51], or shifting control of governance toward unaccountable private act…”
BackgroundSelf-citationWysocki, O., Davies, J. K., Vigo, M., Armstrong, A. C., Landers, D., Lee, R., & Freitas, A. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2022.103839 · Cited by 161
“Although explainable ML is advocated as the methodological silver bullet for addressing transparency issues [1], some user studies, similarly to this paper, show evidence of a limited or even no impact of the explanation on performance in a task [54, 18, 55, 56, 57].”
Humans inherit artificial intelligence biases
Vicente, L., & Matute, H. (2023) · Scientific Reports · doi:10.1038/s41598-023-42384-8 · Cited by 158
“These impressions were confirmed by a mixed ANOVA 3 (Block) × 2 (Group) that showed a main effect of Group, F (1,195) = 7.”
BackgroundMethodInfluentialUnderstanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220
“However, many empirical studies on human-AI decision-making do not observe complementary performance [6, 15, 34, 47, 53, 56, 60, 71, 73, 76].”
BackgroundKawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143
“In some studies, human and machine intelligence combined has been shown to outperform either humans or AI alone [5, 24, 48], while in others human–AI partnerships have failed to improve or have even harmed decision quality, for various reasons [19, 50].”
BackgroundHuman-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145
“Another rich area to study human-AI collaboration is AI-assisted [13, 83, 89] or “machine/algorithm-in-the-loop” decision-making [37, 51].”
BackgroundInfluentialHow Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review
Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534164 · Cited by 129
Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131
“More recently, with the widespread usage of machine learning (ML) based technologies in aiding human decision-making, a new line of research examining human’s reliance on ML emerges [22, 55, 64].”
BackgroundComplementarity in human-AI collaboration: concept, sources, and evidence
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2475962 · Cited by 237
“Specifically, we focused on a team setting in which a human decision-maker has access to AI advice and is subsequently responsible for making the final team decision based on his/her judgment and that of the AI (Green & Chen, 2019).”
Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Gajos, K. Z., & Mamykina, L. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511138 · Cited by 127
“• Update This condition is illustrated in Figure 1 (bottom) and follows the general design used in prior research [9, 22, 29].”
BackgroundMethodResultInfluentialMetahuman systems = humans + machines that learn
Lyytinen, K., Nickerson, J. V., & King, J. L. (2020) · Journal of Information Technology · doi:10.1177/0268396220915917 · Cited by 187
“Using a machine in a cognitive loop will, for example, causes humans also to anthropomorphize the machine expert (Cai et al., 2019), or only selectively follow machine advice when it reinforces human’s own biases (Green and Chen, 2019).”
Background"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123
“In addition to studies mentioned in the introduction, researchers have investigated other desiderata of explanations [5, 8, 17, 18, 32, 49, 65].”
BackgroundRams, hounds and white boxes: investigating human–AI collaboration protocols in medical diagnosis
Cabitza, F., Campagner, A., Ronzio, L., Cameli, M., Mandoli, G. E., Pastore, M. C., Sconfienza, L. M., Folgado, D., Barandas, M., & Gamboa, H. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102506 · Cited by 117
Wilder, B., Horvitz, E., & Kamar, E. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/212 · Cited by 118
“How knowledge workers think generative AI will (not) transform their industries,”
Woodruff, A., Shelby, R., Kelley, P. G., Rousso-Schindler, S., Smith-Loud, J., & Wilcox, L. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642700 · Cited by 105
“However, the HITL literature points out serious, unsolved practical problems that were not apparent to participants.”
BackgroundInfluentialHou, Y. T.-Y., & Jung, M. F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479864 · Cited by 105
“Second, although there is increasing interest in CSCW on AI-supported decision making [6,14,15,19], the CSCW literature has paid little attention to growing work on algorithm aversion and algorithm appreciation.”
Background“Who goes first? Influences of human-AI workflow on decision making in clinical imaging,”
Fogliato, R., Chappidi, S., Lungren, M., Fisher, P., Wilson, D., Fitzke, M., Parkinson, M., Horvitz, E., Inkpen, K., & Nushi, B. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533193 · Cited by 103
“Green and Chen [34] find that requiring participants to register provisional predictions before AI recommendations are revealed results in marginal gains in overall predictive performance.”
BackgroundDoes Explainable Artificial Intelligence Improve Human Decision-Making?
Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcioglu, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16819 · Cited by 106
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99
“3533113 services [32, 43, 53], and criminal justice [26, 38], among many other areas.”
BackgroundLiu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111
“On the other hand, similar to ethical concerns about static explanations raised in prior work [2, 20, 21], interactive explanations might reinforce”
BackgroundMethodResultInfluentialArtificial intelligence in government: Concepts, standards, and a unified framework
Straub, V. J., Morgan, D., Bright, J., & Margetts, H. Z. (2023) · Government Information Quarterly · doi:10.1016/j.giq.2023.101881 · Cited by 175
Keddell, E. (2019) · Social Sciences · doi:10.3390/socsci8100281 · Cited by 109
“While some tool developers have made inroads into responding to some of these criticisms, several issues remain unresolved (Chouldechova et al. 2018; Gurses et al. 2019).”
InfluentialA Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making
Schemmer, M., Hemmer, P., Nitsche, M., Kühl, N., & Vössing, M. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534128 · Cited by 91
“With the ongoing development of XAI techniques, researchers have started to evaluate AI with and without explanations to assess whether their utility for better decision-making can be quantified [4, 8, 9, 12, 14, 22, 23, 27, 36, 37, 39, 60, 65, 67].”
BackgroundHow Child Welfare Workers Reduce Racial Disparities in Algorithmic Decisions
Cheng, H.-F., Stapleton, L., Kawakami, A., Sivaraman, V., Cheng, Y., Qing, D., Perer, A., Holstein, K., Wu, Z. S., & Zhu, H. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501831 · Cited by 134
“2 RELATED WORK 2.1 Algorithm-in-the-loop decision-making “Algorithm-in-the-loop” decision-making is commonly characterized by frst having an algorithm-produced prediction or classif-cation, with a human making the fnal decision after considering an algorithmic recommendation [38, 85].”
BackgroundMethodInfluentialThe false promise of risk assessments: epistemic reform and the limits of fairness
Green, B. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372869 · Cited by 80
Self-citationORES: Lowering Barriers with Participatory Machine Learning in Wikipedia
Halfaker, A., & Geiger, R. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415219 · Cited by 82
“CSCW and HCI has also long studied human-in-the-loop systems [14, 37, 111], as well as explored implications of “algorithm-in-the-loop” socio-technical systems [44].”
Prabhudesai, S., Yang, L., Asthana, S., Huan, X., Liao, Q. V., & Banovic, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584033 · Cited by 82
“More recently, Machine Learning (ML) has been used in DSS, to extract patterns from large volumes of historic data, and make inferences to inform decision-makers [48].”
MethodHuman-AI Collaboration for UX Evaluation: Effects of Explanation and Synchronization
Fan, M., Yang, X., Yu, T., Liao, Q. V., & Zhao, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512943 · Cited by 81
“For example, an “algorithm-in-the-loop” process [31] is adopted in many high-stakes decision-making contexts, where ML models are used to inform people its assessment or to alert cases falling in a targeted category.”
MethodFok, R., & Weld, D. S. (2024) · AI Magazine · doi:10.1002/aaai.12182 · Cited by 81
Effects of Explanations in AI-Assisted Decision Making: Principles and Comparisons
Wang, X., & Yin, M. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3519266 · Cited by 74
“1An example of a task that may not be applicable to some decision making context is to have the human decision maker detect fairness problems of the AI model or utilize the AI model in a fair way [26, 34].”
MethodBuilding machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi‐Xuan, T., Ho, M. K., Mansinghka, V. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 216
Lee, M. H., & Chew, C. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610218 · Cited by 78
“In addition, an additional study is required to investigate how people can effectively evaluate the performance and trustworthiness of an ML model and calibrate their trust and reliance to improve human-AI/algorithm interaction [22].”
BackgroundExplanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making
Schoeffer, J., De‐Arteaga, M., & Kühl, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642621 · Cited by 166
“A recent meta-study [108] on the effectiveness of explanations, however, implies that explanations in most empirical studies did not yield any significant benefits with respect to decision-making accuracy; e.g., in [4, 47, 75, 87, 128].”
ResultToward involving end-users in interactive human-in-the-loop AI fairness
Nakao, Y., Stumpf, S., Ahmed, S., Naseer, A., & Strappelli, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514258 · Cited by 69
“In the context of financial decisions, there has been some work to involve users in assessing the credit risk, for example, to investigate decision-making [29], or to examine people’s fairness perceptions and definitions [63].”
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
“Decision Making Green and Chen [37] AI-supported decisions were more accurate, but not more reliable or fair Mixed outcomes”
BackgroundGreen, B., & Chen, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479562 · Cited by 69
“Several studies have found that algorithmic advice can improve the accuracy of human predictions, but people’s decisions about when and how to diverge from algorithmic recommendations are typically incorrect [28, 29, 32, 46].”
BackgroundResultInfluentialSelf-citationThe impact of AI errors in a human-in-the-loop process
Agudo, U., Liberal, K. G., Arrese, M., & Matute, H. (2024) · Cognitive Research: Principles and Implications · doi:10.1186/s41235-023-00529-3 · Cited by 66
“Although in this research we have used accuracy as a measure to assess the human-in-the-loop process, decisions in the field of justice must be fair, correctable, and ethical among other things, in addition to being accurate (Green & Chen, 2019b).”
BackgroundMethodResultInfluentialBaroni, I., Re Calegari, G., Scandolari, D., & Celino, I. (2022) · Human Computation · doi:10.15346/hc.v9i1.134 · Cited by 65
“Ethical guidelines for the adoption of AI solutions have been defined (AI HLEG, 2019) and legislation is being introduced to regulate the use of AI, especially in high-risk situations (European Commission, 2021), even if the ethical and legal aspect of AI involvement in decision making is still an open point (Green an…”
BackgroundHard choices in artificial intelligence
Dobbe, R., Krendl Gilbert, T., & Mintz, Y. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103555 · Cited by 65
The Role of Explainable AI in the Research Field of AI Ethics
Vainio-Pekka, H., Agbese, M., Jantunen, M., Vakkuri, V., Mikkonen, T., Rousi, R. A., & Abrahamsson, P. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3599974 · Cited by 99
The Impact of Imperfect XAI on Human-AI Decision-Making
Morrison, K., Spitzer, P., Turri, V., Feng, M., Kühl, N., & Perer, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641022 · Cited by 60
“The structure of phases one and two are corroborated with previous work [30].”
ResultGroh, M., Harris, C., Daneshjou, R., Badri, O., & Koochek, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555634 · Cited by 101
“Moreover, these systems should be transparent such that medical experts can reliably assess algorithmic performance [39, 47].”
BackgroundConsistent Estimators for Learning to Defer to an Expert
Mozannar, H., & Sontag, D. (2020) · arXiv · doi:10.48550/arxiv.2006.01862 · Cited by 61
“These models are either used as a tool to help the downstream human decision maker with judges relying on algorithmic risk assessment tools [GC19b] and risk scores being used in the ICU [FHH + 17], or instead these learning models are solely used to make the final prediction on a selected subset of…”
MethodChiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581015 · Cited by 98
Kim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58
“…providing information about an AI system’s accuracy [43, 128, 129] and (un)certainty [6, 14, 17, 41, 132], explanations of outputs [6, 14, 17, 40, 41, 69, 132], and onboarding materials [18, 68] impact user reliance, as well as the roles played by human intuition [20], task complexity [101,…”
Pater, J., Coupe, A., Pfafman, R., Phelan, C., Toscos, T., & Jacobs, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445734 · Cited by 52
Selective Explanations: Leveraging Human Input to Align Explainable AI
Lai, V. W.-M., Zhang, Y., Chen, C., Liao, Q. V., & Tan, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610206 · Cited by 93
“However, empirical human-subject studies that examine how people interact with state-of-the-art XAI techniques have not found conclusive evidence that these techniques help end-users better complete AI-assisted tasks [6, 11, 13, 41, 56, 57].”
BackgroundFok, R., & Weld, D. S. (2023) · arXiv · doi:10.48550/arxiv.2305.07722 · Cited by 54
“Unfortunately, achieving this notion of appropriate reliance through explanations has been elusive [3, 7, 16, 18, 31, 35, 37, 39, 50, 57–59, 63, 72, 76].”
BackgroundInfluentialShen, H., Jin, H., Cabrera, Á. A., Perer, A., Zhu, H., & Hong, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415224 · Cited by 79
“, race) as targeting variables [44], which statistical definitions of fairness people perceive to be the fairest in the context of loan decisions [47], as well as how humans use AI systems to make decisions [23, 24].”
BackgroundAI-Assisted Decision-making: a Cognitive Modeling Approach to Infer Latent Reliance Strategies
Tejeda, H., Kumar, A., Smyth, P., & Steyvers, M. (2022) · Computational Brain & Behavior · doi:10.1007/s42113-022-00157-y · Cited by 53
“Some researchers have argued that the sequential paradigm is superior to the concurrent paradigm because the initial unassisted prediction encourages independent reflection which could lead to retrieval of additional problem-relevant information (Green and Chen, 2019).”
BackgroundJansen, B. J. J., Salminen, J. O., Jung, S., & Guan, K. (2021) · Synthesis lectures on human-centered informatics · doi:10.1007/978-3-031-02231-9 · Cited by 70
Kuang, E., Li, M., Fan, M., & Shinohara, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642168 · Cited by 86
Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure
Swaroop, S., Buçinca, Z., Gajos, K. Z., & Doshi-Velez, F. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645206 · Cited by 49
“AI-after has been found to improve decision accuracy compared to AI-before [26] and to reduce overreliance on AI recommendations [6].”
BackgroundResultInfluentialMa, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713423 · Cited by 48
“With remarkable technological advancements, AI has been increasingly used to support people in making decisions in various domains, including criminal justice [32, 35], admissions [25, 140], financial investment [53], and medical diagnosis [21, 79], among others.”
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 115
“This setup is similar to the update condition in work by Green and Chen [39].”
Eisbach, S., Langer, M., & Hertel, G. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100015 · Cited by 46
Enabling Fairness in Healthcare Through Machine Learning
Grote, T., & Keeling, G. (2022) · Ethics and Information Technology · doi:10.1007/s10676-022-09658-7 · Cited by 59
“For example, a study by Green and Chen (2019) investigated how the use of predictive models affects actual decision-making processes within the context of criminal justice.”
BackgroundInfluentialKawakami, A., Sivaraman, V., Stapleton, L., Cheng, H.-F., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533556 · Cited by 74
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42
“For example, recidivism prediction algorithms have been used to help judges determine whether defendants are likely to re-offend [26, 31, 65, 69], medical diseases are being diagnosed with AI systems [19, 57, 62], and loan risk prediction algorithms are employed to approve or reject loan applications [11, 25, 38, 95].”
BackgroundMethodEffect of Explanation Conceptualisations on Trust in AI-assisted Credibility Assessment
Pareek, S., van Berkel, N., Velloso, E., & Goncalves, J. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686922 · Cited by 42
“A multitude of contextual and personal determinants, such as risk perception [35, 36], AI literacy [17], confidence [124], and level of anthropomorphism of the automated aid [54], have been found to influence trust.”
BackgroundSalimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“For example, in high-stake scenarios such as in the medical [39, 61, 67, 97], legal [6, 81, 86, 131], and financial [27, 36, 43, 45, 46] domains, individuals tend to exhibit a preference for human decision-makers over AI systems.”
Machine learning in the loop for tuberculosis diagnosis support
Orjuela-Cañón, Á. D., Jutinico, A. L., Awad, C., Vergara, E., & Palencia, M. A. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.876949 · Cited by 51
He, G., Buijsman, S., & Gadiraju, U. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610067 · Cited by 71
“…setup of an initial unaided decision and the presentation of system advice in order to make a second and final choice is similar to the update condition in [27], and in line with findings that people first make a decision on their own and only then decide whether to incorporate system advice [26].”
BackgroundMethodResultInfluentialHow was my performance? Exploring the role of anchoring bias in AI-assisted decision making
Carter, L. D., & Liu, D. (2025) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2025.102875 · Cited by 63
Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making
Lu, Z., Wang, D., & Yin, M. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653708 · Cited by 77
“…Carton et al. [11] found that the feature-based explanation does not help people detect online toxic content more accurately when they are assisted by toxic text classifiers, but the same type of explanation was shown to improve people’s accuracy in AI-assisted recidivism risk assessments [31].”
ResultCharacterizing Interaction Design by Its Ideals: A Discipline in Transition
Höök, K., & Löwgren, J. (2021) · She ji · doi:10.1016/j.sheji.2020.12.001 · Cited by 45
Holstein, K., De‐Arteaga, M., Tumati, L., & Cheng, Y. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579628 · Cited by 62
“In other studies, human–AI collaboration has failed to improve or have even harmed decision quality [20, 39, 43].”
BackgroundNudging through Friction: An Approach for Calibrating Trust in Explainable AI
Naiseh, M., Al-Mansoori, R. S., Al‐Thani, D., Jiang, N., & Ali, R. (2021) · International Conference on Behavioral, Economic, and Socio-Cultural Computing · doi:10.1109/besc53957.2021.9635271 · Cited by 46
Inherent Limitations of AI Fairness
Buyl, M., & Bie, T. D. (2024) · Communications of the ACM · doi:10.1145/3624700 · Cited by 73
“38 For example, a study conducted by Green and Chen 23 asked participants to give a final ruling on defendants in a pre-trial setting.”
BackgroundHa, T., & Kim, S. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2285640 · Cited by 59
“Pre-reviewing has also been noted to improve performance in AI-supported decision-making and prevent potential bias (Buc¸inca et al., 2020; 2021; Green & Chen, 2019).”
Never tell me the odds: Investigating pro-hoc explanations in medical decision making
Cabitza, F., Natali, C., Famiglini, L., Campagner, A., Caccavella, V. M., & Gallazzi, E. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102819 · Cited by 58
Understanding Human-AI Workflows for Generating Personas
Shin, J., Hedderich, M. A., Rey, B. J., Lucero, A., & Oulasvirta, A. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660729 · Cited by 82
“Accordingly, we designed three persona-generation workfows by varying the reliance between user researchers and LLMs (Figure 3).”
MethodInvestigations of Performance and Bias in Human-AI Teamwork in Hiring
Peng, A., Nushi, B., Kıcıman, E., Inkpen, K. M., & Kamar, E. S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21468 · Cited by 71
“Role of Human-AI interaction in selective prediction,”
Bondi, E., Koster, R., Sheahan, H., Chadwick, M., Bachrach, Y., Cemgil, T., Paquet, U., & Dvijotham, K. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20465 · Cited by 30
Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
Chu, E., Roy, D., & Andreas, J. (2020) · arXiv · doi:10.48550/arxiv.2007.12248 · Cited by 34
“Other work has shown feature-wise importances for low-dimensional inputs to both slightly increase [17] and decrease accuracy [49].”
BackgroundMethodInfluentialInvestigations of Performance and Bias in Human-AI Teamwork in Hiring
Peng, A., Nushi, B., Kiciman, E., Inkpen, K., & Kamar, E. (2022) · arXiv · doi:10.48550/arxiv.2202.11812 · Cited by 34
Spatola, N. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100099 · Cited by 73
Buçinca, Z., Swaroop, S., Paluch, A. E., Doshi‐Velez, F., & Gajos, K. Z. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713229 · Cited by 75
“This contrasts with prior research showing that providing unilateral explanations after an initial decision reduces overreliance [14, 36], as people are less likely to follow incorrect AI advice once they have made a decision.”
de Jong, S., Paananen, V., Tag, B., & van Berkel, N. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710946 · Cited by 32
“Several papers on AI-assisted decision-making investigated the use of cognitive forcing functions [18] to reduce overreliance on AI suggestions and explanations [6, 29, 57].”
InfluentialOwning Decisions: AI Decision-Support and the Attributability-Gap
Zeiser, J. (2024) · Science and Engineering Ethics · doi:10.1007/s11948-024-00485-1 · Cited by 59
“How - ever, most AI systems today actually operate as decision-support systems (AI-DSS), or “algorithms in the loop” as Green and Chen (2019) put it.”
Background
Metadata
- Authors
- Green, B., & Chen, Y.
- Affiliations
- Harvard University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2019-11-07
- Volume
- 3
- Issue
- CSCW
- Pages
- 1-24
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2979893369
- Cited by
- 392
- References
- 64
- Citations by year
- 2026: 94 · 2025: 61 · 2024: 62 · 2023: 54 · 2022: 58 · 2021: 36 · 2020: 22 · 2019: 5
- FWCI
- 52.66
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Funders
- National Science Foundation (DGE1745303)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- human-algorithm interaction, risk assessment, pretrial release, lending, interpretability, socio-technical context
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.