The DOI matches its DataCite record.
Investigations of Performance and Bias in Human-AI Teamwork in Hiring
Open accessAbstract
In AI-assisted decision-making, effective hybrid (human-AI) teamwork is not solely dependent on AI performance alone, but also on its impact on human decision-making. While prior work studies the effects of model accuracy on humans, we endeavour here to investigate the complex dynamics of how both a model's predictive performance and bias may transfer to humans in a recommendation-aided decision task. We consider the domain of ML-assisted hiring, where humans -- operating in a constrained selection setting -- can choose whether they wish to utilize a trained model's inferences to help select candidates from written biographies. We conduct a large-scale user study leveraging a re-created dataset of real bios from prior work, where humans predict the ground truth occupation of given candidates with and without the help of three different NLP classifiers (random, bag-of-words, and deep neural network). Our results demonstrate that while high-performance models significantly improve human performance in a hybrid setting, some models mitigate hybrid bias while others accentuate it. We examine these findings through the lens of decision conformity and observe that our model architecture choices have an impact on human-AI conformity and bias, motivating the explicit need to assess these complex dynamics prior to deployment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pretrained models often show unintended behavior such as generating harmful, biased, untruthful, and low-quality content.
“Moreover, pretrained models often show unintended behavior such as generating harmful, biased, untruthful, and low-quality content [142].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [142]
The Results (p. 5) report that the hiring models exhibit different biases, supporting the claim that models can behave in biased ways. The paper studies hiring predictions, not pretrained models generating content, and the passages do not address harmful, untruthful, or low-quality generated content. (LLM, conf=0.84)
Experimental — accuracy not yet measuredWhile RLHF substantially enhances the performance of LLMs and aligns them with human preferences, it is not without limitations, including text hallucination, bias and toxicity, and generation of harmful text when probed.
“While RLHF substantially enhances the performance of LLMs and aligns them with human preferences, it is not without its limitations.These include, but are not limited to, issues such as text hallucination [115], bias and toxicity [37,48,58], and the generation of harmful text when probed [142,189].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [142]
The Results (p. 5) report bias in NLP classifiers and human–AI hiring decisions, supporting the bias portion at a different level of specificity. The paper does not address toxicity or establish these as limitations of RLHF-aligned LLMs; searches for RLHF and toxicity found no relevant passage. (LLM, conf=0.83)
Experimental — accuracy not yet measuredSFT can lead to harmful and unintended behavior, making it easier to prompt such models to elicit them.
“This can lead to harmful and unintended behavior, making it easier to prompt such models to elicit them [50,142].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [142]
The Experimental Setup (p. 3) and Results (p. 5) concern model recommendations and bias in human-AI hiring decisions. Searches for SFT, supervised fine-tuning, prompting, and elicited harmful behavior found no passage addressing the attributed claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredCreating a harmless and helpful chatbot necessitates a diverse group of labelers with varying backgrounds and demographics, as otherwise this may result in implicit biases in the model.
“For instance, creating a harmless and helpful chatbot necessitates a diverse group of labelers with varying backgrounds and demographics [12,136] as otherwise this may result in implicit biases in the model [141].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [141]
The Introduction (p. 1) and Experimental Setup (pp. 3–4) examine bias in AI-assisted hiring decisions, not the creation of chatbots. Searches for chatbot labelers and demographic diversity found no passage linking labeler diversity to implicit model bias. The copy therefore does not address the claim attributed to this source. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 30
- Bansal, G.; Nushi, B.; Kamar, E.; Lasecki, W.; Weld, D.; and Horvitz, E. 2019a. Beyond accuracy: the role of mental models in human-AI team performance. In Proceedings of the 2019 AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2019) . AAAI.
- Bansal, G.; Nushi, B.; Kamar, E.; Weld, D.; and Horvitz, E. 2019b. Updates in human-AI teams: understanding and addressing the performance-compatibility tradeoff. In Proceedings of the 2019 AAAI Conference on Artificial Intelligence (AAAI 2019) . AAAI.
- Bansal, G.; Wu, T.; Zhou, J.; Fok, R.; Nushi, B.; Kamar, E.; Ribeiro, M. T.; and Weld, D. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 Conference on Human Factors in Computing Systems (CHI 2021) , 1-16.
- Barocas, S.; and Selbst, A. 2016. Big data's disparate impact. California Law Review , 671.
- Bolukbasi, T.; Chang, K.-W.; Zou, J.; Saligrama, V.; and Kalai, A. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proceedings of the 2016 Conference on Neural Information Processing Systems (NIPS 2016) . NIPS.
- De-Arteaga, M.; Romanov, A.; Wallach, H.; Chayes, J.; Borgs, C.; Chouldechova, A.; Geyik, S.; Kenthapadi, K.; and Kalai, A. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency (FAT* 2019) . ACM.
- Dwork, C.; Hardt, M.; Pitassi, T.; Reingold, O.; and Zemel, R. 2012. Fairness through awareness. In Proceedings of the 2012 Innovations in Theoretical Computer Science Foundations Conference (ITCS 2012) . ACM.
- Dwork, C.; and Ilvento, C. 2018. Group fairness under composition. In Proceedings of the 2018 Conference on Fairness, Accountability, and Transparency (FAT* 2018) . ACM.
- Feng, S.; and Boyd-Graber, J. 2019. What Can AI Do for Me? Evaluating Machine Learning Interpretations in Cooperative Play. In Proceedings of the 24th International Conference on Intelligent User Interfaces , IUI '19, 229-239. New York, NY, USA: Association for Computing Machinery. ISBN 9781450362726.
- Gillies, M.; Fiebrink, R.; Tanaka, A.; Caramiaux, B.; Garcia, J.; Bevilacqua, F.; Heloir, A.; Nunnari, F.; Mackay, W.; and Amershi, S. 2016. Human-centered machine learning. In Proceedings of the 2016 Conference on Human Factors in Computing Systems (CHI 2016) . ACM.
- Gilpin, L.; Bau, D.; Yuan, B.; Bajwa, A.; Specter, M.; and Kagal, L. 2018. Explaining explanations: an overview of interpretability in machine learning. In Proceedings of the 2018 IEEE Conference on Data Science and Advanced Analytics (DSAA 2018) . IEEE.
- Gonen, H.; and Goldberg, Y. 2019. Lipstick on a pig: debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2019) . ACM.
- Green, B.; and Chen, Y. 2019. The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction , 3(CSCW): 1-24.
- Hardt, M.; Price, E.; and Srebro, N. 2016. Equality of opportunity in supervised learning. In Proceedings of the 2016 Conference on Neural Information Processing Systems (NeurIPS 2016) . NeurIPS.
- Isaac, C.; Lee, B.; and Carnes, M. 2009. Interventions that affect gender bias in hiring: a systematic review. U.S. National Library of Medicine , 84: 1440-1446.
- Kamar, E.; Hacker, S.; and Horvitz, E. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 2012 International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012) . IFAAMAS.
- Kleinberg, J.; Lakkaraju, H.; Leskovec, J.; Ludwig, J.; and Mullainathan, S. 2018. Human decisions and machine predictions. The Quarterly Journal of Economics , 133.
- Kleinberg, J.; Ludwig, J.; Mullainathan, S.; and Sunstein, C. R. 2019. Discrimination in the age of algorithms. SSRN .
- Lai, V.; and Tan, C. 2019. On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency , FAT* '19, 29-38. New York, NY, USA: Association for Computing Machinery. ISBN 9781450361255.
- Lichtenstein, S.; and Slovic, P. 2006. The Construction of Preference . Cambridge University Press.
- Mandl, M.; Felfernig, A.; Teppan, E.; and Schubert, M. 2011. Consumer decision making in knowledge-based recommendation. Journal of Intelligent Information Systems , 37: 1-22.
- Mehrabi, N.; Morstatter, F.; Saxena, N.; Lerman, K.; and Galstyan, A. 2021. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) , 54(6): 1-35.
- Peng, A.; Nushi, B.; Kiciman, E.; Inkpen, K.; Suri, S.; and Kamar, E. 2019. What you see is what you get? The Impact of Representation Criteria on Human Bias in Hiring. In Proceedings of the 2019 AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2019) . AAAI.
- Poursabzi-Sangdeh, F.; Goldstein, D.; Hofman, J.; Wortman Vaughn, J.; and Wallach, H. 2021. Manipulating and measuring model interpretability. In Proceedings of the 2021 Conference on Human Factors in Computing Systems (CHI 2021) .
- Raghavan, M.; Barocas, S.; Kleinberg, J.; and Levy, K. 2020. Mitigating bias in algorithmic hiring: Evaluating claims and practices. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* 2020) , 469481.
- Romanov, A.; De-Arteaga, M.; Wallach, H.; Chayes, J.; Borgs, C.; Chouldechova, A.; Geyik, S.; Kenthapadi, K.; Rumshisky, A.; and Kalai, A. 2019. What's in a name? Reducing bias in bios without access to protected attributes. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2019) . NAACL.
- Simon, H. 1955. A Behavioral Model of Choice. Quarterly Journal of Economics , 69: 99-118.
- Thaler, R.; and Sunstein, C. 2008. Nudge: improving decisions about health, wealth, and happiness . Penguin Books.
- Yin, M.; Wortman Vaughan, J.; and Wallach, H. 2019. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the 2019 Conference on Human Factors in Computing Systems (CHI 2019) . ACM.
- Zhang, Y.; Liao, V.; and Bellamy, R. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* 2020) , 295-305.
Cited by
Sorted by citations
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92
Metadata
- Authors
- Peng, A., Nushi, B., Kiciman, E., Inkpen, K., & Kamar, E.
- Affiliations
- Massachusetts Institute of Technology (US)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Volume
- 36
- Issue
- 11
- Pages
- 12089-12097
- ISSN
- 2159-5399 (ISSN-L)
- arXiv
- 2202.11812
- OpenAlex
- W4221152344
- Cited by
- 34
- References
- 0
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 8 · 2023: 8 · 2022: 4
- FWCI
- 7.70
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- human-AI teaming, hybrid decision-making, algorithmic bias, candidate selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
