The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
BBQ: A hand-built bias benchmark for question answering
Open accessAbstract
It is well documented that NLP models learn social biases, but little work has been done on how these biases manifest in model outputs for applied tasks like question answering (QA).We introduce the Bias Benchmark for QA (BBQ), a dataset of question sets constructed by the authors that highlight attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts.Our task evaluates model responses at two levels: (i) given an under-informative context, we test how strongly responses refect social biases, and (ii) given an adequately informative context, we test whether the model's biases override a correct answer choice.We fnd that models often rely on stereotypes when the context is under-informative, meaning the model's outputs consistently reproduce harmful biases in this setting.Though models are more accurate when the context provides an informative answer, they still rely on stereotypes and average up to 3.4 percentage points higher accuracy when the correct answer aligns with a social bias than when it conficts, with this difference widening to over 5 points on examples targeting gender for most models tested.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs may exhibit social biases and toxicity [37, 53, 153] during the generation process, resulting in the production of biased outputs.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [153]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evaluating the expanding capabilities of LLMs is a complex research area28,29, with group-specific bias benchmarks shown to be time-consuming to develop and utilize30–33, and the overall field lacking measurement validity and theoretical grounding30,34.”
From Generative language models exhibit social identity biases (2024) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Questions ask for a social group that meets either a negative or positive stereotype, with an ambiguous context that does not contain enough information to answer the question”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Parrish et al. 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Bias Benchmark for QA (BBQ) (Parrish et al. 2022) is a questionanswering dataset of 58,492 examples to assess bias across nine social groups: age, disability status, gender, nationality, physical appearance, race/ethnicity, religion, and socioeconomic status.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Parrish et al. 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Provider documentation indicates that SES does appear in some evaluation suites. For example, BBQ (Bias Benchmark for QA), a standardized test measuring social biases in question-answering, includes SES dimensions [72]. Google's Gemini evaluations separately incorporate Dollar Street, a dataset of household photographs across income levels, for socioeconomic contexts [34, 84].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [72]
The cited source says what the claim attributes to it.
In the citing paper
“Question-answering models may rely on stereotypes to answer questions in ambiguous contexts (Dhamala et al. 2021; Parrish et al. 2022).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Parrish et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“as well as a disambiguated context that specifies the correct answer.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Parrish et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“The dataset is designed to measure when a model relies on stereotypes.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Parrish et al. 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
He, P., Gao, J., & Chen, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09543 · Cited by 1,358
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 6×
“There are many, often conflicting, ways that bias is defined in NLP (Blodgett et al., 2020); we focus on stereotyping behavior and extend the definitions and treatment of bias in QA from Li et al. (2020), who have shown that the marginal probabilities a model associates with different answer…”
BackgroundInfluentialRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044
Gender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684
Updating Occupational Prestige and Socioeconomic Scores: How the New Measures Measure up
Nakao, K., & Treas, J. (1994) · Sociological Methodology · doi:10.2307/270978 · Cited by 552
The Woman Worked as a Babysitter: On Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1339 · Cited by 537 · cited 2×
“LMs have been found to reproduce social biases in downstream tasks, such as textual entailment (Dev et al., 2020; Sharma et al., 2021), language generation (Sheng et al., 2019), and coreference resolution (Rudinger et al., 2018).”
BackgroundUNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514
Social Bias Frames: Reasoning about Social and Power Implications of Language
Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., & Choi, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.486 · Cited by 469
Research works printed but not linked: 2
- Kate Crawford. 2017. The trouble with bias. http: //youtube.com/watch?v=fMym_BKWQzk. Talk given at NeurIPS December 2017.
- Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Ha- jishirzi. 2020. UnifedQA: Crossing format bound- aries with a single QA system. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1896-1907.
Cited by
Sorted by citations
Items 101–196 of 196
Instruction Makes a Difference
Adewumi, T., Habib, N., Alkhaled, L., & Barney, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70442-0_5 · Cited by 2
Bias Exposed: The BiaXposer Framework for NLP Fairness
Gaci, Y., Benatallah, B., Casati, F., & Benabdeslem, K. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0805-8_22 · Cited by 1
Detecting implicit biases of large language models with Bayesian hypothesis testing
Si, S., Jiang, X., Su, Q., & Carin, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-95825-x · Cited by 5
RVBench: Role values benchmark for role-playing LLMs
Wang, Y., Li, T., Li, M., Cheng, Z., Wang, G., Kang, H., Deng, Y., Xiao, H., & Zhang, Y. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100184 · Cited by 4
Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement
Mousavian, M., Abbasiantaeb, Z., Aliannejadi, M., & Crestani, F. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744620 · Cited by 1
Chand, S., Baca, F., & Ferrara, E. (2026) · AI · doi:10.3390/ai7010024 · Cited by 5
ReFaRAG: Re-ranking for Bias Mitigation in Retrieval-Augmented Generation
Zhao, Y., Efthymiou, V., Nummenmaa, J., & Stefanidis, K. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-05727-3_42 · Cited by 3
Rad, A.-C., & Lemnaru, C. (2025) · International Conference on Computational Photography · doi:10.1109/iccp68926.2025.11427185 · Cited by 1
Bombieri, M., Ponzetto, S. P., & Rospocher, M. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3806202 · Cited by 1
When to Invoke: Refining LLM Fairness with Toxicity Assessment
Ren, J., Li, B., Xu, Z., Luo, R., Yu, S. S., Ye, X., Fayek, H. M., Li, X., & Xia, F. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3793061 · Cited by 1
Operationalizing Responsible AI Policies with LLMs: an End-to-End Monitoring Prototype
Zieliński, P., Kelly, M., Barreto, R. D. C., & Tahaei, M. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799675 · Cited by 2
Gómez, R., Miranda, C. E., Romero-González, J.-A., Esparza, D. M. C., Alfonso-Francia, G., Chávez‐Urbiola, E. A., Ramírez-Pedraza, A., & Terven, J. R. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8060141 · Cited by 2
An, H., & Rudinger, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.34 · Cited by 7
GECOBench: a gender-controlled text dataset and benchmark for quantifying biases in explanations
Wilming, R., Dox, A., Schulz, H., Oliveira, M., Clark, B., & Haufe, S. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1694388 · Cited by 6
BRoverbs - Measuring how much LLMs understand Portuguese proverbs
Almeida, T. S., Bonás, G. K., & Santos, J. G. A. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5797 · Cited by 3
On Degrees of Freedom in Defining and Testing Natural Language Understanding
Sugawara, S., & Tsugita, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.861 · Cited by 1
Bondaronek, P., Ward, E., Beecham, E., Zhang, E. Z., Huang, Y., Ive, J., Naughton, F., Wu, H., & Vindrola‐Padros, C. (2026) · medRxiv · doi:10.64898/2026.05.20.26353755 · Cited by 1
Smaller Large Language Models Can Do Moral Self-Correction
Liu, G., Xue, Z., Zhang, X., Wang, R., & Johnson, K. (2025) · Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025) · doi:10.18653/v1/2025.trustnlp-main.5 · Cited by 1
All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs?
Kim, S., Janssen, M. A., Wang, J., Min-Venditti, A., Karanjia, N., & Anderies, J. M. (2026) · arXiv · doi:10.48550/arxiv.2604.17247
Systèmes de langage : d’où viennent leurs coups de génie ?
Ornes, S. (2023) · Pour la Science · doi:10.3917/pls.549.0046
A Study on Prompt Types for Harmlessness Assessment of Large-Scale Language Models
Shin, Y., Kim, S.-Y., & Byun, E. Y. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-62110-9_24
Evaluating Nuanced Bias in Large Language Model Free Response Answers
Healey, J., Byrum, L., Akhtar, M. N., & Sinha, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_36
Understanding Gender and Ethnicity Bias in Large Language Models
Gupta, O., Jaiswal, R. R., Marrone, S., Marassi, L., & Gargiulo, F. (2024) · Conference on Human Centred Artificial Intelligence - Education and Practice (HCAIep) · doi:10.1145/3701268.3701287
A Fairness Taxonomy for the Assessment of Korean Large Language Models
Shin, Y. (2024) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc62082.2024.10827134
Adversarially Exploring Vulnerabilities in LLMs to Evaluate Social Biases
Ong, Y. J., Gala, J. P., An, S., Moore, R. J., & Jadav, D. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825408 · Cited by 1
Atalan, Y., Jensen, B., Reynolds, I., Woo, A., García, P., Chen, T., & howarth, T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5152917
Expert-Guided Toxicity Filtration for Debiased Generation
Sun, X., Shi, K., Tang, H., Xu, G., & Li, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8183-9_10
Examining and Mitigating Ability-bias in LLMs via Self-Reflection
Iyer, N., Jha, A., & Pradhan, A. (2025) · International Web for All Conference (W4A) · doi:10.1145/3744257.3744268
Iyer, N., & Pradhan, A. (2025) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3715070.3749272 · Cited by 1
Advancing Safe Language Generation: Exploring Alternative Constrained RLHF
Meng, F., Bai, Z., Wang, Y., Huo, J., Wang, B., Yang, X., & Gao, Y. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11208943
Exploring Causal Effect of Social Bias on Faithfulness Hallucinations in Large Language Models
Zhang, Z., Zhang, J., Hu, X., Zhang, H., & Wan, X. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761298
Decoding the Mind of Large Language Models: A Quantitative Evaluation of Ideology and Biases
Hirose, M., & Uchida, M. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228711
Working Towards Toxic datasets for LLM Safeguarding
Guo, L., Wang, Z., Wang, Z., & He, T. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227669
BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context
Tomar, A., Sahoo, N. R., & Bhattacharyya, P. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.55
Dataset-centric AI ethics classification
Kartik, A., Raj, S., Rattan, A., & Sahu, D. (2025) · AI and Ethics · doi:10.1007/s43681-025-00904-4
A red teaming framework for large language models: a case study on faithfulness evaluation
Alotaibi, A., Mughus, R., & Ahmed, M. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09779-y · Cited by 1
Exacerbating Differences in Polarity: Bias Adversarial Attack on Generative Large Language Models
Gu, T., Qing, G., Hao, F., & Chang, L. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3645995
Concept Control for LLM Safety Using Radial Basis Function Representations
Amos, M., Song, Y., & Pagnucco, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4969-6_17
EVENS: Equality versus Equity Notion Spectrum of LLMs
Chen, Q., Cheng, R., Liu, Y., Xie, Z., Zhao, K., Li, P., Rotolo, A., Liu, Y., & Shen, W. (2025) · International Conference on Artificial Intelligence and Law · doi:10.1145/3769126.3769262
ZhePing, Y., Yang, H., Jin, R., Zhang, T., Ren, Y., & Zhang, G. Y. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356674
Detecting and Evaluating Bias in Large Language Models: Concepts, Methods, and Challenges
Gao, Z., Tong, L., & Zhang, Z. J. (2026) · Journal of Behavioral Data Science · doi:10.35566/jbds/gao · Cited by 1
PRISM: Principled Reasoning for Identifying and Suppressing Model Biases at Scale
Zhu, X., Li, W., & Lin, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7081-2_3
Design and Application of an Evaluation System for Large Language Models
Shen, Y., Cao, Z., & Chen, M. (2025) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai67447.2025.11412740
Ibitoye, A., Johnson, D. R., Sorinolu, B. G., Orji, R., & Abikoye, O. C. (2026) · AI and Ethics · doi:10.1007/s43681-026-01072-9
Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Study
Bouchouchi, N., Lesot, M., Laugel, T., Renard, X., Marsala, C., & Detyniecki, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37654-1_21 · Cited by 1
S-Eval: Towards Automated Safety Evaluation with Enhancement for Large Language Models
Yuan, X., Li, J., Wang, D., Chen, Y., Mao, X., Huang, L., Chen, J., Xue, H., Liu, X., Wang, W., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3805706
ViBBQ: Probing Social Biases in Vietnamese LLMs via Translated and Culturally Extended Benchmarks
Huynh, M.-P., Tu, P.-H., Trang, H. T. N., & Le, A.-C. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-21625-0_18
On Subjectivity, Bias and Fairness in Language Model Learning
Gaci, Y. (2023) · theses.fr (ABES) · doi:10.70675/dca82074za0c4z46d1z8a6bz96ce30bea3b6
Yu, M., Li, H., Singh, P., Li, X., Wang, D., & Hu, L. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792273 · Cited by 11
SCOPE: A Dataset of Stereotyped Prompts for Counterfactual Fairness Assessment of LLMs
Parziale, A., Voria, G., Pontillo, V., Lucia, A. D., Catolino, G., & Palomba, F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c67878.2026.00025
Calibrating Stereotype Preference with Multi-Agent Graph Reasoning in Large Language Models
Amin, M. N., & Jesser, A. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491743
Bias Ahead: Sensitive Prompts as Early Warnings for Fairness in Large Language Models
Voria, G., de Lucía, M. E., Raia, A., Lucia, A. D., Catolino, G., & Palomba, F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c67878.2026.00023
A Representation-Level Assessment of Bias Mitigation in Foundation Models
Nizhnichenkov, S., Nair, R., Daly, E. M., & Namee, B. M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-19096-3_13
AGWAN, S., Tambe, A., Shah, S., & Pole, G. S. (2026) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-026-05091-6
The More You Say, the More You Risk: Ethical Concerns in Large Language Model Reasoning Frameworks
Zhao, J., Cao, L., Zhang, X., Shi, K., Wang, Y., & Penn, G. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3795475
Pagliaro, A. (2026) · Electronics · doi:10.3390/electronics15112428
Counterfactual Consistency LoRA for Bias Mitigation on BBQ
Feng, D., Chen, H., & Zhou, Z. (2026) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit70033.2026.11557867
Fu, W., Lu, Z., Qu, G., & Guo, X. (2026) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3787109.3815216
Does It Lead to Prejudice or Discrimination? An Assessment of Stereotypes in LLM Decision-Making
Aoshima, T., & Akiyama, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812286
Automated bias blind spots: Examining stereotype associations in VLM-as-a-judge paradigms
Hong, R. S., Ovalle, A., Ung, M., Spiliopoulou, E., Sagun, L., Williams, A., & Ross, C. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806459 · Cited by 1
Hong, C.-H., Liu, Y.-Z., & Chen, C. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100350
Towards Trustworthy AI: A Comprehensive Framework for Evaluating Model Safety Across Dimensions
Hasan, M. M., Logofătu, D., & Ray, J. K. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-31141-2_25
Mitigating Bias in Large Language Model Based Question Answering through Causal Front Door Prompting
Yang, Y., Xu, Z., Li, J., Ma, C., Chan, J., Sanderson, M., Zheng, X., & Ren, Y. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809679 · Cited by 1
Understanding and Mitigating Competing Demographic Bias via Attention Disparity Analysis
Peng, T., Ci, R., Liu, Q., & Zhao, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3417-2_7
Data Governance for Generative AI
Armand, K. W. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0368-8.ch006
Wang, Y., & Steven, X. (2026) · International Conference on Artificial Intelligence and Digital Ethics (ICAIDE) · doi:10.1109/icaide69370.2026.11604541
Orthogonal Projection Steering for Bias Mitigation and ICAO Compliance in Diffusion Transformers
Mudgalgundurao, R., Khurana, A., Schuch, P., Ramachandra, R., & Raja, K. (2026) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw68408.2026.00105
Quantifying Social Biases in Language Model Classifiers is Domain-Dependent
Quiroga, T., Bravo-Márquez, F., & Barrière, V. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3834859
FairBotBench: A Bilingual Benchmark for Evaluating Chatbot Behavior in E-commerce
Janusko, T., Bochnia, R., Hempel, G., Tomschke, S., Anke, J., & Thiele, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26717-7_8
BALM: Bias-Aware Generation for Large Language Models
Dorleon, G., & Shujaa, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-34896-8_2
Deb, K., Mukherjee, I., & Sanyal, D. K. (2026) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-026-00651-7
Huang, Y., Jiang, W., & Ye, Y. (2026) · Quality and Reliability Engineering International · doi:10.1002/qre.70358
Nitsche, T., & Rubart, J. (2026) · Mensch und Computer (MuC) · doi:10.1145/3820253.3831386
StarDTox : Is Fairness in Language Models Just a Few Prompts Away?
Tahmasebi, S., Nikzad-Khasmakhi, N., Payberah, A. H., Asgari-Chenaghlu, M., & Matskin, M. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00438
Cicatiello, C., & Fusco, P. (2026) · Encyclopedia · doi:10.3390/encyclopedia6090182
Cross-Cultural Scenario Benchmark: Evaluating LLMs’ Cross-Cultural Understanding
Guo, M., Gao, L., Zeng, W. H., Rao, G., & Huang, C. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10090290
A review of bias detection and fairness auditing techniques in LLMs
Kaveti, N. K., & Pattanshetti, T. R. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02121-2
The Core Levers of LLM Evaluation: Sets, Templates, and Raters
Robsky, A., Lavitas, L., & Wang, Y. (2026) · Springer eBooks · doi:10.1007/978-3-032-26749-8_3
Measuring structural value alignment in sixteen models: LLMs have human-like moral spaces
Tang, Y. (2026) · AI and Ethics · doi:10.1007/s43681-026-01368-w
Bias2Risk: Dual-Task Evaluation of Bias-Induction Risk in Chinese Large Language Models
Wang, Q., Li, Z., Gong, M., Qi, Y., & He, Y. (2026) · International Conference on Image, Vision and Computing (ICIVC) · doi:10.1109/icivc70586.2026.11686557
A Survey of Evaluation Methods for the Security and Trustworthiness of Large Language Models
Zhao, L., Xu, Z., Zhao, R., Li, C., Yang, B., & Feng, D. (2026) · International Conference on Machine Learning Technologies (ICMLT) · doi:10.1109/icmlt69916.2026.11689272
FairCareer-CHN: A Dataset for Evaluating Bias in LLM-Based Career Recommendation
Li, X. (2026) · International Conference on Computational Intelligence and Machine Learning (CIML) · doi:10.1109/ciml69155.2026.11692774
Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines
Choi, J., Choi, Y., & Jang, B. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.806
LangBiTe: model-driven bias testing of text-to-text large language models
Morales, S., Clarisó, R., & Cabot, J. (2026) · Software & Systems Modeling · doi:10.1007/s10270-026-01425-2
CARP: Constraint-Aware Re-Prompting for Zero-Shot Bias Mitigation in Compact LLMs
Jagtap, P. A., & Ugale, T. (2026) · International Conference on Computing Communication Control and automation · doi:10.1109/iccubea69821.2026.11702098
Do LLMs Take Sides? Measuring Opinion Bias and Sycophancy via Multi-turn Probing
Nogueira, R. F., Bonás, G. K., Roque, A., Pires, R., Larcher, C. H. N., Abonizio, H., Piau, M., Malaquias, R. D., Almeida, T. S., & Laitz, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39892-5_21
Parziale, A., D'Auria, A., Scaparra, D. P., & Pontillo, V. (2026) · doi:10.1145/3843782.3844655
Gender attribution under workplace role-and-register prompts in contemporary large language models
Shukla, R. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01332-1
Fair-MAD:For FGKB-Guided Adjudication of Stereotype-Prone Questions
Dong, Z., Yu, Z., Zuo, J., & Zhu, E. (2026) · International Conference on Computer Science, Machine Learning and Agents (CSMLA) · doi:10.1109/csmla70906.2026.11724473
Cardoso, L. R., & Ribeiro, L. S. F. (2026) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2026.26812
FAIRITHM: Unlearning Bias and Rewriting for Fairness
Nam, Y., Lim, D., Im, S., Xiong, Z., & Park, J. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2744493
AUTOMATED EVALUATION OF ETHICAL ASPECTS IN LARGE LANGUAGE MODELS
Kolchenko, V., & Sabodashko, D. (2026) · Cybersecurity Education Science Technique · doi:10.28925/2663-4023.2026.34.1357
赵玉荣, Bi, Y., & Wang, S. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3829069
Serçe, A., Kaçıkan, E., & Onan, A. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11727894
Dolunay, A., & Arpalıklı, F. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1980883
Metadata
- Authors
- Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S.
- Affiliations
- New York University (US)
- Venue
- Findings of the Association for Computational Linguistics: ACL 2022
- Published
- 2022
- Pages
- 2086-2105
- OpenAlex
- W3207316473
- Cited by
- 186
- References
- 34
- Citations by year
- 2026: 67 · 2025: 62 · 2024: 29 · 2023: 23 · 2022: 5
- FWCI
- 29.56
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2046556, 1922658)
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- social bias, question answering, bias benchmark, stereotypes, protected classes, gender bias, model bias evaluation, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
