Gender and Dialect Bias in YouTube’s Automatic Captions
Open accessAbstract
This project evaluates the accuracy of YouTube's automatically-generated captions across two genders and five dialects of English.Speakers' dialect and gender was controlled for by using videos uploaded as part of the "accent tag challenge", where speakers explicitly identify their language background.The results show robust differences in accuracy across both gender and dialect, with lower accuracy for 1) women and 2) speakers from Scotland.This finding builds on earlier research finding that speaker's sociolinguistic identity may negatively impact their ability to use automatic speech recognition, and demonstrates the need for sociolinguistically-stratified validation of systems.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Models can perform well on average but poorly on some users.
“These models can perform well on average but poorly on some users (Tatman, 2017; Caldas et al., 2018; Li et al., 2019b; Koenecke et al., 2020).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tatman, 2017
Performance disparities across individuals have been observed in a wide range of tasks and applications, including natural language processing, automatic speech recognition, federated learning, and medical imaging.
“Performance disparities across individuals have been observed in a wide range of tasks and applications, including in natural language processing (Geva et al., 2019), automatic speech recognition (Koenecke et al., 2020; Tatman, 2017), federated learning (Li et al., 2019b; Caldas et al., 2018), and medical imaging (Badgeley et al., 2019).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tatman, 2017
Models can perform well on average but poorly on some individuals.
“These models can perform well on average but poorly on some individuals (Tatman, 2017; Caldas et al., 2018; Li et al., 2019b; Koenecke et al., 2020).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tatman, 2017
Cites
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Praat, a system for doing phonetics by computer
Boersma, P. (2002) · UvA-DARE (University of Amsterdam) · Cited by 4,417 · see citing works
TIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,509 · see citing works
Torralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · see citing works
SWITCHBOARD: telephone speech corpus for research and development
Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225858 · Cited by 2,134 · cited 1× · see citing works
“Switchboard (Godfrey et al., 1992) undersamples women, Southerns and non-college-educated speakers.”
BackgroundSex, covert prestige and linguistic change in the urban British English of Norwich
TRUDGILL, P. W. (1972) · Language in Society · doi:10.1017/s0047404500000488 · Cited by 1,515 · see citing works
The whole woman: Sex and gender differences in variation
Eckert, P. (1989) · Language Variation and Change · doi:10.1017/s095439450000017x · Cited by 1,021 · see citing works
Rapid adaptation to foreign-accented English
Clarke, C. M., & Garrett, M. F. (2004) · The Journal of the Acoustical Society of America · doi:10.1121/1.1815131 · Cited by 633 · see citing works
The effect of experience on the perception and representation of dialect variants
Sumner, M., & Samuel, A. G. (2009) · Journal of Memory and Language · doi:10.1016/j.jml.2009.01.001 · Cited by 364 · see citing works
The social impact of natural language processing
Hovy, D., & Spruit, S. L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2096 · Cited by 329 · see citing works
The microsoft 2016 conversational speech recognition system
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M., Stolcke, A., Yu, D., & Zweig, G. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953159 · Cited by 326 · see citing works
Real English: The Grammar Of English Dialects In The British Isles
Milroy, J., & Milroy, L. (2014) · Cited by 296 · see citing works
Demographic dialectal variation in social media: a case study of African-American English
Blodgett, S. L., Green, L., & O'Connor, B. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1120 · Cited by 291 · see citing works
Johnson, E. (1998) · Language · doi:10.2307/417880 · Cited by 248 · see citing works
Liao, H., McDermott, E., & Senior, A. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707758 · Cited by 194 · see citing works
Gelfer, M. P., & Mikos, V. A. (2005) · Journal of Voice · doi:10.1016/j.jvoice.2004.10.006 · Cited by 190 · see citing works
The Dictionary of American Regional English
Shortridge, J. R. (1986) · Geographical Review · doi:10.2307/214786 · Cited by 189 · see citing works
Large Margin Hidden Markov Models for Automatic Speech Recognition
Sha, F., & Saul, L. K. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0161 · Cited by 177 · see citing works
Goldwater, S., Jurafsky, D., & Manning, C. D. (2010) · Speech Communication · doi:10.1016/j.specom.2009.10.001 · Cited by 166 · see citing works
Gender recognition from speech. Part I: Coarse analysis
Wu, K., & Childers, D. G. (1991) · The Journal of the Acoustical Society of America · doi:10.1121/1.401663 · Cited by 156 · see citing works
Hay, J. B., Maclagan, M., & Gordon, E. (2008) · Edinburgh University Press eBooks · doi:10.1515/9780748630882 · Cited by 134 · see citing works
New telephone speech corpora at CSLU
Cole, R. A., Noel, M., Lander, T., & Durham, T. (1995) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1995-188 · Cited by 129 · see citing works
Pépiot, E. (2014) · Speech prosody · doi:10.21437/speechprosody.2014-49 · Cited by 101 · see citing works
THE POPULATION OF THE UNITED STATES
(1925) · Science · doi:10.1126/science.61.1582.xii-t · Cited by 95 · see citing works
Prosodic and other cues to speech recognition failures
Hirschberg, J., Litman, D. J., & Swerts, M. G. J. (2004) · Speech Communication · doi:10.1016/j.specom.2004.01.006 · Cited by 89 · see citing works
The Social Impact Of Natural Language Processing
Hovy, D., & Spruit, S. L. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1000924 · Cited by 80 · see citing works
GDNN: a gender-dependent neural network for continuous speech recognition
Konig, Y., & Morgan, N. (2003) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.226966 · Cited by 33 · see citing works
Hiring by Algorithm: Predicting and Preventing Disparate ImpactPreprint
Ajunwa, I., Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2016) · SSRN Electronic Journal · Cited by 33 · see citing works
Wheatley, B., & Picone, J. (1991) · Digital Signal Processing · doi:10.1016/1051-2004(91)90095-3 · Cited by 27 · see citing works
Speaker-independent ASR for Modern Standard Arabic: effect of regional accents
Droua-Hamdani, G., Selouani, S., & Boudraa, M. (2012) · International Journal of Speech Technology · doi:10.1007/s10772-012-9146-4 · Cited by 18 · see citing works
The Population of the United States.
McNicoll, G., Anderton, D. L., Barrett, R., & Bogue, D. J. (1997) · Population and Development Review · doi:10.2307/2137588 · Cited by 15 · see citing works
Sawalha, M., & Shariah, M. A. (2013) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 10 · see citing works
Geo-location dependent deep neural network acoustic model for speech recognition
Ye, G., Liu, C., & Gong, Y. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472803 · Cited by 6 · see citing works
Assessing the efficacy of benchmarks for automatic speech accent recognition
Bock, B. R., & Shamir, L. (2015) · EAI International Conference on Mobile Multimedia Communications (MobiMedia) · doi:10.4108/icst.mobimedia.2015.259033 · Cited by 6 · see citing works
The Population of the United States
P., P., & Bogue, D. J. (1962) · Population · doi:10.2307/1525771 · Cited by 4 · see citing works
ATAROS Technical Report 1: Corpus collection and initial task validation
Freeman, V., Chan, J., Levow, G., WRIGHT, R. A., Mari Ostendorf, E., Zayats, V., Luan, Y., Morrison, H., Fox, L., Antoniak, M., et al. (2014) · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993 · see citing works
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · see citing works
Racial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765 · see citing works
Measuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · see citing works
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
Mitigating Gender Bias in Natural Language Processing: Literature Review
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1159 · Cited by 492 · see citing works
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · see citing works
Racial Bias in Hate Speech and Abusive Language Detection Datasets
Davidson, T., Bhattacharya, D., & Weber, I. (2019) · Proceedings of the Third Workshop on Abusive Language Online · doi:10.18653/v1/w19-3504 · Cited by 414 · see citing works
Gender bias and stereotypes in Large Language Models
Kotek, H., Dockum, R., & Sun, D. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615599 · Cited by 381 · see citing works
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · see citing works
Five sources of bias in natural language processing
Hovy, D., & Prabhumoye, S. (2021) · Language and Linguistics Compass · doi:10.1111/lnc3.12432 · Cited by 298 · see citing works
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · see citing works
“These models can perform well on average but poorly on some users (Tatman, 2017; Caldas et al., 2018; Li et al., 2019b; Koenecke et al., 2020).”
InfluentialGender Bias in Neural Natural Language Processing
Lu, K., Mardziel, P., Wu, F., Amancharla, P., & Datta, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62077-6_14 · Cited by 284 · see citing works
Zhai, C., & Wibowo, S. (2023) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2023.100134 · Cited by 363 · see citing works
Towards generalisable hate speech detection: a review on obstacles and solutions
Yin, W., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.598 · Cited by 220 · see citing works
Learning models with uniform performance via distributionally robust optimization
Duchi, J. C., & Namkoong, H. (2021) · The Annals of Statistics · doi:10.1214/20-aos2004 · Cited by 228 · see citing works
Gender Bias in Machine Translation
Savoldi, B., Gaido, M., Bentivogli, L., Negri, M., & Turchi, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00401 · Cited by 189 · see citing works
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01589 · Cited by 163 · see citing works
The digital scribe in clinical practice: a scoping review and research agenda
van Buchem, M. M., Boosman, H., Bauer, M. P., Kant, I. M. J., Cammel, S. A., & Steyerberg, E. W. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00432-5 · Cited by 152 · see citing works
Fairway: a way to build fair ML software
Chakraborty, J., Majumder, S., Yu, Z., & Menzies, T. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409697 · Cited by 144 · see citing works
Brun, Y., & Meliou, A. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3264838 · Cited by 138 · see citing works
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · see citing works
Toward fairness in AI for people with disabilities SBG@a research roadmap
Guo, A., Kamar, E., Vaughan, J. W., Wallach, H., & Morris, M. R. (2020) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3386296.3386298 · Cited by 128 · see citing works
Gradient Matching for Domain Generalization
Shi, Y., Seely, J., Torr, P. H. S., Siddharth, N., Hannun, A., Usunier, N., & Synnaeve, G. (2021) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.2104.09937 · Cited by 159 · see citing works
Fairness and Bias in Algorithmic Hiring: A Multidisciplinary Survey
Fabris, A., Baranowska, N., Dennis, M. J., Graus, D., Hacker, P., Saldivar, J., Zuiderveen Borgesius, F., & Biega, A. J. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3696457 · Cited by 117 · see citing works
Springer, A., & Whittaker, S. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302322 · Cited by 115 · see citing works
“Major public concerns have arisen following demonstrations of bias in algorithmic systems with regards to gender, race, and other characteristics [9,75,79].”
BackgroundInfluentialAlgorithmic fairness datasets: the story so far
Fabris, A., Messina, S., Silvello, G., & Susto, G. A. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00854-z · Cited by 128 · see citing works
Mengesha, Z., Heldreth, C., Lahav, M., Sublewski, J., & Tuennerman, E. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.725911 · Cited by 140 · see citing works
Gupta, S., Langhans, S. D., Domisch, S., Nerini, F. F., Felländer, A., Battaglini, M., Tegmark, M., & Vinuesa, R. (2021) · Transportation Engineering · doi:10.1016/j.treng.2021.100064 · Cited by 110 · see citing works
HopeEDI: A Multilingual Hope Speech Detection Dataset for Equality, Diversity, and Inclusion
Chakravarthi, B. R. (2020) · Cited by 107 · see citing works
Towards inclusive automatic speech recognition
Feng, S., Halpern, B. M., Kudina, O., & Scharenborg, O. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101567 · Cited by 106 · see citing works
Systematic Inequalities in Language Technology Performance across the World’s Languages
Blasí, D. E., Anastasopoulos, A., & Neubig, G. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.376 · Cited by 140 · see citing works
It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations
Tan, S., Joty, S., Kan, M., & Socher, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.263 · Cited by 101 · see citing works
Blodgett, S. L., & O'Connor, B. (2017) · arXiv · doi:10.48550/arxiv.1707.00061 · Cited by 153 · see citing works
Stereotypical Bias Removal for Hate Speech Detection Task using Knowledge-based Generalizations
Badjatiya, P., Gupta, M., & Varma, V. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313504 · Cited by 97 · see citing works
Themis: automatically testing software for discrimination
Angell, R., Johnson, B., Brun, Y., & Meliou, A. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3264590 · Cited by 94 · see citing works
Evaluating OpenAI's Whisper ASR: Performance analysis across diverse accents and speaker traits
Graham, C., & Roll, N. (2024) · JASA Express Letters · doi:10.1121/10.0024876 · Cited by 90 · see citing works
Effects of Talker Dialect, Gender & Race on Accuracy of Bing Speech and YouTube Automatic Captions
Tatman, R., & Kasten, C. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1746 · Cited by 89 · see citing works
Self-citationMulti-Dimensional Gender Bias Classification
Dinan, E., Fan, A., Wu, L., Weston, J., Kiela, D., & Williams, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.23 · Cited by 94 · see citing works
Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information?
Springer, A., & Whittaker, S. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3374218 · Cited by 84 · see citing works
Handling Bias in Toxic Speech Detection: A Survey
Garg, T., Masud, S., Suresh, T., & Chakraborty, T. (2023) · ACM Computing Surveys · doi:10.1145/3580494 · Cited by 86 · see citing works
Mirages. On Anthropomorphism in Dialogue Systems
Abercrombie, G., Curry, A., Dinkar, T., Rieser, V., & Talat, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.290 · Cited by 83 · see citing works
Assessing the accuracy of automatic speech recognition for psychotherapy
Miner, A. S., Haque, A., Fries, J., Fleming, S. L., Wilfley, D. E., Wilson, G. T., Milstein, A. S., Jurafsky, D., Arnow, B. A., Agras, W. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0285-8 · Cited by 82 · see citing works
Markl, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533117 · Cited by 86 · see citing works
Uneven success: automatic speech recognition and ethnicity-related dialects
Wassink, A. B., Gansen, C., & Bartholomew, I. (2022) · Speech Communication · doi:10.1016/j.specom.2022.03.009 · Cited by 77 · see citing works
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M. R., Vaughan, J. W., Wadsworth, W. D., & Wallach, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462610 · Cited by 72 · see citing works
Artificial intelligence in the work context
Jarrahi, M. H., Lutz, C., Boyd, K., Oesterlund, C., & Willis, M. (2022) · J. Assoc. Inf. Sci. Technol. · doi:10.1002/asi.24730 · Cited by 42 · see citing works
Boyd, K., & Andalibi, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579528 · Cited by 69 · see citing works
Quantifying Bias in Automatic Speech RecognitionPreprint
Feng, S., Kudina, O., Halpern, B. M., & Scharenborg, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15122 · Cited by 61 · see citing works
A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings
Sweeney, C., & Najafian, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1162 · Cited by 58 · see citing works
Dataset for Identification of Homophobia and Transophobia in Multilingual YouTube CommentsPreprint
Chakravarthi, B. R., Priyadharshini, R., Ponnusamy, R., Kumaresan, P. K., Sampath, K., Thenmozhi, D., Thangasamy, S., Nallathambi, R., & McCrae, J. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.00227 · Cited by 82 · see citing works
An annotated dataset of literary entities
Bamman, D., Popat, S., & Shen, S. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1220 · Cited by 56 · see citing works
Returning the N to NLP: Towards Contextually Personalized Classification Models
Flek, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.700 · Cited by 57 · see citing works
The Dawn of the Human-Machine Era: A forecast of new and emerging language technologies
Sayers, D., Sousa‐Silva, R., Höhn, S., Ahmedi, L., Allkivi-Metsoja, K., Anastasiou, D., Beňuš, Š., Bowker, L., Bytyçi, E., Catalá, A., et al. (2021) · doi:10.17011/jyx/reports/20210518/1 · Cited by 59 · see citing works
Harris, C., Johnson, A. G., Palmer, S., Yang, D.-Y., & Bruckman, A. (2023) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3610169 · Cited by 54 · see citing works
Designing an Online Infrastructure for Collecting AI Data From People With Disabilities
Park, J. S., Bragg, D., Kamar, E., & Morris, M. R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445870 · Cited by 50 · see citing works
Eschewing Gender Stereotypes in Voice Assistants to Promote Inclusion
Danielescu, A. (2020) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3405755.3406151 · Cited by 49 · see citing works
Cao, Y. T., & Daumé, H. (2021) · Computational Linguistics · doi:10.1162/coli_a_00413 · Cited by 47 · see citing works
Rio, D. D. F. D., Sovacool, B. K., & Martiskainen, M. (2021) · Energy Research & Social Science · doi:10.1016/j.erss.2021.102105 · Cited by 47 · see citing works
Artificial intelligence and labour market matching
Broecke, S. (2023) · OECD social employment and migration working papers · doi:10.1787/2b440821-en · Cited by 64 · see citing works
Waelen, R. A., & Wieczorek, M. (2022) · Philosophy & Technology · doi:10.1007/s13347-022-00548-w · Cited by 43 · see citing works
Hope speech detection in YouTube comments
Chakravarthi, B. R. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00901-z · Cited by 42 · see citing works
Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance
Garnerin, M., Rossato, S., & Besacier, L. (2019) · International Workshop on AI for Smart TV Content Production, Access and Delivery (AI4TV) · doi:10.1145/3347449.3357480 · Cited by 42 · see citing works
Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions
Liu, C., Picheny, M., Sarı, L., Chitkara, P., Xiao, A., Zhang, X., Chou, M., Alvarado, A., Hazırbaş, C., & Saraf, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747501 · Cited by 40 · see citing works
Data Representativeness in Accessibility Datasets: A Meta-Analysis
Kamikubo, R., Wang, L., Marte, C., Mahmood, A., & Kacorri, H. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544826 · Cited by 41 · see citing works
Hey ASR System! Why Aren’t You More Inclusive?
Ngueajio, M. K., & Washington, G. J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21707-4_30 · Cited by 43 · see citing works
Zolnoori, M., Vergez, S., Xu, Z., Esmaeili, E., Zolnour, A., Briggs, K. A., Scroggins, J. K., Ebrahimabad, S. F. H., Noble, J. M., Topaz, M., et al. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae130 · Cited by 38 · see citing works
Reducing sentiment polarity for demographic attributes in word embeddings using adversarial learning
Sweeney, C., & Najafian, M. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372837 · Cited by 38 · see citing works
AI Consent Futures: A Case Study on Voice Data Collection with Clinicians
Wilcox, L. G., Brewer, R. N., & Díaz, F. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610107 · Cited by 37 · see citing works
Understanding the Representation and Representativeness of Age in AI Data Sets
Park, J., Bernstein, M. S., Brewer, R. N., Kamar, E., & Morris, M. (2021) · AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462590 · Cited by 51 · see citing works
Fair Enough: Searching for Sufficient Measures of Fairness
Majumder, S., Chakraborty, J., Bai, G. R., Stolee, K. T., & Menzies, T. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3585006 · Cited by 36 · see citing works
Rismani, S., Shelby, R. M., Smart, A., Jatho, E. W., Kroll, J. A., Moon, A., & Rostamzadeh, N. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581407 · Cited by 38 · see citing works
Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding
Tan, S., Joty, S., Varshney, L. R., & Kan, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.455 · Cited by 35 · see citing works
Personal Bias in Prediction of Emotions Elicited by Textual Opinions
Miłkowski, P., Gruza, M., Kanclerz, K., Kazienko, P., Grimling, D., & Kocoń, J. (2021) · doi:10.18653/v1/2021.acl-srw.26 · Cited by 34 · see citing works
Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning
Yin, D., Li, L. H., Hu, Z.-N., Peng, N., & Chang, K.-W. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.162 · Cited by 78 · see citing works
Mitigating Gender Bias in Natural Language Processing: Literature ReviewPreprint
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08976 · Cited by 109 · see citing works
Search-based fairness testing for regression-based machine learning systems
Perera, A., Aleti, A., Tantithamthavorn, C. K., Jiarpakdee, J., Turhan, B., Kühn, L., & Walker, K. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10116-7 · Cited by 33 · see citing works
Nee, J., Smith, G. M., Sheares, A. M., & Rustagi, I. (2022) · Big Data & Society · doi:10.1177/20539517221090930 · Cited by 38 · see citing works
Intrinsic bias metrics do not correlate with application bias
Goldfarb-Tarrant, S., Marchant, R., Muñoz Sánchez, R., Pandya, M., & Lopez, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.150 · Cited by 33 · see citing works
Can Whisper Perform Speech-Based In-Context Learning?
Wang, S., Yang, C.-H. H., Wu, J. Y., & Zhang, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446502 · Cited by 32 · see citing works
Affordances and Constraints of Intelligent Personal Assistants for Second-Language Learning
Dizon, G. (2021) · RELC Journal · doi:10.1177/00336882211020548 · Cited by 32 · see citing works
Automatic Fairness Testing of Neural Classifiers Through Adversarial Sampling
Zhang, P., Wang, J., Sun, J., Wang, X., Dong, G., Wang, X., Dai, T., & Dong, J. S. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3101478 · Cited by 32 · see citing works
NLPositionality: Characterizing Design Biases of Datasets and Models
Santy, S., Liang, J. T., Bras, R. J. L., Reinecke, K., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.505 · Cited by 45 · see citing works
An Automated Toxicity Classification on Social Media Using LSTM and Word Embedding
Alsharef, A., Aggarwal, K., Sonia, S., Koundal, D., Alyami, H., & Ameyed, D. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/8467349 · Cited by 31 · see citing works
RetractedCase Study: Deontological Ethics in NLP
Prabhumoye, S., Boldt, B., Salakhutdinov, R., & Black, A. W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.297 · Cited by 32 · see citing works
Intersectional Bias in Hate Speech and Abusive Language DatasetsPreprint
Kim, J., Ortiz, C., Nam, S., Santiago, S., & Datta, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05921 · Cited by 37 · see citing works
"Play PRBLMS": Identifying and Correcting Less Accessible Content in Voice Interfaces
Springer, A., & Cramer, H. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173870 · Cited by 27 · see citing works
Augmented Datasheets for Speech Datasets and Ethical Decision-Making
Papakyriakopoulos, O., Choi, A. S. G., Thong, W. E., Zhao, D., Andrews, J. T. A., Bourke, R., Xiang, A., & Koenecke, A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594049 · Cited by 30 · see citing works
Assessing and Addressing Algorithmic Bias - But Before We Get TherePreprint
Garcia-Gathright, J. I., Springer, A., & Cramer, H. S. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03332 · Cited by 27 · see citing works
Responsible natural language processing: A principlist framework for social benefits
Behera, R. K., Bala, P. K., Rana, N. P., & Irani, Z. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122306 · Cited by 26 · see citing works
How Might We Create Better Benchmarks for Speech Recognition?
Aksënova, A., van Esch, D., Flynn, J., & Golik, P. (2021) · Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future · doi:10.18653/v1/2021.bppf-1.4 · Cited by 25 · see citing works
Multilingual hope speech detection in English and Dravidian languages
Chakravarthi, B. R. (2022) · International Journal of Data Science and Analytics · doi:10.1007/s41060-022-00341-0 · Cited by 25 · see citing works
Dealing with Gender Bias Issues in Data-Algorithmic Processes: A Social-Statistical Perspective
Castaneda, J., Jover-Leal, A., Calvet, L., Torrado, S. Y., Juan, A. A., & Sáinz, M. (2022) · Algorithms · doi:10.3390/a15090303 · Cited by 25 · see citing works
D’Arcy, A., & Bender, E. M. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-015324 · Cited by 28 · see citing works
Hollands, S., Blackburn, D. J., & Christensen, H. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10433 · Cited by 24 · see citing works
Reed, C., Wynn, M. G., & Bown, R. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9020040 · Cited by 39 · see citing works
ALGORITHMIC BIAS IN AI-ENHANCED EDUCATION: CULTURAL DIMENSIONS AND PEDAGOGICAL IMPACT
Fadil, H. O. C. A., & Nuredin, A. (2025) · Symposium Proceedings · doi:10.55843/isl2025symp163h · Cited by 25 · see citing works
Fine-tuning neural machine translation on gender-balanced datasets
Costa‐jussà, M. R., & Sánchez, A. D. J. (2020) · Cited by 24 · see citing works
Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
Weerts, H., Pfisterer, F., Feurer, M., Eggensperger, K., Bergman, E., Awad, N. H., Vanschoren, J., Pechenizkiy, M., Bischl, B., & Hutter, F. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14747 · Cited by 27 · see citing works
- Authors
- Tatman, R.
- Affiliations
- University of Washington (US)
- Corresponding author
- Tatman, Rachael
- Venue
- Proceedings of the First ACL Workshop on Ethics in Natural Language Processing
- Published
- 2017
- Pages
- 53-59
- OpenAlex
- W2607719644
- Cited by
- 338
- References
- 36
- Citations by year
- 2026: 34 · 2025: 33 · 2024: 35 · 2023: 50 · 2022: 56 · 2021: 62 · 2020: 33 · 2019: 21 · 2018: 11 · 2017: 3
- FWCI
- 8.48
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Funders
- National Science Foundation (1256082)
- University of Washington
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Social Sciences › Linguistics and Language › Linguistic Variation and Morphology
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- speech recognition, speech recognition accuracy, gender bias, Scottish English, accent, sociolinguistic identity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review