Measuring and Mitigating Gender Bias in Legal Contextualized Language Models
Open accessThe DOI matches its Crossref record; 97 of the 98 references it cites were found in a registry or an index.
Abstract
Transformer-based contextualized language models constitute the state-of-the-art in several natural language processing (NLP) tasks and applications. Despite their utility, contextualized models can contain human-like social biases, as their training corpora generally consist of human-generated text. Evaluating and removing social biases in NLP models has been a major research endeavor. In parallel, NLP approaches in the legal domain, namely, legal NLP or computational law, have also been increasing. Eliminating unwanted bias in legal NLP is crucial, since the law has the utmost importance and effect on people. In this work, we focus on the gender bias encoded in BERT-based models. We propose a new template-based bias measurement method with a new bias evaluation corpus using crime words from the FBI database. This method quantifies the gender bias present in BERT-based models for legal applications. Furthermore, we propose a new fine-tuning-based debiasing method using the European Court of Human Rights (ECtHR) corpus to debias legal pre-trained models. We test the debiased models’ language understanding performance on the LexGLUE benchmark to confirm that the underlying semantic vector space is not perturbed during the debiasing process. Finally, we propose a bias penalty for the performance scores to emphasize the effect of gender bias on model performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Notably, there is growing attention on assessing LLM bias [63-64] and robustness [323-324].”
From A Survey of Large Language Models (2026) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Measuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION
He, P., Liu, X., Gao, J., & Chen, W. (2021) · International Conference on Learning Representations · Cited by 2,445
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Proceedings of the 24th annual meeting on Association for Computational Linguistics -
(1986) · doi:10.3115/981131 · Cited by 1,081
LEGAL-BERT: The Muppets straight out of Law School
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., & Androutsopoulos, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.261 · Cited by 994
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829
Aletras, N., Tsarapatsanis, D., Preoţiuc-Pietro, D., & Lampos, V. (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.93 · Cited by 810
AI can be sexist and racist — it’s time to make it fair
Zou, J., & Schiebinger, L. (2018) · Nature · doi:10.1038/d41586-018-05707-8 · Cited by 765
A general approach for predicting the behavior of the Supreme Court of the United States
Katz, D. M., Bommarito, M. J., & Blackman, J. (2017) · PLOS ONE · doi:10.1371/journal.pone.0174698 · Cited by 528
On Measuring Social Biases in Sentence Encoders
May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1063 · Cited by 461
Gonen, H., & Goldberg, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1061 · Cited by 454 · cited 1×
“Following these works, the human-like biases in contextualized language models are analyzed in various other studies [11, 15, 37, 45, 46, 49, 76, 85, 90].”
BackgroundInfluentialDeBERTa: Decoding-enhanced BERT with Disentangled Attention
He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 420
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384
How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence
Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.466 · Cited by 367
Biases in Large Language Models: Origins, Inventory, and Discussion
Navigli, R., Conia, S., & Ross, B. (2023) · Journal of Data and Information Quality · doi:10.1145/3597307 · Cited by 364 · cited 1×
“Finally, a recent survey profoundly overviews and categorizes biases in large language models [61].”
BackgroundNeural Legal Judgment Prediction in English
Chalkidis, I., Androutsopoulos, I., & Aletras, N. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1424 · Cited by 352 · cited 3×
“In May, Fujisawa joined Mari Motohashi ’s rink as the team’s skip, moving back from Karuizawa to Kitami where she had spent her junior days.”
BackgroundMeasuring Bias in Contextualized Word Representations
Kurita, K., Vyas, N., Pareek, A., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3823 · Cited by 348 · cited 3×
“Along with the quantification of bias in contextualized language models, different approaches for debiasing these embeddings are also proposed [11, 15, 45, 46, 49, 56, 85].”
BackgroundMethodInfluentialEnhancing clinical concept extraction with contextual embeddings
Si, Y., Wang, J., Xu, H., & Roberts, K. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz096 · Cited by 322
Gender Bias in Contextualized Word Embeddings
Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., & Chang, K.-W. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1064 · Cited by 319
WELFake: Word Embedding Over Linguistic Features for Fake News Detection
Verma, P. K., Agrawal, P., Amorim, I., & Prodan, R. (2021) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2021.3068519 · Cited by 313
Evaluating Gender Bias in Machine Translation
Stanovsky, G., Smith, N. A., & Zettlemoyer, L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1164 · Cited by 292
Manzini, T., Yao Chong, L., Black, A. W., & Tsvetkov, Y. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1062 · Cited by 255
Mind the GAP: A balanced corpus of gendered ambiguous pronouns
Webster, K., Recasens, M., Axelrod, V., & Baldridge, J. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00240 · Cited by 223
Reasoning with cases and hypotheticals in HYPO
Ashley, K. D. (1991) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(91)90011-u · Cited by 213
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D., & Aletras, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.297 · Cited by 201
Large-Scale Multi-Label Text Classification on EU Legislation
Chalkidis, I., Fergadiotis, E., Malakasiotis, P., & Androutsopoulos, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1636 · Cited by 191
Deep learning in law: early adaptation and legal word embeddings trained on large corpora
Chalkidis, I., & Kampas, D. (2019) · Artificial Intelligence and Law · doi:10.1007/s10506-018-9238-9 · Cited by 189
CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service
Lippi, M., Pałka, P., Contissa, G., Lagioia, F., Micklitz, H.-W., Sartor, G., & Torroni, P. (2019) · Artificial Intelligence and Law · doi:10.1007/s10506-019-09243-2 · Cited by 169
A history of AI and Law in 50 papers: 25 years of the international conference on AI and Law
Bench-Capon, T., Araszkiewicz, M., Ashley, K., Atkinson, K., Bex, F., Borges, F., Bourcier, D., Bourgine, P., Conrad, J. G., Francesconi, E., et al. (2012) · Artificial Intelligence and Law · doi:10.1007/s10506-012-9131-x · Cited by 168
Aleven, V. (2003) · Artificial Intelligence · doi:10.1016/s0004-3702(03)00105-x · Cited by 164
Automatically classifying case texts and predicting outcomes
Ashley, K. D., & Brüninghaus, S. (2009) · Artificial Intelligence and Law · doi:10.1007/s10506-009-9077-9 · Cited by 158
(2021) · doi:10.1145/3462757.3466088 · Cited by 151
Natural language processing in law: Prediction of outcomes in the higher courts of Turkey
Mumcuoğlu, E., Öztürk, C. E., Özaktaş, H. M., & Koç, A. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102684 · Cited by 144
Self-citationSome Speculation about Artificial Intelligence and Legal Reasoning
Buchanan, B. G., & Headrick, T. E. (1970) · Stanford Law Review · doi:10.2307/1227753 · Cited by 141
Competing Approaches to Predicting Supreme Court Decision Making
Martin, A. D., Quinn, K. M., Ruger, T. W., & Kim, P. T. (2004) · Perspectives on Politics · doi:10.1017/s1537592704040502 · Cited by 133
Chalkidis, I., Androutsopoulos, I., & Michos, A. (2017) · Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law · doi:10.1145/3086512.3086515 · Cited by 132
Investigating Gender Bias in BERT
Bhardwaj, R., Majumder, N., & Poria, S. (2021) · Cognitive Computation · doi:10.1007/s12559-021-09881-2 · Cited by 112
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M. J., Androutsopoulos, I., Katz, D. M., & Aletras, N. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3936759 · Cited by 111
Gender-preserving Debiasing for Pre-trained Word Embeddings
Kaneko, M., & Bollegala, D. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1160 · Cited by 109
A Survey on Contextual Embeddings
Coelho, J., Neto, A., Tavares, M., Coutinho, C., Ribeiro, R., & Batista, F. (2020) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2021.12 · Cited by 109
Case-based reasoning and its implications for legal expert systems
Ashley, K. D. (1992) · Artificial Intelligence and Law · doi:10.1007/bf00114920 · Cited by 106
Thirty years of artificial intelligence and law: the third decade
Villata, S., Araszkiewicz, M., Ashley, K. D., Bench-Capon, T. J. M., Branting, L. K., Conrad, J. G., & Wyner, A. (2022) · Artificial Intelligence and Law · doi:10.1007/s10506-022-09327-6 · Cited by 105
It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution
Hall Maudslay, R., Gonen, H., Cotterell, R., & Teufel, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1530 · Cited by 103
In-Processing Modeling Techniques for Machine Learning Fairness: A Survey
Wan, M., Zha, D., Liu, N., & Zou, N. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3551390 · Cited by 99
Extreme Multi-Label Legal Text Classification: A Case Study in EU Legislation
Chalkidis, I., Fergadiotis, E., Malakasiotis, P., Aletras, N., & Androutsopoulos, I. (2019) · Proceedings of the Natural Legal Language Processing Workshop 2019 · doi:10.18653/v1/w19-2209 · Cited by 89
Debiasing Pre-trained Contextualised Embeddings
Kaneko, M., & Bollegala, D. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.107 · Cited by 87
Chalkidis, I., Fergadiotis, M., & Androutsopoulos, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.559 · Cited by 84
Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias
Bartl, M., Nissim, M., & Gatt, A. (2020) · arXiv · doi:10.48550/arxiv.2010.14534 · Cited by 84 · cited 6×
“Target word is masked μ ← μ + AssociationScore ( S ( A , T )) 7: end for by the equity evaluation corpus (EEC) , a benchmark corpus developed to evaluate gender and racial biases in NLP models by using emotion features [48].”
BackgroundMethodInfluentialChalkidis, I., Fergadiotis, M., Tsarapatsanis, D., Aletras, N., Androutsopoulos, I., & Malakasiotis, P. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.22 · Cited by 78
Gender Bias in the Job Market: A Longitudinal Analysis
Tang, S., Zhang, X., Cryan, J., Metzger, M. J., Zheng, H.-T., & Zhao, B. Y. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134734 · Cited by 74
Soh, J., Khang, L. H., & Chai, I. E. (2019) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/w19-2208 · Cited by 72
Nguyen, T.-S., Nguyen, L.-M., Tojo, S., Satoh, K., & Shimazu, A. (2018) · Artificial Intelligence and Law · doi:10.1007/s10506-018-9225-1 · Cited by 67
Neill, J. O., Buitelaar, P., Robin, C., & Brien, L. O. (2017) · International Conference on Artificial Intelligence and Law · doi:10.1145/3086512.3086528 · Cited by 65
Obligation and Prohibition Extraction Using Hierarchical RNNs
Chalkidis, I., Androutsopoulos, I., & Michos, A. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2041 · Cited by 62
MasakhaNER: Named entity recognition for African languages
(2021) · HAL · doi:10.1162/tacl · Cited by 61
Named-entity recognition in Turkish legal texts
Çetindağ, C., Yazıcıoğlu, B., & Koç, A. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000304 · Cited by 58
Self-citationThirty years of Artificial Intelligence and Law: the first decade
Governatori, G., Bench-Capon, T. J. M., Verheij, B., Araszkiewicz, M., Francesconi, E., & Grabmair, M. (2022) · Artificial Intelligence and Law · doi:10.1007/s10506-022-09329-4 · Cited by 57
The role of context in case-based legal reasoning: teleological, temporal, and procedural
Hafner, C. D., & Berman, D. H. (2002) · Artificial Intelligence and Law · doi:10.1023/a:1019516031847 · Cited by 55
Ethnic Bias in Judicial Decision Making: Evidence from Criminal Appeals in Kenya
Choi, D. D., Harris, J. A., & Shen-Bayh, F. F. (2022) · American Political Science Review · doi:10.1017/s000305542100143x · Cited by 51
LegalGNN: Legal Information Enhanced Graph Neural Network for Recommendation
Yang, J., Ma, W., Zhang, M., Zhou, X. Q., Liu, Y., & Ma, S. (2021) · ACM Transactions on Information Systems · doi:10.1145/3469887 · Cited by 48
Thirty years of Artificial Intelligence and Law: the second decade
Sartor, G., Araszkiewicz, M., Atkinson, K. M., Bex, F., van Engers, T., Francesconi, E., Prakken, H., Sileno, G., Schilder, F., Wyner, A., et al. (2022) · Artificial Intelligence and Law · doi:10.1007/s10506-022-09326-7 · Cited by 48
Gender Attitudes in the Judiciary: Evidence from U.S. Circuit CourtsPreprint
Ash, E., Chen, D. L., & Ornaghi, A. (2021) · RePEc: Research Papers in Economics · doi:10.3929/ethz-b-000472797 · Cited by 45
Elnaggar, A., Gebendorfer, C., Glaser, I., & Matthes, F. (2018) · Artificial Intelligence and Cloud Computing Conference (AICCC) · doi:10.1145/3299819.3299844 · Cited by 39
Gender bias in legal corpora and debiasing it
Sevim, N., Şahinuç, F., & Koç, A. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000122 · Cited by 31
Self-citationUnderstanding Relevance Judgments in Legal Case Retrieval
Shao, Y., Wu, Y., Liu, Y., Mao, J., & Ma, S. (2022) · ACM Transactions on Information Systems · doi:10.1145/3569929 · Cited by 31
A Two-Phase Framework for Learning Logical Structures of Paragraphs in Legal Articles
Bach, N. X., Minh, N. L., Tran, O. T., & Shimazu, A. (2013) · ACM Transactions on Asian Language Information Processing · doi:10.1145/2425327.2425330 · Cited by 30
Minot, J. R., Cheney, N., Maier, M., Elbers, D. C., Danforth, C. M., & Dodds, P. S. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3524887 · Cited by 23
Emerging trends: Unfair, biased, addictive, dangerous, deadly, and insanely profitable
Church, K., Schoene, A. M., Ortega, J. E., Chandrasekar, R., & Kordoni, V. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000481 · Cited by 22
Thirty years of Artificial Intelligence and Law: Editor’s Introduction
Bench-Capon, T. J. M. (2022) · Artificial Intelligence and Law · doi:10.1007/s10506-022-09325-8 · Cited by 21
Autoencoding Improves Pre-trained Word Embeddings
Kaneko, M., & Bollegala, D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.149 · Cited by 20
Deep Learning for Named-Entity Linking with Transfer Learning for Legal Documents
Elnaggar, A., Otto, R., & Matthes, F. (2018) · Artificial Intelligence and Cloud Computing Conference (AICCC) · doi:10.1145/3299819.3299846 · Cited by 19
Mulan: A Multiple Residual Article-Wise Attention Network for Legal Judgment Prediction
Chen, J., Du, L., Liu, M., & Zhou, X. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3503157 · Cited by 18
Thirty years of Artificial Intelligence and Law: overviews
Araszkiewicz, M., Bench-Capon, T. J. M., Francesconi, E., Lauritsen, M., & Rotolo, A. (2022) · Artificial Intelligence and Law · doi:10.1007/s10506-022-09324-9 · Cited by 17
Proceedings of the Natural Legal Language Processing Workshop 2019
(2019) · doi:10.18653/v1/w19-22 · Cited by 13
Legal Judgment Elements Extraction Approach with Law Article-aware Mechanism
Hu, Z., Pan, B., & Li, R. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3485244 · Cited by 12
DWE-Med: Dynamic Word Embeddings for Medical Domain
Jha, K., Xun, G., Gopalakrishnan, V., & Zhang, A. (2019) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3310254 · Cited by 8
Asmat, R., & Kossuth, L. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3964747 · Cited by 2
(2015) · doi:10.4230/oasics.slate.2021.8 · Cited by 1
(2014) · Institutional Repository University of Antioquia (University of Antioquia) · doi:10.1371/journal.pone
Research works printed but not linked: 9
- [5] Kevin D. Ashley. 1988. Modelling Legal Argument: Reasoning with Cases and Hypotheticals . Ph. D. Dissertation. University of Massachusetts. -
- [16] Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS'16) . Curran Associates Inc., Red Hook, NY, 435-364. -
- [26] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2019. Neural contract element extraction revisited. In Proceedings of the Workshop on Document Intelligence at NeurIPS . Retrieved from https://openreview.net/forum?id=B1x6fa95UH -
- [36] Filippo Galgani, Paul Compton, and Achim Hoffmann. 2012. Combining different summarization techniques for legal text. In Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data (HYBRID'12) . Association for Computational Linguistics, 115-123. -
- [50] [Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: Alite BERT for self-supervised learning of language representations. In Proceedings of the 8th International Conference on Learning Representations (ICLR'20) . OpenReview.net, 17 pages. Retrieved from https://openreview.net/forum?id= H1eA7AEtvS](https://openreview.net/forum?id=H1eA7AEtvS) -
- [64] Conor O'Sullivan and Joeran Beel. 2019. Predicting the outcome of judicial decisions made by the European Court of Human Rights. In Proceedings of the 27th AIAI Irish Conference on Artificial Intelligence and Cognitive Science . -
- [67] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9 . -
- [78] [Don Tuggener, Pius von Däniken, Thomas Peetz, and Mark Cieliebak. 2020. LEDGAR: A large-scale multi-label corpus for text classification of legal provisions in contracts. In Proceedings of the 12th Language Resources and Evaluation Conference . European Language Resources Association, 1235-1241. Retrieved from https://aclanthology.org/2020.lrec1.155](https://aclanthology.org/2020.lrec-1.155) -
- [87] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 17283-17297. Retrieved from https://proceedings.neurips.cc/paper/2020/file/ c8512d142a2d849725f31a9a7a361ab9-Paper.pdf -
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Sociodemographic biases in medical decision making by large language models
Omar, M., Soffer, S., Agbareia, R., Bragazzi, N. L., Apakama, D. U., Horowitz, C. R., Charney, A. W., Freeman, R., Kummer, B., Glicksberg, B. S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03626-6 · Cited by 173
“Our findings align with earlier reports identifying gender biases in LLM-based clinical recommendations 20,35,36 .”
Omar, M., Sorin, V., Agbareia, R., Apakama, D. U., Soroush, A., Sakhuja, A., Freeman, R., Horowitz, C. R., Richardson, L. D., Nadkarni, G. N., et al. (2025) · International Journal for Equity in Health · doi:10.1186/s12939-025-02419-0 · Cited by 110
Navigating LLM ethics: advancements, challenges, and future directions
Jiao, J., Afroogh, S., Xu, Y., & Phillips, C. (2025) · AI and Ethics · doi:10.1007/s43681-025-00814-5 · Cited by 216
Enhancing Sentiment Analysis in Stock Market Tweets Through BERT-Based Knowledge Transfer
Cicekyurt, E., & Bakal, G. (2025) · Computational Economics · doi:10.1007/s10614-025-10901-8 · Cited by 39
Metadata
- Authors
- Bozdag, M., Sevim, N., & Koç, A.
- Affiliations
- Bilkent University (TR)
- Venue
- ACM Transactions on Knowledge Discovery from Data
- Published
- 2024-02-13
- Volume
- 18
- Issue
- 4
- Pages
- 1-26
- ISSN
- 1556-4681 (ISSN-L)
- 1556-472X
- OpenAlex
- W4387735167
- Cited by
- 15
- References
- 89
- Citations by year
- 2026: 5 · 2025: 7 · 2024: 3
- FWCI
- 3.76
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Knowledge Discovery from Data (publishedVersion)
- Bilkent University Institutional Repository (Bilkent University) (submittedVersion, cc-by)
- Funders
- TUBITAK 1001 grant (120E346)
- BAGEP 2023 Young Scientist Award
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- gender bias, social bias, contextual language models, BERT, legal NLP, computational law, bias measurement, bias mitigation, fine-tuning, transformer models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.