The DOI matches its Crossref record; 39 of the 40 references it cites were found in a registry or an index.
Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction
Open accessAbstract
Legal Judgment Prediction (LJP) aims to predict judgment results according to the facts of cases. In recent years, LJP has drawn increasing attention rapidly from both academia and the legal industry, as it can provide references for legal practitioners and is expected to promote judicial justice. However, the research to date usually suffers from the lack of interpretability, which may lead to ethical issues like inconsistent judgments or gender bias. In this paper, we present QAjudge, a model based on reinforcement learning to visualize the prediction process and give interpretable judgments. QAjudge follows two essential principles in legal systems across the world: Presumption of Innocence and Elemental Trial. During inference, a Question Net will select questions from the given set and an Answer Net will answer the question according to the fact description. Finally, a Predict Net will produce judgment results based on the answers. Reward functions are designed to minimize the number of questions asked. We conduct extensive experiments on several real-world datasets. Experimental results show that QAjudge can provide interpretable judgments while maintaining comparable performance with other state-of-the-art LJP models. The codes can be found from https://github.com/thunlp/QAjudge.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“With the aim of providing an explanation for legal judgment predictions generated by Legal Judgement Prediction (LJP), Zhong et al. [71] designed the QAjudge.”
From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [71]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
(2017) · HAL · Cited by 10,512
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Ho, T. K. (2002) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.1995.598994 · Cited by 6,311
Deep Reinforcement Learning for Dialogue Generation
Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., & Gao, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1127 · Cited by 1,393
Deep Pyramid Convolutional Neural Networks for Text Categorization
Johnson, R., & Zhang, T. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1052 · Cited by 877
Self-citationΑλέτρας, Ν., Tsarapatsanis, D., Preoțiuc-Pietro, D., & Lampos, V. (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.93 · Cited by 813
A general approach for predicting the behavior of the Supreme Court of the United States
Katz, D. M., Bommarito, M. J., & Blackman, J. (2017) · PLoS ONE · doi:10.1371/journal.pone.0174698 · Cited by 562
Entity-Relation Extraction as Multi-Turn Question Answering
Li, X., Yin, F., Sun, Z., Li, X., Yuan, A., Chai, D., Zhou, M., & Li, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1129 · Cited by 431
CAIL2018: A Large-Scale Legal Dataset for Judgment PredictionPreprint
Xiao, C., Zhong, H., Guo, Z., Tu, C., Liu, Z., Sun, M., Feng, Y., Han, X., Hu, Z., Wang, H., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02478 · Cited by 424
Self-citationSammut, C., & Webb, G. I. (2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_100391 · Cited by 387
Legal Judgment Prediction via Topological Learning
Zhong, H., Guo, Z., Tu, C., Xiao, C., Liu, Z., & Sun, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1390 · Cited by 377
Self-citationPredicting Supreme Court Cases Probabilistically: The Search and Seizure Cases, 1962-1981
Segal, J. A. (1984) · American Political Science Review · doi:10.2307/1955796 · Cited by 364
Reinforcement Learning for Relation Classification From Noisy Data
Feng, J., Huang, M., Li, Z., Yang, Y., & Zhu, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12063 · Cited by 360
Learning to Predict Charges for Criminal Cases with Legal Basis
Luo, B., Feng, Y., Xu, J., Zhang, X., & Zhao, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1289 · Cited by 348
Language Understanding for Text-based Games using Deep Reinforcement Learning
Narasimhan, K., Kulkarni, T. D., & Barzilay, R. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1001 · Cited by 341
Deep Reinforcement Learning with a Natural Language Action Space
He, J., Chen, J., He, X., Gao, J., Li, L. C., Deng, L., & Mari Ostendorf, E. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1153 · Cited by 245
Grgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207
Few-Shot Charge Prediction with Discriminative Legal Attributes
Hu, Z., Li, X., Tu, C., Liu, Z., & Sun, M. (2018) · International Conference on Computational Linguistics · Cited by 172
Self-citationYe, H., Jiang, X., Luo, Z., & Chao, W. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1168 · Cited by 159
The Supreme Court's Many Median Justices
Lauderdale, B. E., & Clark, T. S. (2012) · American Political Science Review · doi:10.1017/s0003055412000469 · Cited by 154
The Presumption of Innocence and the Human Rights Act
Tadros, V., & Tierney, S. (2004) · Modern Law Review · doi:10.1111/j.1468-2230.2004.00493.x · Cited by 120
Predicting associated statutes for legal problems
Liu, Y.-H., Chen, Y., & Ho, W.-L. (2014) · Information Processing & Management · doi:10.1016/j.ipm.2014.07.003 · Cited by 75
Exploring Phrase-Based Classification of Judicial Documents for Criminal Charges in Chinese
Liu, C.-L., & Hsieh, C.-D. (2006) · Lecture notes in computer science · doi:10.1007/11875604_75 · Cited by 73
Lin, W.-C., Kuo, T.-T., Chang, T.-J., Yen, C.-A., Chen, C.-J., & Lin, S.-D. (2012) · Cited by 72
The Presumption of Innocence in the French and Anglo-American Legal Traditions
Quintard-Morénas, F. (2010) · The American Journal of Comparative Law · doi:10.5131/ajcl.2009.0005 · Cited by 66
Wu, J., Li, L., & Wang, W. Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1113 · Cited by 60
Quantitative Analysis of Judicial Processes: Some Practical and Theoretical Applications
Ulmer, S. S. (1963) · Law and Contemporary Problems · doi:10.2307/1190728 · Cited by 57
Case Instance Generation and Refinement for Case-Based Criminal Summary Judgments in Chinese
Liu, C.-L., Chang, C.-T., & Ho, J.-H. (2004) · Cited by 50
The Criminal Procedure Law of the People's Republic of China
Cohen, J. A. (1982) · The Journal of Criminal Law and Criminology (1973-) · doi:10.2307/1143030 · Cited by 31
Playing 20 Question Game with Policy-Based Reinforcement Learning
Hu, H., Wu, X., Luo, B., Tao, C., Xu, C., Wu, W. B., & Chen, Z. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1361 · Cited by 28
Cited by
Sorted by citations
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“With the aim of providing an explanation for legal judgment predictions generated by Legal Judgement Prediction (LJP), Zhong et al. [71] designed the QAjudge.”
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M. J., Androutsopoulos, I., Katz, D. M., & Αλέτρας, Ν. (2021) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.297 · Cited by 511
How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence
Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.466 · Cited by 485
Self-citationLarge language models in law: A survey
Lai, J., Gan, W., Wu, J., Qi, Z., & Yu, P. S. (2024) · AI Open · doi:10.1016/j.aiopen.2024.09.002 · Cited by 263
ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation
Malik, V., Sanjay, R., Nigam, S. K., Ghosh, K., Guha, S. K., Bhattacharya, A., & Modi, A. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.313 · Cited by 198
Metadata
- Authors
- Zhong, H., Wang, Y., Tu, C., Zhang, T., Liu, Z., & Sun, M.
- Affiliations
- Tsinghua University (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2020-04-03
- Volume
- 34
- Issue
- 01
- Pages
- 1250-1257
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2997429021
- Cited by
- 113
- References
- 40
- Citations by year
- 2026: 8 · 2025: 16 · 2024: 17 · 2023: 30 · 2022: 18 · 2021: 21 · 2020: 3
- FWCI
- 84.97
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- National Natural Science Foundation of China (61661146007, 61572273)
- National Key Research and Development Program of China (2018YFC0831900)
- Topics
- Social Sciences › Law › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Legal processes and jurisprudence
- Keywords
- legal judgment prediction, interpretability, reinforcement learning, question answering, presumption of innocence, gender bias, judicial justice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
