CoChat
Sign Up
PreprintCited by 15Artificial Intelligence in LawJudicial and Constitutional StudiesComputational and Text Analysis Methods

The DOI matches its DataCite record.

Blind Judgement: Agent-Based Supreme Court Modelling With GPT

Open access
Hamilton, Sil·2023·arXiv
DOI 10.48550/arxiv.2301.05327arXiv 2301.05327
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a novel Transformer-based multi-agent system for simulating the judicial rulings of the 2010-2016 Supreme Court of the United States. We train nine separate models with the respective authored opinions of each supreme justice active ca. 2015 and test the resulting system on 96 real-world cases. We find our system predicts the decisions of the real-world Supreme Court with better-than-random accuracy. We further find a correlation between model accuracy with respect to individual justices and their alignment between legal conservatism & liberalism. Our methods and results hold significance for researchers interested in using language models to simulate politically-charged discourse between multiple agents.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2301.05327

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 17

  1. Allcorn, S.; and Stein, H. F. 2021. Unpacking the Supreme Court: The Age of Trump, Law, and Psychohistory. Journal of Psychohistory, 49(1).
  2. Argyle, L. P.; Busby, E. C.; Fulda, N.; Gubler, J.; Rytting, C.; and Wingate, D. 2022. Out of One, Many: Using Lan- guage Models to Simulate Human Samples. arXiv preprint arXiv:2209.06899. · arXiv
  3. Betz, G. 2022. Natural-Language Multi-Agent Simu- lations of Argumentative Opinion Dynamics. Journal of Artificial Societies and Social Simulation, 25(1): 2. ArXiv:2104.06737 · arXiv
  4. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Ad- vances in neural information processing systems, 33: 1877- 1901.
  5. Drori, I.; Zhang, S.; Shuttleworth, R.; Tang, L.; Lu, A.; Ke, E.; Liu, K.; Chen, L.; Tran, S.; Cheng, N.; et al. 2022. A neu- ral network solves, explains, and generates university math problems by program synthesis and few-shot learning at hu- man level. Proceedings of the National Academy of Sci- ences, 119(32): e2123433119.
  6. Hamilton, S.; and Piper, A. 2022. The COVID That Wasn't: Counterfactual Journalism Using GPT. In Proceedings of the 6th Joint SIGHUM Workshop on Computational Lin- guistics for Cultural Heritage, Social Sciences, Humanities and Literature, 83-93. Gyeongju, Republic of Korea: Inter- national Conference on Computational Linguistics.
  7. Jr, R. L. P.; Curry, B. W.; and Marshall, B. W. 2011. Decision Making by the Modern Supreme Court. Cambridge Univer- sity Press. ISBN 978-1-139-49879-1. Google-Books-ID:
  8. Katz, D. M.; Bommarito, M. J., II; and Blackman, J. 2017. A general approach for predicting the behavior of the Supreme Court of the United States. PLOS ONE, 12(4): 1-18.
  9. Kim, L.-S.; Kim, S.-s.; Jang, H.-S.; Park, S.-W.; and Kang, I.-H. 2020. Long-tail Query Expansion using Extractive and Generative Methods. In Annual Conference on Human and Language Technology, 267-273. Human and Language Technology.
  10. Kingma, D. P.; and Ba, J. 2014. Adam: A Method for Stochastic Optimization. Kromphardt, C. D. 2017. Evaluating the effect of law clerk gender on voting at the United States Supreme Court. Justice System Journal, 38(2): 183-201.
  11. Liu, P.; Yuan, W.; Fu, J.; Jiang, Z.; Hayashi, H.; and Neubig, G. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. · arXiv
  12. Peterson, J. C.; Giallouri, T.; and Menounou, E. 2021. The Personal Finances of United States Supreme Court Justices and Decision-making in Economic Litigation. The Journal of Legal Studies, 50(2): 379-405.
  13. Portelli, B.; Scaboro, S.; Santus, E.; Sedghamiz, H.; Cher- soni, E.; and Serra, G. 2022. Generalizing over Long Tail Concepts for Medical Term Normalization. arXiv preprint arXiv:2210.11947. · arXiv
  14. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9.
  15. Roberts, A.; Chung, H. W.; Levskaya, A.; Mishra, G.; Brad- bury, J.; Andor, D.; Narang, S.; Lester, B.; Gaffney, C.; Mo- hiuddin, A.; Hawthorne, C.; and Lewkowycz, A. 2022. Scal- ing Up Models and Data with t5x and seqio. arXiv preprint arXiv:2203.17189. · arXiv
  16. Rohrbach, A.; Hendricks, L. A.; Burns, K.; Darrell, T.; and Saenko, K. 2018. Object Hallucination in Image Captioning. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, 4035-4045. Brussels, Belgium: Association for Computational Linguistics.
  17. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. · arXiv

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Hamilton, S.
Affiliations
McGill University (CA)
Corresponding author
Hamilton, Sil
ORCID
Hamilton, Sil
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2301.05327
arXiv
2301.05327
OpenAlex
W4316829821
Cited by
13
References
0
Citations by year
2025: 3 · 2024: 4 · 2023: 6
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
McGill University
Topics
  • Social Sciences › Law › Artificial Intelligence in Law
  • Social Sciences › Law › Judicial and Constitutional Studies
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
United States Supreme Court, judicial decision prediction, multi-agent systems, transformer language models, GPT, legal opinions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review