AI Assistance in Legal Analysis: An Empirical Study
Open accessAbstract
Can artificial intelligence (AI) augment human legal reasoning? To find out, we designed a novel experiment administering law school exams to students with and without access to GPT-4, the best-performing AI model currently available. We found that assistance from GPT-4 significantly enhanced performance on simple multiple-choice questions but not on complex essay questions. We also found that GPT-4’s impact depended heavily on the student’s starting skill level; students at the bottom of the class saw huge performance gains with AI assistance, while students at the top of the class saw performance declines. This suggests that AI may have an equalizing effect on the legal profession, mitigating inequalities between elite and nonelite lawyers. In addition, we graded exams written by GPT-4 alone to compare it with humans alone and AI-assisted humans. We found that GPT-4’s performance varied substantially depending on prompting methodology. With basic prompts, GPT-4 was a mediocre student, but with optimal prompting it outperformed both the average student and the average student with access to AI. This finding has important implications for the future of work, hinting that it may become advantageous to entirely remove humans from the loop for certain tasks.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Less experienced and lower-skilled workers tend to benefit most from AI assistance.
“While the skill-based effects differ across studies, a consistent pattern emerges in call center work, consulting, legal question-answering, and writing: less experienced and lower-skilled workers tend to benefit most [Brynjolfsson et al., 2025, Dell'Acqua et al., 2023, Choi and Schwarcz, 2023, Noy and Zhang, 2023].”
From How AI Impacts Skill Formation (2026) · cites this paper as Choi and Schwarcz, 2023
Cited by
Sorted by citations
Dell’Acqua, F., McFowland, E., Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026) · Organization Science · doi:10.1287/orsc.2025.21838 · Cited by 1,189 · see citing works
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083 · see citing works
Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794 · see citing works
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 288 · see citing works
Legalbench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D., et al. (2023) · doi:10.2139/ssrn.4583531 · Cited by 154 · see citing works
Self-citationAkiba, D., & Fraboni, M. C. (2023) · Education Sciences · doi:10.3390/educsci13090885 · Cited by 115 · see citing works
Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2405782 · Cited by 89 · see citing works
“…domain (Gu et al., 2023; 2023; Srinivasa Ragavan et al., 2022); in the legal domain, where “AI-assisted exams were more likely to miss hidden issues” (Choi & Schwarcz, 2023); and in the design domain, where one participant commented, “I would never design it like that, but this [GenAI system]…”
InfluentialNyberg, A. J., Schleicher, D. J., Bell, B. S., Boon, C., Cappelli, P., Collings, D. G., Dalle Molle, J. E., Feuerriegel, S., Gerhart, B., Jeong, Y., et al. (2025) · Journal of Management · doi:10.1177/01492063251325188 · Cited by 47 · see citing works
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Becker, J., Rush, N., Barnes, E., & Rein, D. (2025) · arXiv · doi:10.48550/arxiv.2507.09089 · Cited by 16 · see citing works
“Existing empirical work on the micro-level effects of generative AI tools tends to find that access to these tools benefits less experienced workers more, compressing performance distributions [22; 23; 10; 24].”
Generative AI’s Impact on Graduate Student Professional Writing Productivity and Quality
Connell Pensky, A. E., Usdan, J. H., & Chang, H. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00528-z · Cited by 9 · see citing works
How AI Impacts Skill Formation
Shen, J. H., & Tamkin, A. (2026) · arXiv · doi:10.48550/arxiv.2601.20245 · Cited by 4 · see citing works
“…effects differ across studies, a consistent pattern emerges in call center work, consulting, legal question-answering, and writing: less experienced and lower-skilled workers tend to benefit most [Brynjolfsson et al., 2025, Dell’Acqua et al., 2023, Choi and Schwarcz, 2023, Noy and Zhang, 2023].”
- Authors
- Choi, J. H., & Schwarcz, D.
- Affiliations
- University of Minnesota (US)
- Corresponding author
- Schwarcz, Daniel
- ORCID
- Schwarcz, Daniel
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2023
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W4385884015
- Cited by
- 80
- References
- 0
- Citations by year
- 2026: 15 · 2025: 32 · 2024: 27 · 2023: 6
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- University of Minnesota Law School Scholarship Repository (University of Minnesota) (submittedVersion)
- Topics
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Social Sciences › Law › Legal Education and Practice Innovations
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- artificial intelligence in legal education, GPT-4, legal reasoning, multiple-choice questions, essay questions, student skill levels, AI-assisted grading, future of work
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
