The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index; none of the 5 sources its claims cite is checked for support yet.
Information Retrieval in the Age of Generative AI: A Mismatch That Matters
Open accessAbstract
This short piece explores a widespread and yet underexamined or even overlooked misconception, that is, large language models (LLMs) function like traditional legal research databases. They do not....
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This short conceptual essay argues that large language models embedded in legal research platforms do not function like traditional legal databases, differing in inputs, retrieval processes, and outputs in ways that undermine transparency, traceability, and replicability. It contends that these differences also undercut the active, cognitively demanding retrieval practices that build legal research skills and cumulative knowledge.
Key findings
The paper's claims are conceptual rather than empirical:
- LLM-based retrieval differs from database retrieval on four dimensions: unclear scope of training input, uncertain timeliness of static training data, opaque data-handling practices (e.g., whether terms like "may," "must," and "shall" are distinguished, or whether overruled decisions are flagged), and the loss of modular search/filtering that let researchers search only statutes, regulations, or case law.
- Traditional databases are evaluated with precision (proportion of retrieved results that are relevant) and recall (proportion of all relevant results retrieved), and their inputs directly determine outputs in a controllable, retraceable, auditable way; LLMs instead generate probabilistically, so identical prompts can yield different responses and outputs cannot be traced to a definitive source.
- The essay cites external evidence that legal AI models hallucinate in "1 out of 6 (or more) benchmark queries" (Stanford HAI, cited in the paper) and that retrieval-augmented generation (RAG) is "not a panacea," since sourcing, cleaning, labeling, and updating of external content remains uncertain.
- It argues LLM retrieval bypasses the query construction, Boolean logic, jurisdiction identification, and authority filtering that support learning, and recommends an open/curious mindset, formal metacognitive training, resistance to shortcuts, and an adaptive framework.
Methods
This is a conceptual/reflective essay, not an empirical study; the author states it is a "short piece" offering "initial thoughts" and recommendations, with no dataset, sample, or statistical analysis. The analysis is a comparative conceptual account of print-based, computer-assisted, AI-enhanced, and generative-AI-driven legal research phases. The author notes that some vendor practices described are "based on conversations with vendors," and the article is expressly limited to the U.S. legal system and common law tradition.
Limitations
Stated by the author: the piece is constrained by a word limit, so recommendations are offered "without fully elaborating on them"; broader theoretical and methodological dimensions of generative AI are declared outside its scope; and the analysis is limited to the U.S. legal system, particularly the common law tradition. Evident from the text: there is no original empirical data, sample, or measurement — the central claims rest on conceptual argument and secondary citations, and the vendor-practice observations are anecdotal ("based on conversations with vendors") rather than systematically documented.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
16 Own interpretation · 5 Sources not found
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Probably more important to realize is that the information retrieval process based on LLMs contradicts the learning theories that underlie legal education and expertise development.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many LLMs are typically trained on static datasets that do not update in real time.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“Beyond the well-documented problem of hallucinations, 11 a particularly concerning characteristic of AI-generated content is its ability to appear fluent and convincing, while potentially being inaccurate.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
What the sources say
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“While prompt engineering may influence the form or direction of the outputs, it does not fundamentally determine how or what information is retrieved and generated.8”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
What the sources say
- Not matchedcites [8]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“large language models (LLMs) function like traditional legal research databases.They do not.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Relying outdated or superseded materials even just as part of the training data, especially without indicating the recency of data relied on, could lead to misleading or legally inaccurate outputs.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fourth, many searching and filtering features are missing in the gen AI drive system.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This opacity undermines two essential pillars of legal research: transparency and accountability.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This dynamic has shifted with the rise of gen AI-driven research, where traditional researcher autonomy is increasingly constrained.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- It may state two separate claims as one. 7
- Probably not a substantial claim. 5
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The input directly determines the output and the search process itself is both controllable and retraceable.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Legal research is an important process for legal reasoning.2”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
What the sources say
- Not matchedcites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“information retrieval 1 from databases functions very differently from LLMs in terms of inputs, retrieval processes, and outputs.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“studies have shown that the very process of information retrieval plays a critical role in reinforcing learning and supporting long-term knowledge construction.3”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
What the sources say
- Not matchedcites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike traditional legal research databases which rely on deterministic queries and verified sources, LLMs operate through probabilistic text generation.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, LLM-based retrieval bypasses much of this active engagement, offering polished-looking outputs without requiring the user to undertake the interpretative work that underpins meaningful legal understanding.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, information retrieval in LLMs exhibits fundamentally different characteristics.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although LLM outputs may appear fluent, they can mask factual inaccuracy, distort precedent, or invent citations altogether, which post serious risks in legal contexts that demand precision, source traceability, and replicable reasoning.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These differences have significant implications for transparency, traceability, and overall effectiveness in AI-driven legal research.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“AI is not just amplifying information overload.It reveals a new, deeper problem in the information ecosystem: information contamination, with hallucinations as one of the most prominent types.7”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
What the sources say
- Not matchedcites [7]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These kinds of data handling practices, from selection, preprocessing, to annotation and training, can vary widely between AI providers but are rarely shared with users, especially by the proprietary, commercial vendors.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Why LLMs Are Not Databases and Why It Matters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without intentional oversight and adaption, these changes could profoundly affect how we develop research skills and a cumulative knowledge base, both of which are essential skills for lifelong learning in the legal field.”
§ Information Retrieval in the Age of Generative AI: A Mismatch That Matters > Abstract
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper relies on this without support.
AI excels at recognizing patterns across large volumes of information.
“it is a well-established fact that AI excels at recognizing patterns across large volumes of information.”
The paper relies on this without support.
Humans have a natural tendency to rely on mental and physical shortcuts, especially under time pressure.
“Third, humans have a natural tendency to rely on mental and physical shortcuts, especially under time pressure.”
The paper relies on this without support.
AI tools are ultimately designed to complete research tasks without disclosing the underlying process and sources used.
“AI tools are ultimately designed to complete research tasks without disclosing the underlying process and sources used.”
Citations
Cites
Oppenheimer, D. M. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.02.014 · Cited by 750
What Prompts Don’t Say: Understanding and Managing Underspecification in LLM Prompts
Yang, C., Shi, Y., Ma, Q., Liu, M. X., Kästner, C., & Wu, T. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.441 · Cited by 59
Sidra, S., & Mason, C. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00211 · Cited by 31
Morris, M. R. (2024) · Communications of the ACM · doi:10.1145/3673861 · Cited by 17
Teaching epistemic integrity to promote reliable scientific communication
Allard, A., & Clavien, C. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1308304 · Cited by 6
What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts
Yang, C., Shi, Y., Ma, Q., Liu, M. X., Kästner, C., & Wu, T. (2025) · arXiv · doi:10.48550/arxiv.2505.13360 · Cited by 3
The Algorithm as a Human Artifact: Implications for Legal {Re}Search
Mart, S. N. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2859720
Research works printed but not linked: 1
- Information contamination existed before the rise of gen AI, but gen AI significantly exacerbates the problem. yujie Sun et al., AI Hallucination: Towards a Comprehensive Classification of Distorted Information in Artificial Intelligence-Generated Content, 11 Human. Soc. Sci. Commun. art. no. 1,278 (2024) (categorizing distorted information based on error types and underlying causes); varun Magesh et al., Hallucination- Free? Assessing the Reliability of Leading AI Legal Research Tools, 22 J. Empirical Legal Stud. 216 (2025) (dis- cussing the hallucinations in the context of legal research tools).
Cited by
Sorted by citations
Reyes-Lillo, D., Morales-Vargas, A., & Rovira, C. (2025) · INFONOMY · doi:10.3145/infonomy.25.034 · Cited by 3
Employee-driven leakage of technical documentation into general-purpose llms: An integrative review
Latinović, L., Zhukovskiy, O. I., Mašić, O., & Živković, D. (2026) · Serbian Journal of Engineering Management · doi:10.5937/sjem2601077l
Girish, M., & Challa, M. (2026) · International Conference on Knowledge Engineering and Communication Systems (ICKECS) · doi:10.1109/ickecs70176.2026.11527706
Михайлюта, М. (2026) · Futurity Economics&Law · doi:10.57125/fel.2026.09.25.02
Metadata
- Authors
- Zhang, A.
- Affiliations
- Duke University (US)
- Corresponding author
- Zhang, Alex
- Venue
- Legal Reference Services Quarterly
- Publisher
- Informa UK Limited
- Published
- 2025-07-03
- Volume
- 44
- Issue
- 3
- Pages
- 297-306
- ISSN
- 0270-319X (ISSN-L)
- 1540-949X
- OpenAlex
- W4412829158
- Semantic Scholar
- b76cc2723894e66abdc74ad141bdc0bc11b6451c
- Cited by
- 4
- References
- 2
- Citations by year
- 2026: 3 · 2025: 1
- FWCI
- 2.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Legal Reference Services Quarterly (publishedVersion, cc-by-nc-nd)
- Duke Law Scholarship Repository (Duke University) (submittedVersion, cc-by-nc-nd)
- Funders
- University of Central Florida
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Law › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, generative AI, legal databases, information retrieval, legal research, AI limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
