How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN
Open accessAbstract
Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? To tease apart these possibilities, we introduce RAVEN, a suite of analyses for assessing the novelty of generated text, focusing on sequential structure (n-grams) and syntactic structure. We apply these analyses to four neural language models trained on English (an LSTM, a Transformer, Transformer-XL, and GPT-2). For local structure—e.g., individual dependencies—text generated with a standard sampling scheme is substantially less novel than our baseline of human-generated text from each model’s test set. For larger-scale structure—e.g., overall sentence structure—model-generated text is as novel or even more novel than the human-generated baseline, but models still sometimes copy substantially, in some cases duplicating passages over 1,000 words long from the training set. We also perform extensive manual analysis, finding evidence that GPT-2 uses both compositional and analogical generalization mechanisms and showing that GPT-2’s novel text is usually well-formed morphologically and syntactically but has reasonably frequent semantic issues (e.g., being self-contradictory).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs can mimic biases and other deficiencies from their training data or even outright plagiarize.
“LLMs can mimic biases and other deficiencies from their training data (38–41) or even outright plagiarize (42).”
From Delving into LLM-assisted writing in biomedical publications through excess vocabulary (2025) · cites this paper as [42]
Similar successes in performance and emergent structure have been shown in other linguistic domains, including emergent phonological structure, productive generation of morphologically complex neologisms, and rich lexical semantic information.
“But similar successes, both in performance and emergent structure, have been shown in other linguistic domains (e.g., emergent phonological structure [72], productive generation of morphologically complex neologisms [73], rich lexical semantic information [74], etc.).”
From Dissociating language and thought in large language models (2024) · cites this paper as [73]
Models seem to often be able to use novel words appropriately.
“Models also seem to (often) be able to use novel words appropriately [69, 73].”
From Dissociating language and thought in large language models (2024) · cites this paper as [73]
LLMs may be capable of generating artifacts that are new.
“Their stochastic nature and the variety of prompts that are usually provided commonly lead to novel outcomes (McCoy et al., 2023); LLMs may therefore be capable of generating artifacts that are also new.”
From On the creativity of large language models (2025) · cites this paper as McCoy et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2× · see citing works
“This acronym is also a nod to Bender et al.’s (2021) comparison of LMs to another utterance-generating bird, the parrot. to be a major concern in practice.”
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691 · see citing works
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,051 · see citing works
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · see citing works
Moses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735 · cited 1× · see citing works
“We tokenize all text with the Moses tokenizer (Koehn et al., 2007), which treats punc-tuation marks as separate tokens but otherwise does not break words into smaller units, and we then analyze n -grams formed from these to-kens.”
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022 · see citing works
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,765 · see citing works
The Child's Learning of English Morphology
Berko, J. (1958) · WORD · doi:10.1080/00437956.1958.11659661 · Cited by 2,504 · see citing works
The Cambridge Grammar of the English Language
Huddleston, R., & Pullum, G. (2005) · Zeitschrift für Anglistik und Amerikanistik · doi:10.1515/zaa-2005-0209 · Cited by 2,410 · see citing works
Understanding deep learning (still) requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021) · Communications of the ACM · doi:10.1145/3446776 · Cited by 2,400 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“On the other hand, larger models are generally stronger (Kaplan et al., 2020), which might include a greater ability to be novel.”
BackgroundMoses : open source toolkit for statistical machine translation
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 1,547 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
The 26th USENIX Security Symposium 2017
Αλεξόπουλος, Ν., Kiayias, A., Zacharias, T., & Talviste, R. (2017) · USENIX Security Symposium · Cited by 1,168 · see citing works
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · cited 1× · see citing works
“Much discussion about evaluating LMs focuses on whether they understand language (Bender and Koller, 2020; Marcus, 2020), whereas we assess the novelty of surface text.”
Overregularization in language acquisition
Marcus, G. F., Pinker, S., Ullman, M., Hollander, M., Rosen, T. J., Xu, F., & Clahsen, H. (1992) · Monographs of the Society for Research in Child Development · doi:10.2307/1166115 · Cited by 1,065 · see citing works
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · see citing works
Rules vs. analogy in English past tenses: a computational/experimental study
Albright, A., & Hayes, B. (2003) · Cognition · doi:10.1016/s0010-0277(03)00146-x · Cited by 777 · see citing works
Overfitting and undercomputing in machine learning
Dietterich, T. (1995) · ACM Computing Surveys · doi:10.1145/212094.212114 · Cited by 776 · see citing works
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · see citing works
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · see citing works
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · see citing works
Montague, R. (1970) · Theoria · doi:10.1111/j.1755-2567.1970.tb00434.x · Cited by 703 · see citing works
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669 · see citing works
Using the Output Embedding to Improve Language Models
Press, O., & Wolf, L. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2025 · Cited by 642 · see citing works
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629 · see citing works
Generalisation of regular and irregular morphological patterns
Prasada, S., & Pinker, S. A. (1993) · Language and Cognitive Processes · doi:10.1080/01690969308406948 · Cited by 547 · see citing works
Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought
Hofstadter, D. R. (1995) · Cited by 527 · see citing works
Recurrent Neural Network Grammars
Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1024 · Cited by 517 · see citing works
Colorless Green Recurrent Networks Dream Hierarchically
Gulordava, K., Bojanowski, P., Grave, É., Linzen, T., & Baroni, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1108 · Cited by 489 · see citing works
Self-citationConstituency Parsing with a Self-Attentive Encoder
Kitaev, N., & Klein, D. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1249 · Cited by 483 · see citing works
(2020) · arXiv (Cornell University) · Cited by 461 · see citing works
Texygen: A Benchmarking Platform for Text Generation Models
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., & Yu, Y. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210080 · Cited by 408 · see citing works
The Price of Linguistic Productivity
Yang, C. D. (2016) · The MIT Press eBooks · doi:10.7551/mitpress/9780262035323.001.0001 · Cited by 340 · see citing works
The acquisition of regular and irregular past tense forms
Kuczaj, S. A. (1977) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(77)80021-2 · Cited by 323 · see citing works
Improving Neural Language Models with a Continuous CachePreprint
Grave, E., Joulin, A., & Usunier, N. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1612.04426 · Cited by 316 · see citing works
Deduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304 · see citing works
The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
Marcus, G. (2020) · arXiv · doi:10.48550/arxiv.2002.06177 · Cited by 297 · see citing works
The Price of Linguistic Productivity: How Children Learn to Break the Rules of Language
Yang, C. D. (2016) · Project Muse (Johns Hopkins University) · Cited by 296 · see citing works
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285 · see citing works
Bauer, L., & Huddleston, R. (2002) · Cambridge University Press eBooks · doi:10.1017/9781316423530.020 · Cited by 267 · see citing works
Adaptive Input Representations for Neural Language ModelingPreprint
Baevski, A., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10853 · Cited by 255 · see citing works
Dialogue Natural Language Inference
Welleck, S., Weston, J., Szlam, A., & Cho, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1363 · Cited by 249 · see citing works
Productivity and Reuse in Language
O’Donnell, T. J. (2015) · The MIT Press eBooks · doi:10.7551/mitpress/9780262028844.001.0001 · Cited by 220 · see citing works
Slot and frame patterns and the development of the determiner category
Pine, J., & Lieven, E. (1997) · Applied Psycholinguistics · doi:10.1017/s0142716400009930 · Cited by 219 · see citing works
Automatic evaluation of text coherence: models and representations
Lapata, M., & Barzilay, R. (2005) · Cited by 218 · see citing works
Measuring Compositional Generalization: A Comprehensive Method on Realistic DataPreprint
Keysers, D., Schärli, N., Scales, N., Buisman, H., Furrer, D., Kashubin, S., Momchev, N., Sinopalnikov, D., Stafiniak, Ł., Tihon, T., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09713 · Cited by 213 · see citing works
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207 · see citing works
Self-citationQuantifying Memorization Across Neural Language Models
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022) · arXiv · doi:10.48550/arxiv.2202.07646 · Cited by 204 · see citing works
Compositionality Decomposed: How do Neural Networks Generalise?
Hupkes, D., Dankers, V., Mul, M., & Bruni, E. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11674 · Cited by 194 · see citing works
Unifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · see citing works
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
Kim, N., & Linzen, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.731 · Cited by 153 · see citing works
Self-citationOntogeny and phylogeny of language
Yang, C. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1216803110 · Cited by 142 · see citing works
Do Massively Pretrained Language Models Make Better Storytellers?
See, A., Pappu, A., Saxena, R., Yerukola, A., & Manning, C. D. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1079 · Cited by 139 · see citing works
A Systematic Assessment of Syntactic Generalization in Neural Language Models
Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.158 · Cited by 134 · see citing works
Systematicity in Connectionist Language Learning
Hadley, R. F. (1994) · Mind & Language · doi:10.1111/j.1468-0017.1994.tb00225.x · Cited by 132 · see citing works
PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking
Rashkin, H., Çelikyılmaz, A., Choi, Y., & Gao, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.349 · Cited by 130 · see citing works
Self-citationDon’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Li, M., Roller, S., Kulikov, I., Welleck, S., Boureau, Y.-L., Cho, K., & Weston, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.428 · Cited by 128 · see citing works
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N., & Choi, Y. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.501 · Cited by 118 · see citing works
Productivity and Reuse in Language: A Theory of Linguistic Computation and Storage
O’Donnell, T. J. (2015) · Project Muse (Johns Hopkins University) · Cited by 100 · see citing works
Efficient softmax approximation for GPUs
Grave, É., Joulin, A., Cissé, M., & Jeǵou, H. (2017) · International Conference on Machine Learning · Cited by 96 · see citing works
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N. A., & Choi, Y. (2021) · arXiv · doi:10.48550/arxiv.2107.01294 · Cited by 90 · see citing works
Efficient Second-Order TreeCRF for Neural Dependency Parsing
Zhang, Y., Li, Z., & Zhang, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.302 · Cited by 89 · see citing works
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85 · see citing works
Lake, B., & Baroni, M. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1711.00350 · Cited by 85 · see citing works
Language Model Evaluation Beyond Perplexity
Meister, C., & Cotterell, R. (2021) · doi:10.18653/v1/2021.acl-long.414 · Cited by 74 · see citing works
McCoy, R. T., Frank, R., & Linzen, T. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00304 · Cited by 72 · see citing works
Self-citationDeduplicating Training Data Mitigates Privacy Risks in Language Models
Kandpal, N., Wallace, E., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2202.06539 · Cited by 68 · see citing works
GLEU: Automatic Evaluation of Sentence-Level Fluency
Mutton, A., Dras, M., Wan, S., & Dale, R. (2007) · Cited by 65 · see citing works
On Compositional Generalization of Neural Machine Translation
Li, Y., Yin, Y.-J., Chen, Y., & Zhang, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.368 · Cited by 60 · see citing works
Do Language Models Plagiarize?
Lee, J., Le, T., Chen, J., & Lee, D. (2023) · Proceedings of the ACM Web Conference 2023 · doi:10.1145/3543507.3583199 · Cited by 51 · see citing works
Enhancing the Transformer with Explicit Relational Encoding for Math Problem SolvingPreprint
Schlag, I., Smolensky, P., Fernandez, R., Jojić, N., Schmidhuber, J., & Gao, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06611 · Cited by 45 · see citing works
Self-citationFrequency Effects on Syntactic Rule Learning in Transformers
Wei, J., Garrette, D., Linzen, T., & Pavlick, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.72 · Cited by 45 · see citing works
Self-citationTowards Tracing Knowledge in Language Models Back to the Training Data
Akyürek, E., Bolukbasi, T., Liu, F., Xiong, B., Tenney, I., Andreas, J., & Guu, K. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.180 · Cited by 40 · see citing works
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages
Ravfogel, S., Goldberg, Y., & Linzen, T. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1356 · Cited by 39 · see citing works
Self-citationInflectional morphology and related matters
Palmer, F. R., Huddleston, R., & Pullum, G. K. (2002) · Cambridge University Press eBooks · doi:10.1017/9781316423530.019 · Cited by 38 · see citing works
Scarecrow: A Framework for Scrutinizing Machine Text.Preprint
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N. A., & Choi, Y. (2021) · arXiv (Cornell University) · Cited by 17 · see citing works
Measuring Causal Effects of Data Statistics on Language Model's `Factual' PredictionsPreprint
Elazar, Y., Kassner, N., Ravfogel, S., Feder, A., Ravichander, A., Mosbach, M., Belinkov, Y., Schütze, H., & Goldberg, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.14251 · Cited by 15 · see citing works
Recurrent babbling: evaluating the acquisition of grammar from limited input data
Pannitto, L., & Herbelot, A. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.13 · Cited by 12 · see citing works
This is a BERT. Now there are several of them. Can they generalize to novel words?
Haley, C. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.31 · Cited by 11 · see citing works
Han, X., & Tsvetkov, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12600 · Cited by 10 · see citing works
Probing BERT’s priors with serial reproduction chains
Yamakoshi, T., Griffiths, T. L., & Hawkins, R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.314 · Cited by 8 · see citing works
Distinguishing rule- and exemplar-based generalization in learning systemsPreprint
Dasgupta, I., Grant, E., & Griffiths, T. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04328 · Cited by 7 · see citing works
Evaluating Distributional Distortion in Neural Language ModelingPreprint
LeBrun, B., Sordoni, A., & O’Donnell, T. J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.12788 · Cited by 7 · see citing works
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Flagged first, then by citations
4 of the 101 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- PMLR.
- Brenden Lake and Marco Baroni. 2018. Gen- eralization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pages 2873-2882.
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mix- ture models. In International Conference on Learning Representations.
Cited by
Sorted by citations
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07016 · Cited by 214
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2025) · Science Advances · doi:10.1126/sciadv.adt3813 · Cited by 213 · see citing works
Terwiesch, C., Meincke, L., Girotra, K., Mollick, E., Nave, G., & Ulrich, K. T. (2026) · Production and Operations Management · doi:10.1177/10591478261474243 · Cited by 191 · see citing works
Analyzing Leakage of Personally Identifiable Information in Language Models
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., & Zanella-Béguelin, S. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179300 · Cited by 181 · see citing works
Theory Is All You Need: AI, Human Cognition, and Causal Reasoning
(2024) · doi:10.1287/stsc.2024.0189 · Cited by 151 · see citing works
On the creativity of large language models
Franceschelli, G., & Musolesi, M. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02127-3 · Cited by 113 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Symbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95 · see citing works
The homogenizing effect of large language models on human expression and thought
Sourati, Z., Ziabari, A. S., & Dehghani, M. (2026) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2026.01.003 · Cited by 62 · see citing works
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models
Jang, J., Ye, S., Lee, C.-H., Yang, S., Shin, J., Han, J., Kim, G., & Seo, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.418 · Cited by 60 · see citing works
Linguistically inspired roadmap for building biologically reliable protein language models
Vu, M. H., Akbar, R., Robert, P. A., Świątczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00637-1 · Cited by 57 · see citing works
Do Language Models Plagiarize?
Lee, J., Le, T., Chen, J., & Lee, D. (2023) · Proceedings of the ACM Web Conference 2023 · doi:10.1145/3543507.3583199 · Cited by 51 · see citing works
Deng, C., Duan, Y., Jin, X., Chang, H., Tian, Y., Liu, H., Wang, Y., Gao, K., Zou, H. P., Jin, Y., et al. (2025) · AI and Ethics · doi:10.1007/s43681-025-00797-3 · Cited by 49 · see citing works
The Ethical Implications of Generative Audio Models: A Systematic Literature Review
Barnett, J. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604686 · Cited by 47 · see citing works
See Widely, Think Wisely: Toward Designing a Generative Multi-agent System to Burst Filter Bubbles
Zhang, Y., Sun, J., Feng, L., Yao, C., Fan, M., Zhang, L., Wang, Q., Geng, X., & Rui, Y. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642545 · Cited by 44 · see citing works
The Next Chapter: A Study of Large Language Models in Storytelling
Xie, Z., Cohn, T., & Lau, J. H. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.23 · Cited by 41 · see citing works
Weissbart, H., & Martin, A. E. (2024) · Nature Communications · doi:10.1038/s41467-024-53128-1 · Cited by 33 · see citing works
Training Data Extraction From Pre-trained Language Models: A Survey
Ishihara, S. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.23 · Cited by 26 · see citing works
Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
Shamardina, T., Mikhailov, V., Chernianskii, D., Fenogenova, A., Saidov, M., Valeeva, A., Shavrina, T., Smurov, I., Tutubalina, E. V., & Artemova, E. (2022) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2022-21-497-511 · Cited by 22 · see citing works
Usage-based constructionist approaches and large language models
Goldberg, A. E. (2024) · Constructions and Frames · doi:10.1075/cf.23017.gol · Cited by 21 · see citing works
Lee, A. V. Y. (2024) · Asia Pacific Journal of Education · doi:10.1080/02188791.2024.2305171 · Cited by 20 · see citing works
Impact of Co-occurrence on Factual Knowledge of Large Language Models
Kang, C., & Choi, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.518 · Cited by 18 · see citing works
LMs go Phishing: Adapting Pre-trained Language Models to Detect Phishing Emails
Misra, K., & Rayz, J. T. (2022) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat55865.2022.00028 · Cited by 16 · see citing works
A multidimensional study of AI adoption among University students in teacher education programs
Bećirović, S., Polz, E., & Tinkel, I. (2025) · Smart Learning Environments · doi:10.1186/s40561-025-00422-0 · Cited by 14 · see citing works
Holtzman, A., West, P., & Zettlemoyer, L. (2025) · Journal of Social Computing · doi:10.23919/jsc.2025.0009 · Cited by 14 · see citing works
Echoes in AI: Quantifying lack of plot diversity in LLM outputs
Xu, W., Jojić, N., Rao, S., Brockett, C., & Dolan, B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2504966122 · Cited by 14 · see citing works
Derivational morphology reveals analogical generalization in large language models
Hofmann, V., Weissweiler, L., Mortensen, D. R., Schütze, H., & Pierrehumbert, J. B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2423232122 · Cited by 13 · see citing works
de Oliveira, R. S., & Nascimento, E. G. S. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104477 · Cited by 12 · see citing works
Lederman, H., & Mahowald, K. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00690 · Cited by 12 · see citing works
A Systematic Review of Generative Artificial Intelligence in Language Education
Wang, Z., Di, Z. O. U., Lap-Kai Keith, L. E. E., Xie, H., & Wang, F. L. (2023) · International Conference on Computers in Education · doi:10.58459/icce.2023.1199 · Cited by 12 · see citing works
Near-Duplicate Sequence Search at Scale for Large Language Model Memorization Evaluation
Peng, Z., Wang, Z., & Deng, D. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3589324 · Cited by 11 · see citing works
You can’t pick your neighbors, or can you? When and How to Rely on Retrieval in the kNN-LM
Drozdov, A., Wang, S., Rahimi, R., McCallum, A., Zamani, H., & Iyyer, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.218 · Cited by 10 · see citing works
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9 · see citing works
Evaluating n-Gram Novelty of Language Models Using Rusty-DAWG
Merrill, W., Smith, N. A., & Elazar, Y. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.800 · Cited by 8 · see citing works
DP-TabICL: In-Context Learning with Differentially Private Tabular Data
Carey, A. N., Bhaila, K., Edemacu, K., & Wu, X. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826053 · Cited by 8 · see citing works
LongStory: Coherent, Complete and Length Controlled Long Story Generation
Park, K., Yang, N., & Jung, K. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2253-2_15 · Cited by 7 · see citing works
Beyond Individual Concerns: Multi-user Privacy in Large Language Models
Zhan, X., Seymour, W., & Such, J. M. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665883 · Cited by 7 · see citing works
kNN-LM Does Not Improve Open-ended Text Generation
Wang, S., Song, Y., Drozdov, A., Garimella, A., Manjunatha, V., & Iyyer, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.929 · Cited by 6 · see citing works
High-Fidelity Cellular Network Control-Plane Traffic Generation without Domain Knowledge
Kong, Z. J., Hu, N., Hu, Y., Meng, J., & Koral, Y. (2024) · ACM Internet Measurement Conference (IMC) · doi:10.1145/3646547.3688422 · Cited by 5 · see citing works
Text Linguistics and the Use of Linguistic Data in Modern Technologies: Prospects for Development
Aliyeva, G. B. (2023) · Futurity of Social Sciences. · doi:10.57125/fs.2023.06.20.02 · Cited by 5 · see citing works
Creative minds like ours? Large Language Models and the creative aspect of language use
Carchidi, V. J. (2024) · Biolinguistics · doi:10.5964/bioling.13507 · Cited by 5 · see citing works
An Accidental Benchmark: The History, Contingent Power, and Lasting Traces of the GTZAN Dataset
Jerzak, A. (2025) · Digital Society · doi:10.1007/s44206-025-00191-w · Cited by 3 · see citing works
Evidence Against Syntactic Encapsulation in Large Language Models
McGee, T. A., Zhang, Y., & Blank, I. (2026) · Cognitive Science · doi:10.1111/cogs.70187 · Cited by 3 · see citing works
ChatGPT vs Bard: Which is a Better Writer?
Ng, A. L., & Ong, J. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00066 · Cited by 3 · see citing works
Self-repetition in abstractive neural summarizers
Salkar, N. (2022) · doi:10.17760/d20467253 · Cited by 3 · see citing works
Psittacines of Innovation? Assessing the True Novelty of AI CreationsPreprint
Mukherjee, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4764101 · Cited by 3 · see citing works
Casal, J. E., Stewart, C. M., & Windsor, A. J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0324611 · Cited by 2 · see citing works
A Semantics-Aware Head-Driven Approach for Multilingual Dependency Parsing
Nguyễn, T. M. H., & Le-Hong, P. (2025) · IEEE Access · doi:10.1109/access.2025.3601225 · Cited by 2 · see citing works
Detecting Training Data For Large Language Models: A Survey
Yang, C., Li, J., Shulin, L. A. N., Wang, Y., Du, H., Gong, C., Yao, X., Niyato, D. T., & Zhu, L. (2026) · ACM Computing Surveys · doi:10.1145/3779430 · Cited by 2 · see citing works
The compositionality continuum as a principle for studying the neural basis of intelligence
Riveland, R., Pouget, A., & Driscoll, L. N. (2026) · Nature Neuroscience · doi:10.1038/s41593-026-02382-1 · Cited by 2 · see citing works
Fine-Tuning GPT-3 for Synthetic Danish News Generation
Almasi, M., & Schiønning, A. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.4 · Cited by 2 · see citing works
Modelling child comprehension: A case of suffixal passive construction in Korean
Shin, G., & Mun, S. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101701 · Cited by 2 · see citing works
Do Language Models Know Who Did What to Whom?
Denning, J. M., Guo, X., Snefjella, B., & Blank, I. (2026) · Open Mind · doi:10.1162/opmi.a.365 · Cited by 2 · see citing works
Language Models Understand Us, Poorly
Moore, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.16 · Cited by 2 · see citing works
Razeghi, Y., Mekala, R. R., Logan, R. L., Gardner, M., & Singh, S. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.39 · Cited by 2 · see citing works
Moisio, A., Creutz, M., & Kurimo, M. (2023) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2023.genbench-1.17 · Cited by 1 · see citing works
An Efficient Approach for Enhancing GenAI Trustworthiness
Yehezkel, A. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00292 · Cited by 1 · see citing works
Arcenal, E. K. E., Capistrano, L. P. V., Guzman, M. J. D. P. D., Forrosuelo, M. I. M., & Miranda, J. M. (2025) · 3L The Southeast Asian Journal of English Language Studies · doi:10.17576/3l-2025-3101-04 · Cited by 1 · see citing works
FEMA: Emotion-Driven Personification in Generative Agents
Shao, S., Qin, X., Wu, S., Lin, G., Zhang, Y., & Chen, L. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229401 · Cited by 1 · see citing works
Misbah, A. M., Farouk, M., & AbdulAzim, M. (2026) · PLoS ONE · doi:10.1371/journal.pone.0341905 · Cited by 1 · see citing works
Incremental Processing of Principle B: Mismatches Between Neural Models and Humans
Davis, F. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.11 · Cited by 1 · see citing works
EpiK-Eval: Evaluation for Language Models as Epistemic Models
Prato, G., Huang, J., Parthasarathi, P., Sodhani, S., & Chandar, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.593 · Cited by 1 · see citing works
LANG: A Lesson Plan Generation Framework via Multi-Form Interaction with Large Language ModelsPreprint
Ouyang, Y., Quan, J., Wang, H., YaWen, Z., & Chen, L. (2025) · Research Square · doi:10.21203/rs.3.rs-6808103/v1 · Cited by 1 · see citing works
Stepping Stones for Self-Learning
Hai-Jew, S. (2023) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-0074-9.ch005
Zacharopoulos, C.-N., Desbordes, T., & Sablé-Meyer, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.998
Mukherjee, A., & Chang, H. H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5213450
A plug-and-play method for linguistic alignment in language models
Pang, K., Bai, M., Yang, J., Gao, Y., Jiang, M., & Huang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113597
Automatic generation of ESL learning materials based on CEFR levels using reinforcement-tuned LLMs
Zuo, Y. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00762-3
Murthy, J. S., Dhanashekar, K., & Siddesh, G. M. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-96-8799-2_20
Sun, X., Luo, L., & Peng, H. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133397
Ha, O.-S. (2026) · Journal of Digital Contents Society · doi:10.9728/dcs.2026.27.3.821
Data-Based Natural Language Generation : Evaluation and Explainability
Faille, J. (2023) · theses.fr (ABES) · doi:10.70675/3e73dd43z7aa6z4405za5cez27c78b408a3c
AI and the Reduction of Truth: A Eucharistic Alternative
Reilly, C. M. (2026) · Religions · doi:10.3390/rel17050551
Unraveling the Mystery of Artifacts in Machine Generated Text
Pu, J., Ziyi, H., Xi, Y., Chen, G., Chen, W., & Zhang, R. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2tif79eje6ro
Ngo, T. Q., Nguyễn, T. M. H., & Le-Hong, P. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2026.102000
Sripada, C., McInnerney, A., & Lewis, R. L. (2026) · Behavioral and Brain Sciences · doi:10.1017/s0140525x26104725
Huang, K., Liu, C., & Yang, J. (2026) · Journal of Intelligence · doi:10.3390/jintelligence14090218
Николаевич, Ц. А., & Valukhov, I. A. (2026) · Известия Южного федерального университета. Технические науки · doi:10.18522/2311-3103-2026-4-160-169
Chen, Z. (2026) · SHS Web of Conferences · doi:10.1051/shsconf/202624002001
- Authors
- McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A.
- Affiliations
- Princeton University (US)
- Microsoft (United States) (US)
- New York University (US)
- Meta AI, USA. aslic@meta.com
- Corresponding author
- McCoy, R. Thomas
- Smolensky, Paul
- Linzen, Tal
- Gao, Jianfeng
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2023
- Volume
- 11
- Pages
- 652-670
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3213407400
- Cited by
- 70
- References
- 101
- Citations by year
- 2026: 13 · 2025: 16 · 2024: 18 · 2023: 15 · 2022: 8
- FWCI
- 5.49
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1746891)
- Microsoft Research
- Johns Hopkins University
- Maryland Advanced Research Computing Center
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- text generation, ravens, n-gram analysis, syntactic structure, compositional generalization, GPT-2, Transformer-XL, LSTM, semantic coherence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
