SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Open accessAbstract
Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models.However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models.We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design.This paper releases two tools of independent value for the computational linguistics community:1.A website, syntaxgym.org,which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization; 2. Two command-line tools, syntaxgym and lm-zoo, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
BLiMP and other related syntactic benchmarks such as those presented in Marvin and Linzen (2018) and Gauthier et al. (2020) were specifically designed to mimic the types of tests invented by linguists and cognitive scientists to reveal syntactic competence, i.e., they are all based on minimal pair sentences.
“BLiMP, and other related syntactic benchmarks such as those presented in Marvin and Linzen (2018) and Gauthier et al. (2020), were specifically designed to mimic the types of tests invented by linguists and cognitive scientists to reveal syntactic competence-i.e., they are all based on minimal pair sentences.”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Gauthier et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · see citing works
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Emmanuel Dupoux, A., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 776 · see citing works
Recurrent Neural Network Grammars
Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1024 · Cited by 517 · see citing works
Colorless Green Recurrent Networks Dream Hierarchically
Gulordava, K., Bojanowski, P., Grave, É., Linzen, T., & Baroni, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1108 · Cited by 489 · see citing works
Targeted Syntactic Evaluation of Language Models
Marvin, R., & Linzen, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1151 · Cited by 334 · see citing works
Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge
Lau, J. H., Clark, A., & Lappin, S. (2016) · Cognitive Science · doi:10.1111/cogs.12414 · Cited by 247 · see citing works
Ordered Neurons: Integrating Tree Structures into Recurrent Neural NetworksPreprint
Shen, Y., Tan, S., Sordoni, A., & Courville, A. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09536 · Cited by 222 · see citing works
Neural language models as psycholinguistic subjects: Representations of syntactic state
Futrell, R., Wilcox, E., Morita, T., Qian, P., Ballesteros, M., & Lévy, R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1004 · Cited by 154 · see citing works
Self-citationA Systematic Assessment of Syntactic Generalization in Neural Language Models
Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.158 · Cited by 134 · see citing works
Self-citationQuantity doesn’t buy quality syntax with neural language models
van Schijndel, M., Mueller, A., & Linzen, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1592 · Cited by 65 · see citing works
BLiMP: the benchmark of linguistic minimal pairs for English
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28 · see citing works
Flagged first, then by citations
3 of the 17 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Cited by
Sorted by citations
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420 · see citing works
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00517 · Cited by 240 · see citing works
What Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Language Models as Agent Models
Andreas, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.423 · Cited by 92 · see citing works
Sinha, K., Parthasarathi, P., Pineau, J., & Williams, A. (2021) · doi:10.18653/v1/2021.acl-long.569 · Cited by 76 · see citing works
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works
“…to such concerns, but we also note that BLiMP, and other related syntactic benchmarks such as those presented in Marvin and Linzen (2018) and Gauthier et al. (2020), were specifically designed to mimic the types of tests invented by linguists and cognitive scientists to reveal syntactic…”
Self-citationLanguage in Brains, Minds, and Machines
Tuckute, G., Kanwisher, N., & Fedorenko, E. G. (2024) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-120623-101142 · Cited by 69 · see citing works
Prompting is not a substitute for probability measurements in large language models
Hu, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.306 · Cited by 60 · see citing works
Self-citationNL-Augmenter 🦎 → 🐍 A Framework for Task-Sensitive Natural Language Augmentation
Dhole, K., Gangal, V., Gehrmann, S., Gupta, A., Li, Z., Mahamood, S., Mahadiran, A., Mille, S., Shrivastava, A., Tan, S., et al. (2023) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2023.4725 · Cited by 57 · see citing works
Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely
Kauf, C., Ivanova, A. A., Rambelli, G., Chersoni, E., She, J. S., Chowdhury, Z., Fedorenko, E. G., & Lenci, A. (2023) · Cognitive Science · doi:10.1111/cogs.13386 · Cited by 56 · see citing works
(2021) · doi:10.18653/v1/2021.conll-1.15 · Cited by 51 · see citing works
Sinclair, A., Jumelet, J., Zuidema, W. H., & Fernández, R. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00504 · Cited by 51 · see citing works
A-maze of Natural Stories: Comprehension and surprisal in the Maze task
Boyce, V., & Lévy, R. (2023) · Glossa Psycholinguistics · doi:10.5070/g6011190 · Cited by 33 · see citing works
MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
Jumelet, J., Weissweiler, L., Nivre, J., & Bisazza, A. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.600 · Cited by 32 · see citing works
How much pretraining data do language models need to learn syntax?
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.118 · Cited by 26 · see citing works
Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction
Eisape, T., Zaslavsky, N., & Lévy, R. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.49 · Cited by 24 · see citing works
A Better Way to Do Masked Language Model Scoring
Kauf, C., & Ivanova, A. A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.80 · Cited by 23 · see citing works
The Low-Dimensional Linear Geometry of Contextualized Word Representations
Hernandez, E. J., & Andreas, J. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.7 · Cited by 21 · see citing works
Expectations over Unspoken Alternatives Predict Pragmatic Inferences
Hu, J. J., Lévy, R., Degen, J., & Schuster, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00579 · Cited by 20 · see citing works
Self-citationEffective Batching for Recurrent Neural Network Grammars
Noji, H., & Oseki, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.380 · Cited by 18 · see citing works
Schrödinger's tree—On syntax and neural language models
Kulmizev, A., & Nivre, J. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.796788 · Cited by 15 · see citing works
Mueller, A., & Linzen, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.629 · Cited by 14 · see citing works
Sensitivity as a Complexity Measure for Sequence Classification Tasks
Hahn, M., Jurafsky, D., & Futrell, R. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00403 · Cited by 14 · see citing works
Automatic Construction of Evaluation Suites for Natural Language Generation DatasetsPreprint
Mille, S., Dhole, K., Mahamood, S., Perez-Beltrachini, L., Gangal, V., Kale, M., van Miltenburg, E., & Gehrmann, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09069 · Cited by 13 · see citing works
Evaluating Neural Language Models as Cognitive Models of Language Acquisition
Martínez, H. J. V., Heuser, A., Yang, C. D., & Kodner, J. (2023) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2023.genbench-1.4 · Cited by 9 · see citing works
Filtered Corpus Training (FiCT) Shows that Language Models Can Generalize from Indirect Evidence
Patil, A., Jumelet, J., Chiu, Y. Y., Lapastora, A., Shen, P. S., Wang, L., Willrich, C., & Steinert‐Threlkeld, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00720 · Cited by 8 · see citing works
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models
Beyer, A., Loáiciga, S., & Schlangen, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.328 · Cited by 8 · see citing works
Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
Taktasheva, E., Mikhailov, V., & Artemova, E. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.17 · Cited by 7 · see citing works
Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number
Hao, S., & Linzen, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.300 · Cited by 7 · see citing works
Visualizing and Explaining Language Models
Brașoveanu, A., & Andonie, R. (2022) · Studies in computational intelligence · doi:10.1007/978-3-030-93119-3_8 · Cited by 7 · see citing works
Jouitteau, M. (2023) · Lapurdum · doi:10.4000/127u1 · Cited by 5 · see citing works
Rambelli, G., Chersoni, E., Senaldi, M. S. G., Blache, P., & Lenci, A. (2023) · Workshop on Multiword Expressions (MWE) · doi:10.18653/v1/2023.mwe-1.13 · Cited by 5 · see citing works
Evaluating the Morphosyntactic Well-formedness of Generated Texts
Pratapa, A., Anastasopoulos, A., Rijhwani, S., Chaudhary, A., Mortensen, D. R., Neubig, G., & Tsvetkov, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.570 · Cited by 5 · see citing works
Learning Hierarchical Structures with Differentiable Nondeterministic StacksPreprint
DuSell, B., & Chiang, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.01982 · Cited by 5 · see citing works
How much pretraining data do language models need to learn syntax?Preprint
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03160 · Cited by 5 · see citing works
Natural Language Processing RELIES on Linguistics
Opitz, J., Wein, S., & Schneider, N. (2025) · Computational Linguistics · doi:10.1162/coli_a_00560 · Cited by 4 · see citing works
Challenges and Opportunities in Text Generation Explainability
Amara, K., Sevastjanova, R., & El‐Assady, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63787-2_13 · Cited by 4 · see citing works
Ivanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., Kauf, C., Hu, J., Pramod, R. T., Grand, G., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.38 · Cited by 4 · see citing works
Self-citationExplaining Dialogue Evaluation Metrics using Adversarial Behavioral Analysis
Khalid, B., & Lee, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.430 · Cited by 4 · see citing works
Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models
Pérez-Mayos, L., García, A. T., Mille, S., & Wanner, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.333 · Cited by 4 · see citing works
de-Dios-Flores, I., Amboage, J. P. G., & García, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.12 · Cited by 3 · see citing works
Floyd, S., Jouravlev, O., Mineroff, Z., Poliak, M., Gibson, E. A. F., & Fedorenko, E. G. (2023) · doi:10.31234/osf.io/e2xta · Cited by 3 · see citing works
BLiMP-NL: A corpus of Dutch minimal pairs and acceptability judgements for language model evaluationPreprint
Suijkerbuijk, M., Prins, Z., Kloots, M. D. H., Zuidema, W. H., & Frank, S. L. (2024) · doi:10.31234/osf.io/mhjbx · Cited by 3 · see citing works
Ivanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., Kauf, C., Hu, J., Pramod, R. T., Grand, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.09605 · Cited by 2
“The minimal pair design is common in datasets inspired by psycholinguistics and cognitive science, such as SyntaxGym (Gauthier et al., 2020), BLiMP (Warstadt et al., 2020), and COMPS (Misra et al., 2023).”
Self-citationHow Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction
Bau, D. A., & Andreas, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.448 · Cited by 2 · see citing works
Vyākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages
Patil, R., Dhillon, J., Mahurkar, S., Kulkarni, S., Malhotra, M., & Baths, V. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.14 · Cited by 1 · see citing works
HILITE: Human-in-the-loop Interactive Tool for Image Editing
Pasumarthi, A., Sharma, A., Patel, J. H., Bheemaiah, A., Vadlamannati, S., Chang, S., Li, S., Barkataki, E., Zhang, Y., Yang, D., et al. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825916 · Cited by 1 · see citing works
Catasso, N. (2026) · AI-Linguistica Linguistic Studies on AI-Generated Texts and Discourses · doi:10.62408/ai-ling.v3i1.88 · Cited by 1 · see citing works
Smith, G., & Vasishth, S. (2021) · doi:10.31234/osf.io/dtazq · Cited by 1 · see citing works
Bau, A., & Andreas, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03108
Evaluating the Morphosyntactic Well-formedness of Generated TextsPreprint
Pratapa, A., Anastasopoulos, A., Rijhwani, S., Chaudhary, A., Mortensen, D. R., Neubig, G., & Tsvetkov, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.16590
Composition, Attention, or Both?
Yoshida, R., & Oseki, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.428
How Much Syntactic Supervision is “Good Enough”?
Noji, H., & Oseki, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.173
Beinborn, L., & Hollenstein, N. (2023) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-43260-6_4
On the utility of enhancing BERT syntactic bias with Token Reordering Pretraining
Mesbahi, Y. E., Mahmud, A., Ghaddar, A., Rezagholizadeh, M., Langlais, P., & Parthasarathi, P. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.12
Parallel Trees: a novel resource with aligned dependency and constituency syntactic representations
Alzetta, C., Miaschi, A., Dell’Orletta⋄, F., Venturi⋄, G., & Montemagni⋄, S. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09826-3
You can’t be neutral on a moving train
Heinz, J. N., & Rawski, J. (2026) · Behavioral and Brain Sciences · doi:10.1017/s0140525x26104798
Three Lenses on Linguistic Knowledge in Pretrained Transformers: A Phenomenon-Centered ReviewPreprint
Wang, Y., & Ling, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-10776566/v1
- Authors
- Gauthier, J., Hu, J., Wilcox, E., Qian, P., & Levy, R.
- Affiliations
- Massachusetts Institute of Technology (US)
- Harvard University (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
- Published
- 2020
- Pages
- 70-76
- OpenAlex
- W3037115370
- Cited by
- 61
- References
- 17
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 7 · 2023: 18 · 2022: 8 · 2021: 20 · 2020: 2
- FWCI
- 4.58
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) (publishedVersion, cc-by)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by)
- Funders
- National Science Foundation
- Open Philanthropy Project
- National Institutes of Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- language models, psycholinguistic experiment, digital platforms, reproducibility, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
