CoChat
Sign Up
Cited by 18Natural Language Processing TechniquesTopic ModelingSyntax, Semantics, Linguistic Variation

Evaluating Morphological Compositional Generalization in Large Language Models

Open access
Ismayilzada, Mete, Circi, Defne, Sälevä, Jonne, Sirin, Hale, Köksal, Abdullatif, Dhingra, Bhuwan, Bosselut, Antoine, Ataman, Duygu +1 more·2025·Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
DOI 10.18653/v1/2025.naacl-long.59
Open this paper in CoChat — free accountOpen PDF

Abstract

Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, Lonneke Van Der Plas. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2025.naacl-long.59

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are especially relevant, but are rarely examined within one unified setting.

    “Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are therefore especially relevant, but are rarely examined within one unified setting [11][12][13][14][15][16][17][18].”

    From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [12]

    The Introduction reports limitations in LLMs’ morphological compositional generalization, and §2.1 describes compositional generalization as challenging. The paper does not address entity binding, long-context evidence use, or numerical grounding, and it does not establish that these four limitations are rarely examined in one unified setting. Page numbers are not available in our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 43

  1. Farrell Ackerman and Robert Malouf. 2013. Morpho- logical organization: The low conditional entropy conjecture. Language, 89:429-464. · DOI
  2. Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nan- ning Zheng, Jian-Guang Lou, and Dongmei Zhang. 2023. How do in-context examples affect compo- sitional generalization? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11027- 11052, Toronto, Canada. Association for Computa- tional Linguistics. · DOI
  3. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chan- dra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaïd Harchaoui, and Yejin Choi. 2023. Faith and fate: Limits of transformers on compo- sitionality. ArXiv, abs/2305.18654. · arXiv
  4. Jerry A. Fodor and Zenon W. Pylyshyn. 1988a. Connec- tionism and cognitive architecture: A critical analysis. Cognition, 28(1):3-71. · DOI · Paper page
  5. Jerry A Fodor and Zenon W Pylyshyn. 1988b. Connec- tionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3-71.
  6. Team Gemini. 2024. Gemini: A family of highly capa- ble multimodal models. Preprint, arXiv:2312.11805. · arXiv
  7. Omer Goldman, Khuyagbaatar Batsuren, Salam Khalifa, Aryaman Arora, Garrett Nicolai, Reut Tsarfaty, and Ekaterina Vylomova. 2023. Sigmorphon-unimorph 2023 shared task 0: Typologically diverse morpho- logical inflection. In Special Interest Group on Com- putational Morphology and Phonology Workshop.
  8. Omer Goldman, David Guriel, and Reut Tsarfaty. 2022. (un)solving morphological inflection: Lemma over- lap artificially inflates models' performance. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Papers), pages 864-870, Dublin, Ireland. Association for Computational Linguistics. · DOI
  9. Emily Goodwin, Koustuv Sinha, and Timothy J. O'Donnell. 2020. Probing linguistic systematicity. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 1958- 1969, Online. Association for Computational Linguis- tics. · DOI
  10. Mika Hämäläinen. 2019. Uralicnlp: An nlp library for uralic languages. Journal of Open Source Software, 4(37):1345. · DOI
  11. Charles F Hockett. 1954. Two models of grammatical description. Word, 10(2-3):210-234.
  12. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Deriva- tional morphology improves BERT's interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3594-3608, Online. Association for Computational Linguistics. · DOI
  13. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2019. Measur- ing compositional generalization: A comprehensive method on realistic data. ArXiv, abs/1912.09713. · arXiv
  14. Najoung Kim and Tal Linzen. 2020. COGS: A compo- sitional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105, Online. As- sociation for Computational Linguistics. · DOI · Paper page
  15. Paul Kiparsky. 1982b. Word formation and the lexi- con. In 1982 Mid America Linguistics Conference Papers/Dept. of Ling., Univ. of Kansas.
  16. Jordan Kodner and Salam Khalifa. 2022. Sigmor- phon-unimorph 2022 shared task 0: Modeling in- flection in language acquisition. Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology.
  17. Brenden M. Lake and Marco Baroni. 2018. General- ization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. Preprint, arXiv:1711.00350. · arXiv
  18. Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In Annual Meeting of the Association for Computational Linguistics.
  19. Risto Luukkonen, Jonathan Burdge, Elaine Zosa, Aarne Talman, Ville Komulainen, Väinö Hatanpää, Pe- ter Sarlin, and Sampo Pyysalo. 2024. Poro 34b and the blessing of multilinguality. Preprint, arXiv:2404.01856. · arXiv
  20. R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jian- feng Gao, and Asli Celikyilmaz. 2023. How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN. Transactions of the Association for Compu- tational Linguistics, 11:652-670. · DOI · Paper page
  21. Francois Meyer and Jan Buys. 2023. Subword segmen- tal machine translation: Unifying segmentation and target sentence generation. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 2795-2809, Toronto, Canada. Association for Computational Linguistics. · DOI
  22. Timothy J O'Donnell. 2015. Productivity and reuse in language: A theory of linguistic computation and storage. MIT Press.
  23. OpenAI. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. · arXiv
  24. Adnan Ozturel, Tolga Kayadelen, and Demirsahin I. 2019. A syntactically expressive morphological an- alyzer for turkish. In Proceedings of the 14th In- ternational Conference on Finite-State Methods and Natural Language Processing, pages 65-75, Dresden, Germany. Association for Computational Linguistics.
  25. Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions. Preprint, arXiv:2308.11483. · arXiv
  26. Tommi A Pirinen. 2015. Development and use of com- putational morphology of finnish in the open source and open science era: Notes on experiences with omorfi development. SKY Journal of Linguistics, 28:381-393.
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics. · DOI · Paper page
  28. Shane Storks and Joyce Chai. 2021. Beyond the tip of the iceberg: Assessing coherence of text classifiers. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3169-3177, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. · DOI
  29. Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinuç, and Oguzhan Ozcelik. 2022. Impact of tokenization on language models: An analysis for turkish. ACM Transactions on Asian and Low-Resource Language Information Processing, 22:1 -21.
  30. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971. · arXiv
  31. Ekaterina Vylomova, Jennifer White, Elizabeth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrod, Josef Valvoda, Svetlana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky, Paula Czarnowska, Irene Nikkarinen, Andrew Krizhanovsky, Tiago Pimentel, Lucas Torroba Henni- gen, Christo Kirov, Garrett Nicolai, Adina Williams, Antonios Anastasopoulos, Hilaria Cruz, Eleanor Chodroff, Ryan Cotterell, Miikka Silfverberg, and Mans Hulden. 2020. SIGMORPHON 2020 shared task 0: Typologically diverse morphological inflec- tion. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 1-39, Online. Association for Computational Linguistics. · DOI
  32. Jiongxiao Wang, Zichen Liu, Keun Hee Park, Zhuo- jun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, and Chaowei Xiao. 2023. Adversarial demon- stration attacks on large language models. Preprint, arXiv:2305.14950. · arXiv
  33. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. Preprint, arXiv:2206.07682. · arXiv
  34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models. Preprint, arXiv:2201.11903. · arXiv
  35. Leonie Weissweiler, Valentin Hofmann, Anjali Kan- tharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vi- jayakumar, Haofei Yu, Hinrich Schuetze, Kemal Oflazer, and David Mortensen. 2023. Counting the bugs in ChatGPT's wugs: A multilingual investiga- tion into the morphological capabilities of a large language model. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6508-6524, Singapore. Associa- tion for Computational Linguistics. · DOI
  36. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366-15394, Bangkok, Thai- land. Association for Computational Linguistics. · DOI
  37. Michael Wilson, Jackson Petty, and Robert Frank. 2023. How abstract is linguistic generalization in large lan- guage models? experiments with argument structure. Transactions of the Association for Computational Linguistics, 11:1377-1395. · DOI
  38. Katherine Wysocki and Joseph R. Jenkins. 1987. Deriv- ing word meanings through morphological general- ization. Reading Research Quarterly, 22(1):66-81.
  39. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics. · DOI · Paper page
  40. Haoran Yang, Hongyuan Lu, Wai Lam, and Deng Cai. 2024. Exploring compositional generalization of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop), pages 16-24.
  41. Ruochen Zhang, Samuel Cahyawijaya, Jan Chris- tian Blaise Cruz, Genta Winata, and Alham Aji. 2023. Multilingual large language models are not (yet) code-switchers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 12567-12582, Singapore. Association for Computational Linguistics. · DOI
  42. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Im- proving few-shot performance of language models. Preprint, arXiv:2102.09690. · arXiv
  43. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. Preprint, arXiv:2303.18223. · arXiv

Cited by

Sorted by citations

Authors
Ismayilzada, M., Circi, D., Sälevä, J., Sirin, H., Köksal, A., Dhingra, B., Bosselut, A., Ataman, D., & Plas, L. V. D.
Affiliations
  • École Polytechnique Fédérale de Lausanne (CH)
  • Brandeis University (US)
  • University of Cambridge (GB)
  • Duke University (US)
  • New York University (US)
  • Università della Svizzera italiana (CH)
ORCID
Venue
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2025
Pages
1270-1305
DOI
10.18653/v1/2025.naacl-long.59
OpenAlex
W4411119511
Cited by
18
References
0
Citations by year
2026: 8 · 2025: 10
FWCI
12.64
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2022040)
  • École Polytechnique Fédérale de Lausanne
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (207437)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
Keywords
large language models, morphological composition, morphological analysis, morphological generation, compositional generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review