CoChat
Sign Up
PreprintCited by 1Topic ModelingNatural Language Processing TechniquesAdversarial Robustness in Machine Learning

Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models

Open access
Shaib, Chantal, Suriyakumar, Vinith M., Sagun, Levent, Wallace, Byron C., Ghassemi, Marzyeh·2025·arXiv
DOI 10.48550/arxiv.2509.21155arXiv 2509.21155
Open this paper in CoChat — free accountOpen PDF

Abstract

For an LLM to correctly respond to an instruction it must understand both the semantics and the domain (i.e., subject area) of a given task-instruction pair. However, syntax can also convey implicit information Recent work shows that syntactic templates -- frequent sequences of Part-of-Speech (PoS) tags -- are prevalent in training data and often appear in model outputs. In this work we characterize syntactic templates, domain, and semantics in task-instruction pairs. We identify cases of spurious correlations between syntax and domain, where models learn to associate a domain with syntax during training; this can sometimes override prompt semantics. Using a synthetic training dataset, we find that the syntactic-domain correlation can lower performance (mean 0.51 +/- 0.06) on entity knowledge tasks in OLMo-2 models (1B-13B). We introduce an evaluation framework to detect this phenomenon in trained models, and show that it occurs on a subset of the FlanV2 dataset in open (OLMo-2-7B; Llama-4-Maverick), and closed (GPT-4o) models. Finally, we present a case study on the implications for safety finetuning, showing that unintended syntactic-domain correlations can be used to bypass refusals in OLMo-2-7B Instruct and GPT-4o. Our findings highlight two needs: (1) to explicitly test for syntactic-domain correlations, and (2) to ensure syntactic diversity in training data, specifically within domains, to prevent such spurious correlations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2509.21155

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLM text exhibits systematic lexical and syntactic patterns.

    “LLM text exhibits systematic lexical and syntactic patterns (Shaib et al., 2026; Kobak et al., 2025; Shaib et al., 2025)”

    From StoryScope: Investigating idiosyncrasies in AI fiction (2026) · cites this paper as Shaib et al., 2025

    The Abstract (p. 1) says syntactic templates often appear in model outputs, supporting the claim about syntactic patterns in LLM text. Related Work (p. 9) mentions lexical patterns as features exploited by NLP models, but does not establish systematic lexical patterns in LLM-generated text. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 34

  1. [1][1] E. Akyurek and J. Andreas. LexSym: Compositionality as lexical symmetry. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 639-657, Toronto, Canada, July 2023. Association for Computational Linguistics. -
  2. [2][2] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019. - · arXiv
  3. [3][3] D. Arora, H. Singh, et al. Have llms advanced enough? a challenging problem solving benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7527-7543, 2023. -
  4. [4][4] H. Bahng, S. Chun, Y. Yoo, J. Choo, and S. J. H. Oh. Learning de-biased representations with biased representations. In ICML , 2020. -
  5. [5][5] B. Cao, D. Cai, Z. Zhang, Y. Zou, and W. Lam. On the worst prompt performance of large language models. arXiv preprint arXiv:2406.10248 , 2024. - · arXiv
  6. [7][7] N. Chomsky. Aspects of the Theory of Syntax . Number 11. MIT press, 2014. -
  7. [8][8] E. Creager, J.-H. Jacobsen, and R. Zemel. Environment inference for invariant learning. In International Conference on Machine Learning , pages 2189-2200. PMLR, 2021. -
  8. [9][9] H. Elsahar, P. Vougiouklis, A. Remaci, C. Gravier, J. Hare, F. Laforest, and E. Simperl. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , 2018. -
  9. [10][10] L. Fang, X. Yu, J. Cai, Y . Chen, S. Wu, Z. Liu, Z. Yang, H. Lu, X. Gong, Y . Liu, et al. Knowledge distillation and dataset distillation of large language models: Emerging trends, challenges, and future directions. arXiv preprint arXiv:2504.14772 , 2025. - · arXiv
  10. [11][11] R. Geirhos, P. Rubisch, C. Michaelis, et al. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR , 2019. -
  11. [12][12] A. E. Goldberg. Usage-based constructionist approaches and large language models. Constructions and Frames , 16(2):220-254, 2024. -
  12. [13][13] S. Gururangan, A. Marasovi´ c, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) , pages 8342-8360. Association for Computational Linguistics, 2020. -
  13. [14][14] S. Gururangan, S. Swayamdipta, O. Levy, et al. Annotation artifacts in natural language inference data. In NAACL , 2018. -
  14. [15][15] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization . In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8320-8329, Los Alamitos, CA, USA, Oct. 2021. IEEE Computer Society. -
  15. [16][16] P. Izmailov, P. Kirichenko, N. Gruver, and A. G. Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems , 35:38516-38532, 2022. -
  16. [17][17] B. Jha and U. Puri. The rosetta paradox: Domain-specific performance inversions in large language models. arXiv preprint arXiv:2412.17821 , 2024. - · arXiv
  17. [19][19] D. Ju, H. Blix, and A. Williams. Domain regeneration: How well do llms match syntactic properties of text domains? arXiv preprint arXiv:2505.07784 , 2025. - · arXiv
  18. [20][20] A. Kim, M. Muhn, and V. Nikolaev. Financial statement analysis with large language models. arXiv preprint arXiv:2407.17866 , 2024. - · arXiv
  19. [22][22] H. Li, M. Wang, S. Liu, and P.-Y. Chen. A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. arXiv preprint arXiv:2302.06015 , 2023. - · arXiv
  20. [23][23] C. Ling, X. Zhao, J. Lu, C. Deng, C. Zheng, J. Wang, T. Chowdhury, Y. Li, H. Cui, X. Zhang, et al. Domain specialization as the key to make large language models disruptive: A comprehensive survey. ACM Computing Surveys , 58(3):1-39, 2025. -
  21. [24][24] T. Litaina, A. Soularidis, G. Bouchouras, K. Kotis, and E. Kavakli. Towards llm-based semantic analysis of historical legal documents. In SemDH2024: First International Workshop of Semantic Digital Humanities, co-located with ESWC2024 , 2024. -
  22. [25][25] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning , pages 22631-22648. PMLR, 2023. -
  23. [26][26] R. T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In A. Korhonen, D. Traum, and L. Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3428-3448, Florence, Italy, July 2019. Association for Computational Linguistics. -
  24. [27][27] R. T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in nli models. In ACL , 2019. -
  25. [28][28] V. Moskvoretskii, A. Panchenko, and I. Nikishina. Are large language models good at lexical semantics? a case of taxonomy learning. arXiv preprint arXiv:2401.00000 , 2024. - · arXiv
  26. [29][29] K. Nikoli´ c, L. Sun, J. Zhang, and F. Tramèr. The jailbreak tax: How useful are your jailbreak outputs? arXiv preprint arXiv:2504.10694 , 2025. - · arXiv
  27. [30][30] T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656 , 2024. - · arXiv
  28. [33][33] X. Qi, A. Panda, K. Lyu, X. Ma, S. Roy, A. Beirami, P. Mittal, and P. Henderson. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946 , 2024. - · arXiv
  29. [34][34] S. Sagawa, P. W. Koh, T. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR , 2020. -
  30. [35][35] C. Shaib, Y. Elazar, J. J. Li, and B. C. Wallace. Detection and measurement of syntactic templates in generated text. arXiv preprint arXiv:2407.00211 , 2024. - · arXiv
  31. [36][36] L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159 , 2024. - · arXiv
  32. [37][37] J. Sun, C. Shaib, and B. C. Wallace. Evaluating the zero-shot robustness of instruction-tuned language models. arXiv preprint arXiv:2306.11270 , 2023. - · arXiv
  33. [38][38] M. Sun, Y. Yin, Z. Xu, J. Z. Kolter, and Z. Liu. Idiosyncrasies in large language models. arXiv preprint arXiv:2502.12150 , 2025. - · arXiv
  34. [40][40] A. Webson and E. Pavlick. Do prompt-based models really understand the meaning of their prompts? In ACL , 2022. -

Cited by

Sorted by citations

  • StoryScope: Investigating idiosyncrasies in AI fiction

    Russell, J., Rajendhran, R., Pham, C. M., Iyyer, M., & Wieting, J. (2026) · arXiv · doi:10.48550/arxiv.2604.03136

    “LLM text exhibits systematic lexical and syntactic patterns (Shaib et al., 2026; Kobak et al., 2025; Shaib et al., 2025), and parallel-corpus studies show LLMs converge on a narrow grammatical and rhetorical style distinct from human variation (Reinhart et al., 2025).”

Authors
Shaib, C., Suriyakumar, V. M., Sagun, L., Wallace, B. C., & Ghassemi, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2509.21155
arXiv
2509.21155
OpenAlex
W7083439586
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
spurious correlations, part-of-speech tags, large language models, GPT-4o

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review