The Larger they are, the Harder they Fail: Language Models do not Recognize Identifier Swaps in Python
Open accessAbstract
Large Language Models (LLMs) have successfully been applied to code generation tasks, raising the question of how well these models understand programming.Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers.We show that LLMs not only fail to properly generate correct Python code when default function names are swapped, but some of them even become more confident in their incorrect predictions as the model size increases, an instance of the recently discovered phenomenon of Inverse Scaling, which runs contrary to the commonly observed trend of increasing prediction quality with increasing model size.Our findings indicate that, despite their astonishing typical-case performance, LLMs still lack a deep, abstract understanding of the content they manipulate, making them unsuitable for tasks that statistically deviate from their training data, and that mere scaling is not enough to achieve such capability.* Equal contribution. 1 Since model capacity in number of parameters and pretraining dataset size are balanced according to a design law that is fixed for each model family and is intended to empirically maximize the pretraining set likelihood given a compute budget (Kaplan et al., 2020), for the reminder of this paper we will jointly refer to them as "model size".
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Given a Python prompt which swaps two builtin functions, large language models prefer the incorrect but statistically common continuation to the correct but unusual one.
“Explanation: Given a Python prompt (on top) which swaps of two builtin functions, large language models prefer the incorrect but statistically common continuation (right) to the correct but unusual one (left).”
From Large Language Model Reasoning Failures (2026) · cites this paper as Miceli-Barone et al., 2023
GPT-3 and related models are fragile under identifier swaps in programs, suggesting only a shallow understanding of code.
“Miceli-Barone et al. (2023) show that GPT-3 and related models are fragile under identifier swaps in programs, suggesting only a shallow understanding of code.”
From Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024) · cites this paper as Miceli-Barone et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · see citing works
Group Equivariant Convolutional NetworksPreprint
Cohen, T., & Welling, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07576 · Cited by 991 · see citing works
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · see citing works
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · see citing works
Classes of recursively enumerable sets and their decision problems
Rice, H. G. (1953) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1953-0053041-6 · Cited by 869 · see citing works
Physics-Inspired Structural Representations for Molecules and Materials
Musil, F., Grisafi, A., Bartók, A. P., Ortner, C., Cśanyi, G., & Ceriotti, M. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.1c00021 · Cited by 556 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
Physics-Inspired Structural Representations for Molecules and Materials
Musil, F., Grisafi, A., Bartók, A., Ortner, C., Cśanyi, G., Ceriotti, M., et al. (2021) · Cambridge University Engineering Department Publications Database · Cited by 440 · see citing works
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · see citing works
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · see citing works
BBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186 · see citing works
Barone, A. V. M., & Sennrich, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02275 · Cited by 103 · see citing works
Deac, A., Weber, T., & Papamakarios, G. (2023) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2302.04798 · Cited by 5
BERT Is Not The Count: Learning to Match Mathematical Statements with Proofs
Li, W. W., Ziser, Y., Coavoux, M., & Cohen, S. B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.260 · Cited by 4 · see citing works
Self-citationShortcut Learning in Deep Neural Networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07780
Flagged first, then by citations
Cited by
Sorted by citations
Vulnerability Detection with Code Language Models: How Far are We?
Ding, Y., Fu, Y., Ibrahim, O., Sitawarin, C., Chen, X., Alomair, B., Wagner, D., Ray, B., & Chen, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00038 · Cited by 187 · see citing works
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.102 · Cited by 58 · see citing works
Peykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 57 · see citing works
Deploying and Evaluating LLMs to Program Service Mobile Robots
Hu, Z., Lucchetti, F., Schlesinger, C., Saxena, Y., Freeman, A., Modak, S., Guha, A., & Biswas, J. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3360020 · Cited by 40 · see citing works
Karjus, A. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04503-w · Cited by 68 · see citing works
A Proposal for a Language Model Based Cognitive Architecture
Knowles, K., Witbrock, M., Dobbie, G., & Yogarajan, V. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27691 · Cited by 23 · see citing works
Ojeda-Ramírez, S., Rismanchian, S., & Doroudi, S. (2023) · doi:10.35542/osf.io/64ekv · Cited by 22 · see citing works
Interactive Table Synthesis With Natural Language
Huang, Y., Zhou, Y., Chen, R., Pan, C., Shu, X., Weng, D., & Wu, Y. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3329120 · Cited by 15 · see citing works
Dialogue-based generation of self-driving simulation scenarios using Large Language Models
Barone, A. V. M., Innes, C., & Alex Lascarides, Y. (2023) · doi:10.18653/v1/2023.splurobonlp-1.1 · Cited by 11 · see citing works
An Investigation of LLMs’ Inefficacy in Understanding Converse Relations
Qi, C., Li, B., Hui, B., Wang, B., Li, J., Wu, J., & Laili, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.429 · Cited by 5 · see citing works
Large Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3 · see citing works
Dramko, L., Bölöni-Turgut, D., Goues, C. L., & Schwartz, E. J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97620-9_14 · Cited by 1 · see citing works
On the Relevance of Logic for Artificial Intelligence, and the Promise of Neurosymbolic Learning
Belle, V. (2025) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732251339951 · Cited by 1 · see citing works
CodeMind: Evaluating Large Language Models for Implicit and Explicit Code Execution Reasoning
Liu, C., Chen, Y., & Jabbarvand, R. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3691250 · Cited by 26 · see citing works
- Authors
- Miceli Barone, A. V., Barez, F., Cohen, S. B., & Konstas, I.
- Affiliations
- Heriot-Watt University (GB)
- University of Edinburgh (GB)
- Venue
- Findings of the Association for Computational Linguistics: ACL 2023
- Published
- 2023
- Pages
- 272-292
- OpenAlex
- W4385571985
- Cited by
- 12
- References
- 20
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 2 · 2023: 4
- FWCI
- 3.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- Funders
- UK Research and Innovation (EP/V026607/1)
- University of Birmingham
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, Python code generation, identifier renaming, equivariance, invariance, model size, program comprehension, overconfidence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
