Syntax and sensibility: Using language models to detect and correct syntax errors
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Syntax errors are made by novice and experienced programmers alike; however, novice programmers lack the years of experience that help them quickly resolve these frustrating errors. Standard LR parsers are of little help, typically resolving syntax errors and their precise location poorly. We propose a methodology that locates where syntax errors occur, and suggests possible changes to the token stream that can fix the error identified. This methodology finds syntax errors by using language models trained on correct source code to find tokens that seem out of place. Fixes are synthesized by consulting the language models to determine what tokens are more likely at the estimated error location. We compare n-gram and LSTM (long short-term memory) language models for this task, each trained on a large corpus of Java code collected from GitHub. Unlike prior work, our methodology does not rely that the problem source code comes from the same domain as the training data. We evaluated against a repository of real student mistakes. Our tools are able to find a syntactically-valid fix within its top-2 suggestions, often producing the exact fix that the student used to resolve the error. The results show that this tool and methodology can locate and suggest corrections for syntax errors. Our methodology is of practical use to all programmers, but will be especially useful to novices frustrated with incomprehensible syntax errors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555
DeepFix: Fixing Common C Language Errors by Deep Learning
Gupta, R., Pal, S., Kanade, A., & Shevade, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10742 · Cited by 426
Are deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., & Devanbu, P. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106290 · Cited by 292 · cited 2×
“Hellendoorn and Devanbu [39] discuss the effectiveness of deep learning (specifically RNNs and LSTMs) on the task of suggesting the next token in a file.”
BackgroundTu, Z., Su, Z., & Devanbu, P. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635875 · Cited by 238 · cited 1×
“Localness [54], online ngram models, or other search-based models might be feasible candidates to help resolve identifiers.”
BackgroundAll syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144 · cited 1×
“[21] categorized common syntax errors that novices make by the error messages the compiler generates and how long it takes for the programmer to fix them.”
BackgroundInvestigating novice programming mistakes: educator beliefs vs. student data
Brown, N. C. C., & Altadmri, A. (2014) · Proceedings of the tenth annual conference on International computing education research · doi:10.1145/2632320.2632343 · Cited by 89 · cited 3×
“[5] investigated the opinions of educators regarding what they believe are the most common Java programming mistakes made by novices, and contrasted it with mistakes mined from the Blackbox dataset.”
BackgroundMethod
Cited by
SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S. J., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 396
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199
Big code != big vocabulary: open-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., & Janes, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380342 · Cited by 182
“[18] used n-gram language models to detect syntax error locations in Java code, and later used an NLM for the same purpose [78].”
BackgroundMethodSynShine: Improved Fixing of Syntax Errors
Ahmed, T., Ledesma, N. R., & Devanbu, P. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3212635 · Cited by 38
“Using the Blackbox [11] dataset, they demonstrated that their approach substantially improved over prior work on the same dataset [8] (which used language models).”
MethodResultInfluential
Metadata
- Authors
- Santos, E. A., Campbell, J. C., Patel, D., Hindle, A., & Amaral, J. N.
- Venue
- 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)
- Publisher
- IEEE
- Published
- 2018-03
- Pages
- 311-322
- OpenAlex
- W2795753518
- Cited by
- 83
- References
- 7
- FWCI
- 7.37
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Syntax and sensibility: Using language models to detect and correct syntax errors. https://app.cochat.ai/papers/10.1109/saner.2018.8330219
- The paper
- Santos, E. A., Campbell, J. C., Patel, D., Hindle, A., & Amaral, J. N. (2018). Syntax and sensibility: Using language models to detect and correct syntax errors. 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). https://doi.org/10.1109/saner.2018.8330219
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.