Toward a Theory of Causation for Interpreting Neural Code Models
Open accessThe DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index.
Abstract
Neural Language Models of Code, or Neural Code Models (NCMs), are rapidly progressing from research prototypes to commercial developer tools. As such, understanding the capabilities and limitations of such models is becoming critical. However, the abilities of these models are typically measured using automated metrics that often only reveal a portion of their real-world performance. While, in general, the performance of NCMs appears promising, currently much is unknown about how such models arrive at decisions. To this end, this paper introducesdocode, a post hoc interpretability method specific to NCMs that is capable of explaining model predictions.docodeis based upon causal inference to enable programming language-oriented explanations. While the theoretical underpinnings ofdocodeare extensible to exploring different model properties, we provide a concrete instantiation that aims to mitigate the impact of spurious correlations by grounding explanations of model behavior in properties of programming languages. To demonstrate the practical benefit ofdocode, we illustrate the insights that our framework can provide by performing a case study on two popular deep learning architectures and ten NCMs. The results of this case study illustrate that our studied NCMs are sensitive to changes in code syntax. All our NCMs, except for the BERT-like model, statistically learn to predict tokens related to blocks of code (e.g., brackets, parenthesis, semicolon) with less confounding bias as compared to other programming language constructs. These insights demonstrate the potential ofdocodeas a useful method to detect and facilitate the elimination of confounding bias in NCMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 1 Common Cause Principle in NCMs [28]](https://figures.cochat.ai/paper-figures/a8530aee695a4063a8a95f591410b2e1/toward-a-theory-of-causation-for-interpreting-neural-code-models-figure-1.png)



















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recent studies indicate that causal interpretability [135] can successfully reduce misleading interpretations by mitigating the influence of confounders.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [135]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“…Recurrent Unit (GRU) [81], which uses reset r t = σ ( W r · [ h t − 1 , w t ]) , and update u t = σ ( W u · [ h t − 1 , w t ]) gates to control how it combines the previous hidden state h t − 1 with a new candidate state Furthermore, GPT and BERT are deep architectures known as Transformers [33].”
BackgroundMethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“IX) on different variations of popular deep learning architectures for the task of code generation, namely RNNs [32] and Transformers [33] trained on the CodeSearchNet dataset [34].”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“Causation Correlation NCM lyr,unt NCM lyr,hds dropout RNN [42] dropout TF optimizer [43] learning rate beta1|beta2 epsilon epochs batch RNN|TG 0.”
BackgroundExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · cited 1×
“For model development, we employed Tensorflow and Pytorch [84], [85], along with Huggingface’s Transformers library [86].”
MethodTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099 · cited 1×
“RNNs can take the form of a Gated Recurrent Unit (GRU) [81], which uses reset r t = σ ( W r · [ h t − 1 , w t ]) , and update u t = σ ( W u · [ h t − 1 , w t ]) gates to control how it combines the previous hidden state h t − 1 with a new candidate state Furthermore, GPT and BERT are deep…”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“Restrictions apply. and thus may inherit the various limitations often associated with such models — including biases, memorization, and issues with data inefficiency, to name a few [22].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 2×
“However, perhaps the most problematic aspect of current neural language models is the fact that they are incapable of explaining their reasoning [23], [24], [25], [26].”
BackgroundThe Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422 · cited 2×
“However, perhaps the most problematic aspect of current neural language models is the fact that they are incapable of explaining their reasoning [23], [24], [25], [26].”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,486
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 1×
“However, perhaps the most problematic aspect of current neural language models is the fact that they are incapable of explaining their reasoning [23], [24], [25], [26].”
BackgroundCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242 · cited 1×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · cited 1×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodCausal Inference in Statistics
Pearl, J., Glymour, M., & Jewell, N. P. (2016) · Cited by 2,024
The Book of Why: The New Science of Cause and Effect
Goldberg, L. R. (2019) · Notices of the American Mathematical Society · doi:10.1090/noti1912 · Cited by 1,655
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 3×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
BackgroundMethodBERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · cited 1×
“…various techniques, which can be classified into two basic groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis…”
MethodKim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · cited 1×
“A major conjecture in interpretability research is that NCMs are more understandable when they reflect human knowledge [92].”
BackgroundData Mining Static Code Attributes to Learn Defect Predictors
Menzies, T., Greenwald, J., & Frank, A. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.256941 · Cited by 1,215 · cited 1×
“For all SCMs proposed in this study, we assumed that confounders are SE quality metrics since they have the potential to influence models’ code predictions, i.e., a NCM may be influenced by more or less for loops, as well as influence interventions, i.e., more code is correlated with more bugs [77].”
On the Properties of Neural Machine Translation: Encoder-Decoder ApproachesPreprint
Cho, K., van Merriënboer, B., Bahdanau, D., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.1259 · Cited by 1,112 · cited 1×
“RNNs can take the form of a Gated Recurrent Unit (GRU) [81], which uses reset r t = σ ( W r · [ h t − 1 , w t ]) , and update u t = σ ( W u · [ h t − 1 , w t ]) gates to control how it combines the previous hidden state h t − 1 with a new candidate state Furthermore, GPT and BERT are deep…”
BackgroundNeural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · cited 1×
“For all scenarios, Byte Pair Encoding (BPE) tokenization [91] was applied to the testbeds before processing them with the NCMs.”
MethodLearning to Represent Programs with GraphsPreprint
Allamanis, M., Brockschmidt, M., & Khademi, M. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1711.00740 · Cited by 966 · cited 1×
“Researchers have also investigated various representations of NCMs for code [50] as well as graph-based representations based on ASTs [51] and program dependency graphs [52].”
BackgroundBeyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
Visualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845 · cited 2×
“were among the first to interpret NLMs for code using general Next Token Predictions [74] as an”
MethodEvaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · cited 3×
“However, perhaps the most problematic aspect of current neural language models is the fact that they are incapable of explaining their reasoning [23], [24], [25], [26].”
BackgroundThe seven tools of causal inference, with reflections on machine learning
Pearl, J. (2019) · Communications of the ACM · doi:10.1145/3241036 · Cited by 659 · cited 2×
“In fact, the back-door criterion uses a graphical mechanism known as d-separation , which facilitates the demonstration that causal models have testable implications in the data they generate [75].”
BackgroundA Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.64 · Cited by 649
Hu, X., Li, G., Xia, X., Lo, D., & Jin, Z. (2018) · Proceedings of the 26th Conference on Program Comprehension · doi:10.1145/3196321.3196334 · Cited by 621 · cited 1×
“Since then, NCMs have been used for a number of SE tasks such as code completion [3], [15], [39], [40], [41], [42], [43], [44] and translation [4], [9], [45], [46], [47], [48], [49].”
MethodCOMET: A Neural Framework for MT Evaluation
Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.213 · Cited by 617
Deep learning code fragments for code clone detection
White, M., Tufano, M., Vendome, C., & Poshyvanyk, D. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970326 · Cited by 558 · cited 1×
“Researchers have also investigated various representations of NCMs for code [50] as well as graph-based representations based on ASTs [51] and program dependency graphs [52].”
BackgroundSelf-citationCode completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555 · cited 2×
“A NCM is a probability distribution P ( w t | w 1 , w 2 , . . . , w t − 1 ) in which the output w t at time step t , given the input sequence ( w 1 , w 2 , .., w t − 1 ) , is inferred through the conditional probability P ( w t | h t ) [2], [40].”
BackgroundMethodCodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 2×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodCodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · cited 4×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
BackgroundMethodInfluentialHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 1×
“Notable work has called for a more systematic approach that aims to understand a given model’s behavior according to its linguistic capabilities [17], while others have suggested the need for holistic evaluations of Language Models [21].”
BackgroundGraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480 · cited 1×
“It has also been shown that using a NCM for code representation and then fine-tuning for specific SE tasks can achieve state-of-the-art performance across tasks [53], [54], [55].”
MethodOn the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425 · cited 1×
“NCMs in SE have a rich history, stemming from the seminal work by Hindle et al. who proposed the concept of the naturalness of software using n -gram models [38].”
MethodHow are Java software developers using the Eclipse IDE?
Murphy, G. C., Kersten, M., & Findlater, L. (2006) · IEEE Software · doi:10.1109/ms.2006.105 · Cited by 413 · cited 1×
“Given the prior popularity of code completion engines within IDEs [14] and investment in commercial tools, NCMs for code generation will almost certainly be used to help build production software systems in the near future if they are not being used already.”
BackgroundSEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S. J., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 396 · cited 2×
“In particular, code generation has been an important area of SE research for decades, enabling tools for downstream tasks such as code completion [8], program repair [9], and test case generation [1].”
BackgroundMethodSelf-citationAn Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., & Poshyvanyk, D. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3340544 · Cited by 370 · cited 1×
“Since then, NCMs have been used for a number of SE tasks such as code completion [3], [15], [39], [40], [41], [42], [43], [44] and translation [4], [9], [45], [46], [47], [48], [49].”
MethodSelf-citationSharp nearby, fuzzy far away: How neural language models use context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1027 · Cited by 303 · cited 1×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodSharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · arXiv · doi:10.48550/arxiv.1805.04623 · Cited by 302 · cited 1×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodAre deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., & Devanbu, P. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106290 · Cited by 292 · cited 1×
“Since then, NCMs have been used for a number of SE tasks such as code completion [3], [15], [39], [40], [41], [42], [43], [44] and translation [4], [9], [45], [46], [47], [48], [49].”
MethodThe adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284 · cited 1×
“In past work, several factors such as code duplication [80] have been illustrated to affect model predictions.”
BackgroundHolistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
On the "naturalness" of buggy code
Ray, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., & Devanbu, P. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884848 · Cited by 252 · cited 1×
“…conditionals, datatypes), we found that our NCMs are sensitive to seemingly subtle changes in code syntax , reinforcing previous studies concluding the same [65], and our models are only marginally impacted by the presence of comments and bugs, which challenges findings from previous work [97].”
ResultA Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00234 · Cited by 248
Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
Mastropaolo, A., Scalabrino, S., Cooper, N., Nader Palacio, D., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00041 · Cited by 247 · cited 3×
“We employed RNNs and Transformers for all our scenarios as RNNs are widely used in SE [82], while Trans-formers have gained popularity in the SE/NLP domain for their high performance [4].”
BackgroundMethodSelf-citationTu, Z., Su, Z., & Devanbu, P. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635875 · Cited by 238 · cited 1×
“Since then, NCMs have been used for a number of SE tasks such as code completion [3], [15], [39], [40], [41], [42], [43], [44] and translation [4], [9], [45], [46], [47], [48], [49].”
MethodInterpretable Machine Learning
Boehmke, B., & Greenwell, B. (2019) · Hands-On Machine Learning with R · doi:10.1201/9780367816377-16 · Cited by 230 · cited 2×
“These desiderata, which are originally suggested in NLP literature, comprise notions such as facilitating debugging, detecting biases, providing re-course, and eventually, increasing the reliability and trust of NCMs [27], [30], [31].”
BackgroundA statistical semantic language model for source code
Nguyen, T. T., Nguyen, A. T., Nguyen, H. A., & Nguyen, T. N. (2013) · Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2491411.2491458 · Cited by 208 · cited 1×
“Since then, NCMs have been used for a number of SE tasks such as code completion [3], [15], [39], [40], [41], [42], [43], [44] and translation [4], [9], [45], [46], [47], [48], [49].”
MethodOn Learning Meaningful Code Changes Via Neural Machine Translation
Tufano, M., Pantiuchina, J., Watson, C., Bavota, G., & Poshyvanyk, D. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00021 · Cited by 207 · cited 1×
“Since then, NCMs have been used for a number of SE tasks such as code completion [3], [15], [39], [40], [41], [42], [43], [44] and translation [4], [9], [45], [46], [47], [48], [49].”
MethodSelf-citationToward Deep Learning Software Repositories
White, M., Vendome, C., Linares-Vasquez, M., & Poshyvanyk, D. (2015) · 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories · doi:10.1109/msr.2015.38 · Cited by 200 · cited 3×
“A NCM is a probability distribution P ( w t | w 1 , w 2 , . . . , w t − 1 ) in which the output w t at time step t , given the input sequence ( w 1 , w 2 , .., w t − 1 ) , is inferred through the conditional probability P ( w t | h t ) [2], [40].”
BackgroundMethodSelf-citationKnowledge Neurons in Pretrained Transformers
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.581 · Cited by 187
Considerations for Evaluation and Generalization in Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_1 · Cited by 176 · cited 2×
“However, perhaps the most problematic aspect of current neural language models is the fact that they are incapable of explaining their reasoning [23], [24], [25], [26].”
BackgroundEvaluating clone detection tools with BigCloneBench
Svajlenko, J., & Roy, C. K. (2015) · 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsm.2015.7332459 · Cited by 172 · cited 1×
“…work and community datasets: (i) buggy/non-buggy [72], (ii) commented/non-commented [60], (iii) type II, and (iv) type III clone pairs’ differences [73], and Abstract Syntax Trees node masking [74]. do code is extensible, meaning that researchers can define their own code corpora interventions.”
BackgroundTransportability of Causal and Statistical Relations: A Formal Approach
Pearl, J., & Bareinboim, E. (2011) · 2011 IEEE 11th International Conference on Data Mining Workshops · doi:10.1109/icdmw.2011.169 · Cited by 171 · cited 1×
“This transportability of assumptions across similar domains have been researched in Causal Inference [98].”
BackgroundSorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities
White, M., Tufano, M., Martinez, M., Monperrus, M., & Poshyvanyk, D. (2019) · 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8668043 · Cited by 163 · cited 3×
“NCMs have (seemingly) illustrated promising performance across a range of different SE tasks [1], [2], [3], [4], [5], [6], [7].”
BackgroundSelf-citationPolyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.523 · Cited by 161 · cited 1×
“Complementary to our work, the work by Polyjuice paper [67] aims to automatically generate perturbations to language, to see how a given model performs in different scenarios.”
MethodDeep learning similarities from different representations of source code
Tufano, M., Watson, C., Bavota, G., Di Penta, M., White, M., & Poshyvanyk, D. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196431 · Cited by 158 · cited 1×
“NCMs have (seemingly) illustrated promising performance across a range of different SE tasks [1], [2], [3], [4], [5], [6], [7].”
BackgroundSelf-citationRabin, M. R. I., Bui, N. D. Q., Wang, K., Yu, Y., Jiang, L., & Alipour, M. A. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106552 · Cited by 142 · cited 2×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodResultTowards Automating Code Review Activities
Tufano, R., Pascarella, L., Tufano, M., Poshyvanyk, D., & Bavota, G. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00027 · Cited by 141 · cited 1×
“NCMs have (seemingly) illustrated promising performance across a range of different SE tasks [1], [2], [3], [4], [5], [6], [7].”
BackgroundSelf-citationA Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
Watson, C., Cooper, N., Palacio, D. N., Moran, K., & Poshyvanyk, D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3485275 · Cited by 137 · cited 1×
“We employed RNNs and Transformers for all our scenarios as RNNs are widely used in SE [82], while Trans-formers have gained popularity in the SE/NLP domain for their high performance [4].”
MethodSelf-citationTufano, M., Watson, C., Bavota, G., Di Penta, M., White, M., & Poshyvanyk, D. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3240732 · Cited by 133 · cited 1×
“The authors introduce the concept of Ladder of Causa-tion to match distinct levels of cognitive ability with concrete actions: seeing ( level one ), doing ( level two ), and imagining ( level three ).”
BackgroundSelf-citationHan, T., Srinivas, S., & Lakkaraju, H. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.01254 · Cited by 129 · cited 1×
“The most common criteria are post hoc vs. intrinsic, model-specific vs. model-agnostic, global vs. local explanation scope, feature importance vs. rule-based unit of explanation, and causal levels [25], [36].”
BackgroundErrudite: Scalable, Reproducible, and Testable Error Analysis
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1073 · Cited by 128 · cited 1×
“Most of the conclusions from Chen et al.’s study were uncovered through manual analysis, e.g., through sourcing counterexamples, making it difficult to rigorously quantify or to systematically apply such an analysis to research prototypes [16].”
MethodPerturbation Sensitivity Analysis to Detect Unintended Model Biases
Prabhakaran, V., Hutchinson, B., & Mitchell, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1578 · Cited by 115 · cited 1×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodGraph-Based Statistical Language Model for Code
Nguyen, A. T., & Nguyen, T. N. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.336 · Cited by 114 · cited 1×
“Researchers have also investigated various representations of NCMs for code [50] as well as graph-based representations based on ASTs [51] and program dependency graphs [52].”
BackgroundOn learning meaningful assert statements for unit test cases
Watson, C., Tufano, M., Moran, K., Bavota, G., & Poshyvanyk, D. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380429 · Cited by 111 · cited 2×
“NCMs have (seemingly) illustrated promising performance across a range of different SE tasks [1], [2], [3], [4], [5], [6], [7].”
BackgroundSelf-citationAn Empirical Study on the Usage of Transformer Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Di Penta, M., & Bavota, G. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3128234 · Cited by 94 · cited 2×
“NCMs have (seemingly) illustrated promising performance across a range of different SE tasks [1], [2], [3], [4], [5], [6], [7].”
BackgroundMethodSelf-citationTo Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation
Kocmi, T., Federmann, C., Grundkiewicz, R., Junczys-Dowmunt, M., Matsushita, H., & Menezes, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10821 · Cited by 79 · cited 1×
“Moreover, skepticism within the NLP research community is growing regarding the efficacy of current accuracy-based metrics, as these metrics tend to overestimate model performance [17], [18], [19].”
BackgroundAn Empirical Study on the Usage of BERT Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Poshyvanyk, D., Di Penta, M., & Bavota, G. (2021) · 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00024 · Cited by 78
Self-citationLearning How to Mutate Source Code from Bug-Fixes
Tufano, M., Watson, C., Bavota, G., Di Penta, M., White, M., & Poshyvanyk, D. (2019) · 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme.2019.00046 · Cited by 69 · cited 1×
“We define four types of SE application settings, adapted from both our prior work and community datasets: (i) buggy/non-buggy [72], (ii) commented/non-commented [60], (iii) type II, and (iv) type III clone pairs’ differences [73], and Abstract Syntax Trees node masking [74]. do code is extensible,…”
MethodSelf-citationOn the Reliability and Explainability of Language Models for Program Generation
Liu, Y., Tantithamthavorn, C. K., Liu, Y., & Li, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3641540 · Cited by 68 · cited 1×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodCounterfactual Explanations for Models of Code
Cito, J., Dillig, I., Murali, V., & Chandra, S. (2022) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip55303.2022.9794112 · Cited by 67
From statistical to causal learning
Schölkopf, B., & von Kügelgen, J. (2023) · EMS Press eBooks · doi:10.4171/icm2022/173 · Cited by 63
Unsupervised translation of programming languages
Roziere, B. (2026) · doi:10.70675/416f6582z4576z444eza710z5e930d59bf97 · Cited by 57
Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., & Vinyals, O. (2021) · arXiv · doi:10.48550/arxiv.2107.07002 · Cited by 48 · cited 1×
“Even benchmarks that span multiple tasks and metrics have been shown to lack robustness, leading to incorrect assumptions of model comparisons [20].”
BackgroundDoWhy: Addressing Challenges in Expressing and Validating Causal AssumptionsPreprint
Sharma, A. K., Syrgkanis, V., Zhang, C., & Kıcıman, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13518 · Cited by 44 · cited 1×
“Inspired by the notion of Pearl’s Ladder of Causation [29], [95], we framed two research questions to explore do code”
MethodExplaining Transformer-based Code Models: What Do They Learn? When They Do Not Work?
Mohammadkhani, A. H., Tantithamthavorn, C. K., & Hemmatif, H. (2023) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam59687.2023.00020 · Cited by 41
Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmaps
Lo, D. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00010 · Cited by 40 · cited 1×
“These desiderata, which are originally suggested in NLP literature, comprise notions such as facilitating debugging, detecting biases, providing re-course, and eventually, increasing the reliability and trust of NCMs [27], [30], [31].”
BackgroundMaybe Deep Neural Networks are the Best Choice for Modeling Source Code
Karampatsis, R.-M., & Sutton, C. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05734 · Cited by 39 · cited 1×
“Since then, NCMs have been used for a number of SE tasks such as code completion [3], [15], [39], [40], [41], [42], [43], [44] and translation [4], [9], [45], [46], [47], [48], [49].”
MethodKnowledge Neurons in Pretrained Transformers
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2104.08696 · Cited by 39 · cited 1×
“…into two basic groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses…”
MethodHan, T., Srinivas, S., & Lakkaraju, H. (2022) · neural information processing systems · doi:10.52202/068431-0380 · Cited by 38
Deep Transfer Learning for Source Code Modeling
Hussain, Y., Huang, Z., Zhou, Y., & Wang, S. (2020) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194020500230 · Cited by 28
Benchmarking Causal Study to Interpret Large Language Models for Source Code
Rodriguez-Cardenas, D., Palacio, D. N., Khati, D., Burke, H., & Poshyvanyk, D. (2023) · 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58846.2023.00040 · Cited by 14 · cited 2×
“To build the data intervention testbeds for our interpretability scenarios, we obtained Java samples from CodeXGLUE [89] and Python samples from Galeras [74].”
BackgroundMethodSelf-citationOn the Relationship Between Explanation and Prediction: A Causal ViewPreprint
Karimi, A.-H., Muandet, K., Kornblith, S., Schölkopf, B., & Kim, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.06925 · Cited by 12 · cited 1×
“Other related works from Explainable Artificial Intelligence (XAI) have focused on the usage of causal theory applied to generate post hoc explanations about the logical structure of a neural network [69] or explore the causal relationships between the explanation and the prediction [70].”
BackgroundOpen-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C. A., & Janes, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377812.3390806 · Cited by 8 · cited 2×
“Hence, rather than using n -grams or Markov Models to approximate P ( w t | w t − 1:1 ) [44], it is convenient to use a latent model P ( w t | w t − 1:1 ) ≈ P ( w t | d t ) , where d t is known as a hidden state that embeds the sequence information from past observations up to the time step t .”
MethodA Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
Watson, C., Cooper, N., Palacio, D. N., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06520 · Cited by 5 · cited 1×
“We employed RNNs and Transformers for all our scenarios as RNNs are widely used in SE [82], while Trans-formers have gained popularity in the SE/NLP domain for their high performance [4].”
MethodSelf-citationExplainable AI for Pre-Trained Code Models: What Do They Learn? When They Do Not Work?Preprint
Mohammadkhani, A. H., Tantithamthavorn, C. K., & Hemmati, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.12821 · Cited by 4 · cited 1×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
MethodTrustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and RoadmapsPreprint
Lo, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.04142 · Cited by 2 · cited 1×
“These desiderata, which are originally suggested in NLP literature, comprise notions such as facilitating debugging, detecting biases, providing re-course, and eventually, increasing the reliability and trust of NCMs [27], [30], [31].”
BackgroundOn the Reliability and Explainability of Language Models for Program GenerationPreprint
Liu, Y., Tantithamthavorn, C. K., Liu, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09587 · cited 1×
“…groups: (i) intrinsic techniques such as probing for specific linguistic characteristics [56] and examining neuron activations [57]; and (ii) post hoc techniques such as benchmarks [15], [34], [58], [59], [60], attention-based analysis [61], [62], and intervention analyses [17], [63], [64], [65].”
Method
Research works printed but not linked: 6
- F. Doshi-Velez and B. Kim, "Towards A Rigorous Science of Interpretable Machine Learning," arXiv: Machine Learning, no. Ml, pp. 1-13, 2017. [Online]. Available: http://arxiv.org/abs/1702. 08608
- B. Sch ölkopf and J. von K ügelgen, "From statistical to causal learning," 4 2022. [Online]. Available: http://arxiv.org/abs/2204. 00607
- Y. Hu and J. Tian, "Neuron dependency graphs: A causal abstraction of neural networks," in Proceedings of the 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka et al., Eds., vol. 162. PMLR, 17-23 Jul 2022, pp. 9020-9040. [Online]. Available: https://proceedings.mlr.press/v162/hu22b.html
- A. Paszke, S. Gross et al., "Pytorch: An imperative style, high- performance deep learning library," in Advances in Neural Infor- mation Processing Systems 32, H. Wallach, H. Larochelle et al., Eds. Curran Associates, Inc., 2019, pp. 8024-8035. · doi:10.5555/3454287.3455008
- R. M. Karampatsis, H. Babii et al., "Big code != big vocabulary: Open-vocabulary models for source code," Proceedings -Interna- tional Conference on Software Engineering, pp. 1073-1085, 2020.
- S. Lu, D. Guo et al., "Codexglue: A machine learning bench- mark dataset for code understanding and generation," CoRR, vol. abs/2102.04664, 2021. · arXiv
Cited by
Artificial Intelligence for Software Engineering: The Journey So Far and the Road Ahead
Ahmed, I., Aleti, A., Cai, H., Chatzigeorgiou, A., He, P., Hu, X., Pezzè, M., Poshyvanyk, D., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719006 · Cited by 54
“…trivially large datasets for specific tasks, for instance, by simply applying semantics-preserving code refactoring schemes, including junk code insertion, to multiply relatively small dataset, the resulting datasets do not significantly increase the knowledge available for training the models.”
Self-citationA Systematic Literature Review on Explainability for ML/DL-based Software Engineering
Cao, S., Sun, X., Widyasari, R., Lo, D., Wu, X., Bo, L., Zhang, J., Li, B., Liu, W., Wu, D., et al. (2025) · ACM Computing Surveys · doi:10.1145/3763230 · Cited by 50
HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent
Wu, J. J., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715109 · Cited by 55
“At a high level, the gap lies in several critical aspects of software development beyond coding, such as effective communications, explanation, requirements, design, domain knowledge, and the broader context of relevant projects and components [47, 51, 63, 65, 69].”
From Triumph to Uncertainty: The Journey of Software Engineering in the AI Era
Mastropaolo, A., Escobar‐Velásquez, C., & Linares‐Vásquez, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709360 · Cited by 30
“By leveraging our expertise in software engineering, we can design robust frameworks and methodologies to mitigate biases, ensure data quality, and facilitate the adoption of SE for AI-based system development (SE4AI) [16, 190].”
How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study
Velasco, A., Rodríguez-Cárdenas, D., Alif, L. R., Palacio, D. N., & Poshyvanyk, D. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00025 · Cited by 22
“By aligning meaningful tokens to highlight smelly segments, our approach offers a novel, interpretable metric [29], [43], suitable for black-box code generation models.”
Self-citation
Metadata
- Authors
- Nader Palacio, D., Velasco, A., Cooper, N., Rodriguez, A., Moran, K., & Poshyvanyk, D.
- Affiliations
- Department of Computer Science, William & Mary, Williamsburg, VA, USA
- William & Mary (US)
- Department of Computer Science, Universidad Nacional de Colombia, Bogota, Colombia
- George Mason University (US)
- ORCID
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2024-05
- Volume
- 50
- Issue
- 5
- Pages
- 1215-1243
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4393058215
- Cited by
- 18
- References
- 86
- Citations by year
- 2026: 3 · 2025: 8 · 2024: 5
- FWCI
- 2.36
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- NSF (CCF-2311469, CNS-2132281, CCF-2007246, CCF-1955853, CCF-2311468, CCF-2132285)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- post-hoc explanation, causal inference, confounding, spurious correlations, programming constructs, token prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.