Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLMs), such as GPT-4, StarCoder, and Code Llama, are transforming the way developers approach programming by automatically generating code based on given contexts, such as natural language descriptions or incomplete surrounding code. Despite advancements, generating syntactically and semantically correct code remains challenging, especially for complex programming tasks. Existing approaches typically generate multiple candidate solutions using LLMs to increase the likelihood of producing correct code. However, selecting the correct code from these candidates — a process known as code ranking — remains a major challenge. Current research on code ranking can be cate-gorized into execution-based and non-execution-based methods. Execution-based methods, although effective, encounter notable limitations, such as scarcity of quality unit tests and security risks. Non-execution-based methods like CodeRanker, which rely solely on classification labels to train a code ranker, struggle to capture subtle errors and provide detailed error insights. Recognizing the strengths and limitations of both approaches, we propose a new method that integrates the advantages of execution-based and non-execution-based techniques. The key insight of our work is that an effective code ranker is expected to truly comprehend the underlying causes of erroneous code, as relying solely on classification labels is insufficient. Inspired by this, this paper puts forward RankEF, an innov…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“The detailed process is illustrated in Figure 2.”
MethodGolub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505
Feature Selection: A Data Perspective
Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R. P., Tang, J., & Liu, H. (2018) · ACM Computing Surveys · doi:10.1145/3136625 · Cited by 2,442
Self-citationSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 1×
“…learning [22, 62, 65], multi-task learning [72], dual In contrast, our approach trains an independent code ranker using a multi-task learning framework, eliminating the dependence on execution feedback during the inference phase, thereby overcoming the limitations of the aforementioned methods.”
MethodWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 1×
“…as the output, such neural comment generation (NCG) approaches automatically learn a function, which is usually a DNN model such as the neural machine Through the integration of execution feedback and classification labels using multi-task learning, RankEF not only distinguishes between…”
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · cited 1×
“…as the output, such neural comment generation (NCG) approaches automatically learn a function, which is usually a DNN model such as the neural machine Through the integration of execution feedback and classification labels using multi-task learning, RankEF not only distinguishes between…”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 13×
“Execution-based code ranking methods, which involve sampling a large number of code candidates and executing a few unit tests to filter out those that pass, have notably improved the code generation capabilities of LLMs [4, 6, 7].”
BackgroundMethodInfluentialData Preprocessing in Data Mining
García, S., Luengo, J., & Herrera, F. (2015) · Intelligent Systems Reference Library · doi:10.1007/978-3-319-10247-4 · Cited by 1,368
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Multi-Task Learning as Multi-Objective Optimization
Sener, O., & Koltun, V. (2018) · arXiv · doi:10.48550/arxiv.1810.04650 · Cited by 852 · cited 1×
“As the RankEF method involves two key tasks: code classification and execution feedback prediction, researches [23, 24] in the field of multitask learning have shown that there may be conflicting phenomena between different tasks, which can lead to a decrease in the effectiveness of multitask…”
MethodMulti-Task Learning for Dense Prediction Tasks: A Survey
Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D., & Van Gool, L. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3054719 · Cited by 840 · cited 1×
“As the RankEF method involves two key tasks: code classification and execution feedback prediction, researches [23, 24] in the field of multitask learning have shown that there may be conflicting phenomena between different tasks, which can lead to a decrease in the effectiveness of multitask…”
MethodFeature Selection: A Data Perspective
Li, J., Cheng, K., Wang, S., Morstatter, F., Treviño, R. P., Tang, J., & Liu, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.07996 · Cited by 739
Self-citationUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487 · cited 1×
“Error Message”
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A., Bui, N., Li, J., & Hoi, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.68 · Cited by 459
Self-citationAn Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3334955 · Cited by 431
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · cited 4×
“In this paper, we employ a diverse array of leading code pre-training models, such as PyCodeGPT (110M) [26], StarCoder (164M, 15B) [2], CodeT5+ (770M) [27], GPT-Neo (1.3B) [28], CodeGen (350M, 2B) [29], CodeLlama (7B) [3], and GPT-3.”
BackgroundMethodInfluentialIs ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., & Ren, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.923 · Cited by 270
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., & Ren, Z. (2023) · arXiv · doi:10.48550/arxiv.2304.09542 · Cited by 270 · cited 1×
“In the field of natural language processing, recent research [33–35] has focused on utilizing LLMs, such as ChatGPT, for text ranking tasks.”
MethodA survey on large language models for code generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2024) · arXiv · doi:10.48550/arxiv.2406.00515 · Cited by 257
A Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251 · cited 1×
“…RELATED WORK : execution-based [4, 8, 9, 39] and non-execution-based [ These execution-based approaches require code execution This code is correct. limitation, CodeRanker Input: {4\n2 5 4 6}, expect output: {5}, but get: {-2}”
BackgroundIs chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235
Self-citationUncovering ChatGPT’s Capabilities in Recommender Systems
Dai, S., Shao, N., Zhao, H., Yu, W., Si, Z., Xu, C., Sun, Z., Zhang, X., & Xu, J. (2023) · Proceedings of the 17th ACM Conference on Recommender Systems · doi:10.1145/3604915.3610646 · Cited by 187 · cited 1×
“In the field of natural language processing, recent research [33–35] has focused on utilizing LLMs, such as ChatGPT, for text ranking tasks.”
MethodSelf-citationMeasuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 3×
“• APPS [5] compiles 10,000 problems from platforms like Code-forces and LeetCode, divided into 5,000 each for training and testing.”
BackgroundMethodCode-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM
Ryan, G., Jain, S., Shang, M., Wang, S., Ma, X., Ramanathan, M. K., & Ray, B. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643769 · Cited by 120 · cited 2×
“Using this, RankEF calculates a score 𝑖 indicating the correctness of 𝑖 , and then ranks the candidates based on these scores, with a goal to position the correct code higher in the list.”
MethodWizardCoder: Empowering Code Large Language Models with Evol-Instruct
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., & Jiang, D. (2023) · arXiv · doi:10.48550/arxiv.2306.08568 · Cited by 120
CodeT: Code Generation with Generated Tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., & Chen, W. (2022) · arXiv · doi:10.48550/arxiv.2207.10397 · Cited by 92 · cited 5×
“Using this, RankEF calculates a score 𝑖 indicating the correctness of 𝑖 , and then ranks the candidates based on these scores, with a goal to position the correct code higher in the list.”
BackgroundMethodInfluentialAdaptive Test Generation Using a Large Language ModelPreprint
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2023) · arXiv.org · doi:10.48550/arxiv.2302.06527 · Cited by 77
LLM-Based Code Generation Method for Golang Compiler Testing
Gu, Q. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3617850 · Cited by 73 · cited 1×
“…RELATED WORK : execution-based [4, 8, 9, 39] and non-execution-based [ These execution-based approaches require code execution This code is correct. limitation, CodeRanker Input: {4\n2 5 4 6}, expect output: {5}, but get: {-2}”
BackgroundCERT: Continual Pre-training on Sketches for Library-oriented Code Generation
Zan, D., Chen, B., Yang, D., Lin, Z., Kim, M., Guan, B., Wang, Y., Chen, W., & Lou, J.-G. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/329 · Cited by 65
An empirical study of automated unit test generation for Python
Lukasczyk, S., Kroiß, F., & Fraser, G. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10248-w · Cited by 64 · cited 1×
“While execution-based methods can effectively filter correct code, their practical applicability is limited, as generating high-quality unit test cases is time-consuming [10–12], and executing code generated by LLMs without robust security guarantees introduces significant security risks [13, 14].”
Fault-Aware Neural Code RankersPreprint
Inala, J., Wang, C., Yang, M., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.03865 · Cited by 62
Coder Reviewer Reranking for Code GenerationPreprint
Zhang, T., Yu, C., Hashimoto, T., Lewis, M., Yih, W.-T., Fried, D., & Wang, S. I. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.16490 · Cited by 57 · cited 7×
“In addition to APPS, we evaluate RankEF’s transfer capabilities on two other prominent code generation benchmarks, namely MBPP and HumanEval.”
BackgroundMethodInfluentialSelf-Edit: Fault-Aware Code Editor for Code Generation
Zhang, K., Li, Z., Li, J., Li, G., & Jin, Z. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.45 · Cited by 55
Self-citationNatural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · arXiv · doi:10.48550/arxiv.2204.11454 · Cited by 49
LEVER: Learning to Verify Language-to-Code Generation with Execution
Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.-T., Wang, S. I., & Lin, X. V. (2023) · arXiv · doi:10.48550/arxiv.2302.08468 · Cited by 46
Natural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.231 · Cited by 46
Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMsPreprint
Sun, Z., Lyu, C., Li, B., et al. (2024) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2403.13271 · Cited by 43
Self-citationKnowledge-Aware Code Generation with Large Language Models
Huang, T., Sun, Z., Jin, Z. G., Li, G., & Lyu, C. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644418 · Cited by 43
Self-citationEnhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs
Sun, Z., Lyu, C., Li, B., Wan, Y., Zhang, H., Li, G., & Jin, Z. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2872j9c7tudz · Cited by 38
Self-citationIRCoCo: Immediate Rewards-Guided Deep Reinforcement Learning for Code Completion
Li, B., Sun, Z., Huang, T., Zhang, H., Wan, Y., Li, G., Jin, Z. G., & Lyu, C. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643735 · Cited by 31 · cited 1×
“Using this, RankEF calculates a score 𝑖 indicating the correctness of 𝑖 , and then ranks the candidates based on these scores, with a goal to position the correct code higher in the list.”
MethodSelf-citationFIFA: Making Fairness More Generalizable in Classifiers Trained on Imbalanced DataPreprint
Deng, Z., Zhang, J., Zhang, L., Ye, T., Coley, Y., Su, W., & Zou, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.02792 · Cited by 17 · cited 1×
“Models trained solely on classification labels are highly sensitive to label distribution, leading to weak generalization ability, as demonstrated in numerous studies [16, 17].”
BackgroundFault-Aware Neural Code Rankers
Inala, J. P., Wang, C., Yang, M., Codas, A., Encarnación, M. J., Lahiri, S. K., Musuvathi, M., & Gao, J. (2022) · neural information processing systems · doi:10.52202/068431-0976 · Cited by 16
Larson, S., Singh, N., Maheshwari, S., Stewart, S., & Krishnaswamy, U. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86159-9_30 · Cited by 10
Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMPreprint
Ryan, G., Jain, S., Shang, M., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.00097 · Cited by 6 · cited 2×
“Using this, RankEF calculates a score 𝑖 indicating the correctness of 𝑖 , and then ranks the candidates based on these scores, with a goal to position the correct code higher in the list.”
Method
Research works printed but not linked: 12
- OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. · arXiv
- Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. · arXiv
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. · arXiv
- Mohammed Latif Siddiq, Joanna Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. Exploring the effectiveness of large language models in generating unit tests. arXiv preprint arXiv:2305.00418, 2023. · arXiv
- Mohammed Latif Siddiq and Joanna Santos. Generate and pray: Using sallms to evaluate the security of llm generated code. arXiv preprint arXiv:2311.00889, 2023. · arXiv
- Rebeka Tóth, Tamas Bisztray, and László Erdodi. Llms in web-development: Evaluating llm-generated php code unveiling vulnerabilities and limitations. arXiv preprint arXiv:2404.14459, 2024. · arXiv
- Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018. · arXiv
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022. · arXiv
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999, 2022. · arXiv
- Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. · arXiv
- Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Cited by
Large Language Model-Aware In-Context Learning for Code Generation
Li, J., Tao, C., Li, J., Li, G., Jin, Z., Zhang, H., Fang, Z., & Liu, F. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715908 · Cited by 32
Self-citationLarge Language Model-Aware In-Context Learning for Code Generation
Li, J., Li, G., Tao, C., Li, J., Zhang, H., Liu, F., & Jin, Z. (2023) · arXiv · doi:10.48550/arxiv.2310.09748 · Cited by 32
Self-citationMulti-stage guided code generation for Large Language Models
Han, Y., & Lyu, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109491 · Cited by 36
Self-citationWebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Gui, Y., Li, Z., Wan, Y., Shi, Y., Zhang, H., Chen, B., Su, Y., Chen, D., Wu, S., Zhou, X., et al. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714889 · Cited by 17
Self-citationUIC opilot : Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs
Gui, Y., Wan, Y., Li, Z., Zhang, Z., Chen, D., Zhang, H., Su, Y., Chen, B., Zhou, X., Jiang, W., et al. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714891 · Cited by 62
Self-citation
Metadata
- Authors
- Sun, Z., Wan, Y., Li, J., Zhang, H., Jin, Z., Li, G., & Lyu, C.
- Affiliations
- Shandong Normal University (CN)
- Huazhong University of Science and Technology (CN)
- Peking University (CN)
- Chongqing University (CN)
- Venue
- Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2024-10-27
- Pages
- 229-241
- OpenAlex
- W4403536858
- Semantic Scholar
- de630124d259fec0133263bd51124a72a5f3b4ed
- Cited by
- 10
- References
- 29
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 1
- FWCI
- 3.81
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Natural Science Foundation of Shandong Province, China?ZR2021MF059
- National Natural Science Foundation of China?62192731, 62072007, 62192733, 61832009, 62192730
- National Key R&D Program?2023YFB4503801
- Key Program of Hubei?JD2023008
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- execution feedback, large language models, code generation, multi-task learning, code classification, code quality evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.