FalconCode: A Multiyear Dataset of Python Code Samples from an Introductory Computer Science Course
Open accessAbstract
The lack of large and diverse datasets of student code samples limits some forms of computer science education research. To address this problem, we created FalconCode, a novel collection of over 1.5 million Python programs from over two thousand undergraduate students at the United States Air Force Academy. FalconCode captures over five semesters worth of code samples from our introduction to computing course, which is taken by every student regardless of their academic major. The dataset contains student code submissions for over 800 programming assignments, as well as additional metadata such as the prompt for each assignment, the testcase(s) used to evaluate student submissions, and the specific skills needed to solve each problem. In this paper, we describe the methodology used to create FalconCode and the steps taken to anonymize the data. We then describe FalconCode's data schema, and show how it can support a wide range of research---including those utilizing machine learning (ML) and artificial intelligence (AI). FalconCode is provided free-of-charge, and is available upon request for computer science education research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
FalconCode is a novel collection of over 1.5 million Python programs from over 2,000 undergraduate students at the United States Air Force Academy.
“FalconCode is a novel collection of over 1.5 million Python programs from over 2,000 undergraduate students at the United States Air Force Academy [64].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [64]
In HumanEval and FalconCode, the vast majority of problems only cover material suitable in an introductory programming setting.
“We found that in these two datasets, the vast majority of problems only cover material suitable in an introductory programming setting.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779 · cited 1× · see citing works
“dents who have voluntarily enrolled for the course (and thus view programming favorably), which introduces self-selection bias [13].”
BackgroundLowering the barriers to programming
Kelleher, C. L., & Pausch, R. F. (2005) · ACM Computing Surveys · doi:10.1145/1089733.1089734 · Cited by 887 · see citing works
Summarizing Source Code using a Neural Attention Model
Iyer, S. V., Konstas, I., Cheung, A., & Zettlemoyer, L. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1195 · Cited by 716 · see citing works
PyDriller: Python framework for mining software repositories
Spadini, D., Aniche, M., & Bacchelli, A. (2018) · ESEC/SIGSOFT FSE · doi:10.1145/3236024.3264598 · Cited by 372 · see citing works
Educational Data Mining and Learning Analytics in Programming
Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., et al. (2015) · doi:10.1145/2858796.2858798 · Cited by 332 · see citing works
Brown, N. C. C., Kölling, M., McCall, D., & Utting, I. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538924 · Cited by 179 · see citing works
Probabilistic model for code with decision trees
Raychev, V., Bielik, P., & Vechev, M. (2016) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2983990.2984041 · Cited by 157 · see citing works
Toward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems
Murray, T., & Arroyo, I. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47987-2_75 · Cited by 119 · see citing works
Comparing effective and ineffective behaviors of student programmers
Edwards, S. H., Snyder, J., Pérez-Quiñones, M. A., Allevato, A., Kim, D., & Tretola, B. (2009) · doi:10.1145/1584322.1584325 · Cited by 117 · see citing works
Proceedings of the 41st ACM technical symposium on Computer science education
Lewandowski, G., Wolfman, S. A., Cortina, T. J., Walker, E., & Musicant, D. R. (2010) · doi:10.1145/1734263 · Cited by 116 · see citing works
Scheuer, O., & McLaren, B. M. (2012) · Springer eBooks · doi:10.1007/978-1-4419-1428-6_618 · Cited by 107 · see citing works
Pereira, F. D., Oliveira, E., de Oliveira, D. B. F., Cristea, A. I., de Carvalho, L. S. G., Fonseca, S. C., Toda, A. M., & Isotani, S. (2020) · British Journal of Educational Technology · doi:10.1111/bjet.12953 · Cited by 102 · see citing works
Investigating novice programming mistakes: educator beliefs vs. student data
Brown, N. C. C., & Altadmri, A. (2014) · Proceedings of the tenth annual conference on International computing education research · doi:10.1145/2632320.2632343 · Cited by 89 · see citing works
Analyzing Student Work Patterns Using Programming Exercise Data
Spacco, J. W., Denny, P. C., Richards, B., Babcock, D., Hovemeyer, D., Moscola, J., & Duvall, R. C. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677297 · Cited by 75 · see citing works
Reproducibility of Data-Oriented Experiments in e-Science (Dagstuhl Seminar 16041)
Freire, J., Fuhr, N., & Rauber, A. (2016) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/dagrep.6.1.108 · Cited by 68 · see citing works
Identifying challenging CS1 concepts in a large problem dataset
Cherenkova, Y., Zingaro, D., & Petersen, A. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538966 · Cited by 55 · see citing works
Mining student CVS repositories for performance indicators
Mierle, K., Laven, K., Roweis, S. T., et al. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083150 · Cited by 51 · see citing works
Brown, N. C. C., Altadmri, A., Sentance, S., & Kölling, M. (2018) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3230977.3230991 · Cited by 49 · see citing works
Understanding semantic style by analysing student code
Ruvo, G. D., Tempero, E. D., Luxton-Reilly, A., Rowe, G. B., & Giacaman, N. (2018) · Australasian Computing Education Conference · doi:10.1145/3160489.3160500 · Cited by 46 · see citing works
A Dataset of Scratch Programs: Scraped, Shaped and Scored
Aivaloglou, E., Hermans, F., Moreno-León, J., & Robles, G. (2017) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.45 · Cited by 33 · see citing works
Promoting Diversity-Inclusive Computer Science Pedagogies: A Multidimensional Perspective
Pournaghshband, V., & Medel, P. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387360 · Cited by 31 · see citing works
Understanding the Effects of Lecturer Intervention on Computer Science Student Behaviour
Szabo, C., Falkner, N. J. G., Dorodchi, M., Knutas, A., & Maiorana, F. (2017) · Annual Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3059009.3081330 · Cited by 18 · see citing works
Making Computer Science Education Mandatory
Repenning, A., Lamprou, A., Petralito, S., et al. (2019) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3304221.3319758 · Cited by 15 · see citing works
Computer Science Educators Stack Exchange
Moudgalya, S. K., Rich, K. M., Yadav, A. K., & Koehler, M. J. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287365 · Cited by 12 · see citing works
Babeş-Vroman, M., Nguyen, T. N., & Nguyen, T. D. (2021) · ACM Transactions on Computing Education · doi:10.1145/3471572 · Cited by 11 · see citing works
Hwang, D., Agarwal, V., Lyu, Y., et al. (2021) · Australasian Computing Education Conference · doi:10.1145/3441636.3442317 · Cited by 5 · see citing works
Computer science in general education: beyond quantitative reasoning
Tartaro, A., Healy, C., & Treu, K. (2016) · Journal of computing sciences in colleges · Cited by 4 · see citing works
Towards a pattern language for increasing gender diversity in computer science education
Köppe, C., & Bartilla, A. (2017) · doi:10.1145/3158491.3158501 · Cited by 4 · see citing works
Student Psychological Factors and Diversity in Computer Science Education
Ihorn, S. M., Yoon, I., & Kulkarni, A. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3372671 · Cited by 3 · see citing works
Modernizing a General Education Requirement in Computing to Emphasize Critical Thinking
O’Neill, B., & Hansen, L. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3432531 · Cited by 2 · see citing works
CodeKey - An Online Code Editor to Study Code Patterns and Enhance Student Performance in CS Courses
Williams, R., & Garcia, D. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3372680 · Cited by 1 · see citing works
A longitudinal dataset of five years of public activity in the Scratch online communityPreprint
Hill, B. M., & Monroy-Hernández, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01184 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Giuseppe De Ruvo, Ewan Tempero, Andrew Luxton-Reilly, Gerard B. Rowe, and Nasser Giacaman. 2018. Understanding Semantic Style by Analysing Student Code. In Proceedings of the 20th Australasian Computing Education Conference (Brisbane, Queensland, Australia) (ACE '18). ACM, New York, NY, USA, 73-82. https://doi.org/10.1145/3160489.3160500 · DOI
- Ruchir Puri, David Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian T Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.), Vol. 1. 13.
- Lev Vygotsky. 1978. Interaction between learning and development. Readings on the development of children 23, 3 (1978), 34-41.
Cited by
Sorted by citations
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
- Authors
- de Freitas, A., Coffman, J., de Freitas, M., Wilson, J., & Weingart, T.
- Affiliations
- United States Air Force Academy, USAF Academy, CO, USA
- United States Air Force Academy (US)
- Academy District 20, Colorado Springs, CO, USA
- Venue
- Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-03-02
- Pages
- 938-944
- OpenAlex
- W4323037917
- Cited by
- 15
- References
- 32
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 7 · 2023: 2
- FWCI
- 4.54
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- Python, introductory computer science, programming exercises, student code submissions, data anonymization, data schema, undergraduate computer science education, United States Air Force Academy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
