Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming
The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
New high-frequency, automated data collection and analysis algorithms could offer new insights into complex learning processes, especially for tasks in which students have opportunities to generate unique open-ended artifacts such as computer programs. These approaches should be particularly useful because the need for scalable project-based and student-centered learning is growing considerably. In this article, we present studies focused on how students learn computer programming, based on data drawn from 154,000 code snapshots of computer programs under development by approximately 370 students enrolled in an introductory undergraduate programming course. We use methods from machine learning to discover patterns in the data and try to predict final exam grades. We begin with a set of exploratory experiments that use fully automated techniques to investigate how much students change their programming behavior throughout all assignments in the course. The results show that students’ change in programming patterns is only weakly predictive of course performance. We subsequently hone in on 1 single assignment, trying to map students’ learning process and trajectories and automatically identify productive and unproductive (sink) states within these trajectories. Results show that our process-based metric has better predictive power for final exams than the midterm grades. We conclude with recommendations about the use of such methods for assessment, real-time feedback, and course improvement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
An Introduction to Support Vector Machines and Other Kernel-based Learning Methods
Cristianini, N., & Shawe-Taylor, J. (2000) · doi:10.1017/cbo9780511801389 · Cited by 13,855
Needleman, S. B., & Wunsch, C. D. (1970) · Journal of Molecular Biology · doi:10.1016/0022-2836(70)90057-4 · Cited by 11,288 · cited 1×
“We used the Needleman–Wunsch 5 algorithm for global DNA alignment (Needleman & Wunsch, 1970) to measure the difference between the lists of commands and functions executed (or API calls) by the two programs.”
MethodWing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006) · Educational Psychologist · doi:10.1207/s15326985ep4102_1 · Cited by 7,776 · cited 1×
“Although this vision has captured the hearts and minds of educators since the turn of the 20th century, its need and feasibility has come under attack several times during the past decade (e.g., Kirschner, Sweller, & Clark, 2006; Klahr & Nigam, 2004).”
Background(1980) · Medical Entomology and Zoology · Cited by 7,534
A vector space model for automatic indexing
Salton, G., Wong, A., & Yang, C. S. (1975) · Communications of the ACM · doi:10.1145/361219.361220 · Cited by 7,521 · cited 1×
“Bag of Words Difference: We built histograms of the different keywords used in a program and used the Euclidean distance between two histograms as a naïve measure of the similarity (Salton, Wong, & Yang, 1975).”
MethodAn introduction to hidden Markov models
Rabiner, L., & Juang, B. (1986) · IEEE ASSP Magazine · doi:10.1109/massp.1986.1165342 · Cited by 4,775
An Introduction to Support Vector Machines and Other Kernel‐based Learning Methods
(2001) · Kybernetes · doi:10.1108/k.2001.30.1.103.6 · Cited by 3,308
Engineering Design Thinking, Teaching, and Learning
Dym, C. L., Agogino, A. M., Eris, O., Frey, D. D., & Leifer, L. J. (2005) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2005.tb00832.x · Cited by 2,801
(2010) · Encyclopedia of Curriculum Studies · doi:10.4135/9781412958806.n175 · Cited by 2,556
Advances in Instructional Psychology
(2019) · Routledge eBooks · doi:10.4324/9781315864341 · Cited by 2,532
Mindstorms: Children, Computers, and Powerful Ideas
Malraison, P. J., & Papert, S. (1981) · The Two-Year College Mathematics Journal · doi:10.2307/3027082 · Cited by 1,824
Dewey, J. (2024) · doi:10.55094/holistence.497 · Cited by 1,484
Scatter/Gather: A Cluster-based Approach to Browsing Large Document Collections
Cutting, D. R., Karger, D. R., Pedersen, J. O., & Tukey, J. W. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130362 · Cited by 1,419
The State of Educational Data Mining in 2009: A Review and Future Visions
Baker, R. S. J. D., & Yacef, K. (2009) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554657 · Cited by 1,359
Klahr, D., & Nigam, M. (2004) · Psychological Science · doi:10.1111/j.0956-7976.2004.00737.x · Cited by 977 · cited 1×
“Although this vision has captured the hearts and minds of educators since the turn of the 20th century, its need and feasibility has come under attack several times during the past decade (e.g., Kirschner, Sweller, & Clark, 2006; Klahr & Nigam, 2004).”
BackgroundScatter/Gather: a cluster-based approach to browsing large document collections
Cutting, D. R., Karger, D. R., Pedersen, J. O., & Tukey, J. W. (1992) · Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '92 · doi:10.1145/133160.133214 · Cited by 953
Jung, C. G. (1956) · Collected Works, Volume 14 · doi:10.1515/9781400845927 · Cited by 885 · cited 1×
“…years, however, because of the transformed societal and economic environment and the urgent need for higher level, complex problem-solving skills (Levy & Murnane, 2004), the need for those new approaches has made a strong comeback both at the K–12 level (e.g., Barron & Darling-Hammond, 2010;…”
BackgroundEmpirical Studies of Programming Knowledge
Soloway, E., & Ehrlich, K. (1984) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1984.5010283 · Cited by 877
Changing minds: computers, learning, and literacy
(2001) · Choice Reviews Online · doi:10.5860/choice.38-2855 · Cited by 816
A Review of Literature on Teaching Engineering Design Through Project‐Oriented Capstone Courses
Dutson, A. J., Todd, R. H., Magleby, S. P., & Sorensen, C. D. (1997) · Journal of Engineering Education · doi:10.1002/j.2168-9830.1997.tb00260.x · Cited by 719
The microgenetic method: A direct means for studying cognitive development.
Siegler, R. S., & Crowley, K. (1991) · American Psychologist · doi:10.1037//0003-066x.46.6.606 · Cited by 682
McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · Working group reports from ITiCSE on Innovation and technology in computer science education · doi:10.1145/572133.572137 · Cited by 626
A multi-national study of reading and tracing skills in novice programmers
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., et al. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1041624.1041673 · Cited by 575 · cited 1×
“, 2001) and elsewhere (Lister et al., 2004), and the lack of conclusive research on the factors that lead to success in introductory programming courses (Dehnadi, 2009).”
BackgroundKafai, Y. B. (2006) · Games and Culture · doi:10.1177/1555412005281767 · Cited by 575
Studying the Novice Programmer
(2013) · Psychology Press eBooks · doi:10.4324/9781315808321 · Cited by 489
Clustering Sequences with Hidden Markov Models
Smyth, P. (1996) · neural information processing systems · Cited by 483
Changing Minds: Computers, Learning, and Literacy
diSessa, A. (2000) · The MIT Press eBooks · doi:10.7551/mitpress/1786.001.0001 · Cited by 457
The Journal of Mathematical Behavior
Maher, C. A., Uptegrove, E., & Wilkinson, L. C. (2019) · ICME-13 Monographs · doi:10.1007/978-3-030-15636-7_22 · Cited by 450
Programming by choice: urban youth learning programming with scratch
Maloney, J. H., Peppler, K., Kafai, Y., Resnick, M., & Rusk, N. (2008) · Proceedings of the 39th SIGCSE technical symposium on Computer science education · doi:10.1145/1352135.1352260 · Cited by 447
The nature of learning : using research to inspire practice
Dumont, H. (2010) · Cited by 427
Epistemological Pluralism and the Revaluation of the Concrete.
Turkle, S., & Papert, S. A. (1992) · The Journal of Mathematical Behavior · Cited by 426
McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · ACM SIGCSE Bulletin · doi:10.1145/572139.572181 · Cited by 424
Karel the Robot: A Gentle Introduction to the Art of Programming
Pattis, R. E., Roberts, J., & Stehlik, M. (1994) · Cumulative Index of Computer Aided Architectural Design · Cited by 353
Methods and tools for exploring novice compilation behaviour
Jadud, M. C. (2006) · Proceedings of the second international workshop on Computing education research · doi:10.1145/1151588.1151600 · Cited by 295
Using learning analytics to assess students' behavior in open-ended programming tasks
Blikstein, P. (2011) · Proceedings of the 1st International Conference on Learning Analytics and Knowledge · doi:10.1145/2090116.2090132 · Cited by 292
Self-citationDeveloping a computer science-specific learning taxonomy
Fuller, U., Johnson, C. G., Ahoniemi, T., Cukierman, D., Hernán-Losada, I., Jackova, J., Lahtinen, E., Lewis, T. L., Thompson, D. M., Riedesel, C., et al. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1345375.1345438 · Cited by 281
Blikstein, P. (2013) · Proceedings of the Third International Conference on Learning Analytics and Knowledge · doi:10.1145/2460296.2460316 · Cited by 274
Self-citationConditions of Learning in Novice Programmers
Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1986) · Journal of Educational Computing Research · doi:10.2190/gujt-jcbj-q6qu-q9pl · Cited by 262 · cited 1×
“One path to determining these factors has been to create taxonomies of programming styles or styles in the learning of programming (Bruce et al., 2004; Perkins et al., 1986; Turkle & Papert, 1992).”
BackgroundComputer Criticism vs. Technocentric Thinking
Papert, S. (1987) · Educational Researcher · doi:10.2307/1174251 · Cited by 259
Learning to program: A phenomenographic perspective
Booth, S. (1992) · Gothenburg University Publications Electronic Archive (Gothenburg University) · Cited by 232
Developing a generalizable detector of when students game the system
Baker, R. S. J. D., Corbett, A. T., Roll, I., & Koedinger, K. R. (2008) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9045-6 · Cited by 203
Modeling how students learn to program
Piech, C., Sahami, M., Koller, D., Cooper, S., & Blikstein, P. (2012) · Proceedings of the 43rd ACM technical symposium on Computer Science Education · doi:10.1145/2157136.2157182 · Cited by 197
Self-citationSpontaneous activity in education
Montessori, M. (1965) · Cited by 177
Tinkering and gender in end-user programmers' debugging
Beckwith, L., Kissinger, C., Burnett, M., Wiedenbeck, S., Lawrance, J., Blackwell, A., & Cook, C. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124808 · Cited by 176
Learning Engineering: Design, Languages, and Experiences*
Dym, C. L. (1999) · Journal of Engineering Education · doi:10.1002/j.2168-9830.1999.tb00425.x · Cited by 171
Recognizing computational thinking patterns
Basawapatna, A., Koh, K. H., Repenning, A., Webb, D. C., & Marshall, K. S. (2011) · Proceedings of the 42nd ACM technical symposium on Computer science education · doi:10.1145/1953163.1953241 · Cited by 159
Bruce, C. S., Buckingham, L., Hynd, J., McMahon, C. A., Roggenkamp, M., & Stoodley, I. D. (2004) · Journal of Information Technology Education Research · doi:10.28945/294 · Cited by 149
Programming by choice: urban youth learning programming with scratch
Maloney, J. H., Peppler, K., Kafai, Y., Resnick, M., & Rusk, N. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1352322.1352260 · Cited by 145
Change Analysis with Evolizer and ChangeDistiller
Gall, H. C., Fluri, B., & Pinzger, M. (2008) · IEEE Software · doi:10.1109/ms.2009.6 · Cited by 143
Prospects and challenges for inquiry-based approaches to learning
Barron, B., & Darling‐Hammond, L. (2010) · Educational research and innovation · doi:10.1787/9789264086487-11-en · Cited by 133
The Advanced Montessori Method
Livingston, A., Simmonds, F., & Montessori, M. (1991) · Medical Entomology and Zoology · Cited by 113
ALEVEN, V., ROLL, I., McLAREN, B. M., & KOEDINGER, K. R. (2010) · Educational Psychologist · doi:10.1080/00461520.2010.517740 · Cited by 111
Commonsense computing: what students know before we teach (episode 1: sorting)
Simon, B., Chen, T.-Y., Lewandowski, G., McCartney, R., & Sanders, K. (2006) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1151588.1151594 · Cited by 93
Gender pluralism in problem-solving software
Burnett, M. M., Beckwith, L., Wiedenbeck, S., Fleming, S. D., Cao, J., Park, T. H., Grigoreanu, V., & Rector, K. (2011) · Interacting with Computers · doi:10.1016/j.intcom.2011.06.004 · Cited by 89
Towards the development of multimodal action based assessment
Worsley, M., & Blikstein, P. (2013) · Proceedings of the Third International Conference on Learning Analytics and Knowledge · doi:10.1145/2460296.2460315 · Cited by 68
Self-citationEpisodic Feelings and Transfer of Learning
Nemirovsky, R. (2010) · Journal of the Learning Sciences · doi:10.1080/10508406.2011.528316 · Cited by 66
Roll, I., Aleven, V. A. W. M. M., McLaren, B. M., & Koedinger, K. R. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21869-9_38 · Cited by 54
Roll, I., Aleven, V. A. W. M. M., & Koedinger, K. R. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13388-6_16 · Cited by 53
Common sense computing (episode 4): debugging
Simon, B., Bouvier, D. J., Chen, T.-Y., Lewandowski, G., McCartney, R., & Sanders, K. (2008) · Computer Science Education · doi:10.1080/08993400802114698 · Cited by 34
Mining problem-solving strategies from HCI data
Fern, X. Z., Komireddy, C., Grigoreanu, V., & Burnett, M. (2010) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1721831.1721834 · Cited by 34
A cognitive study of learning to program in introductoryprogramming courses.
Dehnadi, S. (2009) · Middlesex University Research Repository (Middlesex University Of London) · Cited by 24
Suthers, D., Verbert, K., Duval, E., & Ochôa, X. (2013) · TU/e Research Portal · Cited by 7
Wilensky, U. J., & Berland, M. (2008) · Cited by 5
(2011) · American Educational Research Association Annual Meeting
Cited by
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242
The application of AI technologies in STEM education: a systematic review from 2011 to 2021
Xu, W., & Ouyang, F. (2022) · International Journal of STEM Education · doi:10.1186/s40594-022-00377-5 · Cited by 495
Automatic feedback in online learning environments: A systematic literature review
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100027 · Cited by 440
Learning analytics to unveil learning strategies in a flipped classroom
Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017) · The Internet and Higher Education · doi:10.1016/j.iheduc.2017.02.001 · Cited by 376
Almasri, F. (2024) · Research in Science Education · doi:10.1007/s11165-024-10176-3 · Cited by 361
Metadata
- Authors
- Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., & Koller, D.
- Venue
- Journal of the Learning Sciences
- Publisher
- Informa UK Limited
- Published
- 2014-10-02
- Volume
- 23
- Issue
- 4
- Pages
- 561-599
- ISSN
- 1050-8406 (ISSN-L)
- 1532-7809
- OpenAlex
- W2123536472
- Cited by
- 267
- References
- 66
- FWCI
- 22.43
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.