CoChat
Sign Up
Cited by 51Teaching and Learning ProgrammingMeta-analysis and systematic reviewsscientometrics and bibliometrics research

A Systematic Investigation of Replications in Computing Education Research

Open access
Hao, Qiang, Smith IV, David H., Iriumi, Naitra, Tsikerdekis, Michail, Ko, Amy J.·2019·ACM Transactions on Computing Education
DOI 10.1145/3345328
Open this paper in CoChat — free accountOpen PDF

Abstract

As the societal demands for application and knowledge in computer science (CS) increase, CS student enrollment keeps growing rapidly around the world. By continuously improving the efficacy of computing education and providing guidelines for learning and teaching practice, computing education research plays a vital role in addressing both educational and societal challenges that emerge from the growth of CS students. Given the significant role of computing education research, it is important to ensure the reliability of studies in this field. The extent to which studies can be replicated in a field is one of the most important standards for reliability. Different fields have paid increasing attention to the replication rates of their studies, but the replication rate of computing education was never systematically studied. To fill this gap, this study investigated the replication rate of computing education between 2009 and 2018. We examined 2,269 published studies from three major conferences and two major journals in computing education, and found that the overall replication rate of computing education was 2.38%. This study demonstrated the need for more replication studies in computing education and discussed how to encourage replication studies through research initiatives and policy making.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3345328

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • This replication challenge exacerbates the existing acknowledged crises in replication studies within computing education research, which existing strategies such as editorial policies and a healthy environment of replication studies will not address.

    “This exacerbates the existing acknowledged crises in replication studies within computing education research [1,24], which existing strategiessuch as editorial policies and a healthy environment of replication studieswill not address [24].”

    From Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment (2024) · cites this paper as [24]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 35 · Cited by 51

Research works printed but not linked: 58

  1. Allison Elliott Tew, Brian Dorn, and Oliver Schneider. 2012. Toward a validated com- puting attitudes survey. In Proceedings of the 2012 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 135-142.
  2. Robert McCartney, Jonas Boustedt, Anna Eckerdal, Kate Sanders, and Carol Zander. 2013. Can first-year students program yet? A study revisited. In Proceedings of the 2012 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 91- 98.
  3. Colleen M. Lewis, Huda Khayrallah, and Amy Tsai. 2013. Mining data from the AP CS A exam: Patterns, non-patterns, and replication failure. In Proceedings of the 2012 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 115- 122.
  4. Elizabeth Patitsas, Michelle Craig, and Steve Easterbrook. 2013. Comparing and contrast- ing different algorithms leads to increased student learning. In Proceedings of the 2013 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 145-152.
  5. Briana B. Morrison, Brian Dorn, and Mark Guzdial. 2014. Measuring cognitive load in introductory CS: Adaptation of an instrument. In Proceedings of the 2014 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 131-138.
  6. Matthew C. Jadud and Brian Dorn. 2015. Aggregate compilation behavior: Findings and implications from 27,698 users. In Proceedings of the 2015 Annual ACM International Com- puting Education Research Conference. ACM, New York, NY, 131-139.
  7. Adam S. Carter, Christopher D. Hundhausen, and Olusola Adesope. 2015. The normal- ized programming state model: Predicting student performance in computing courses based on programming behavior. In Proceedings of the 2015 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 141-150.
  8. Stephen Cooper, Karen Wang, Maya Israni, and Sheryl Sorby. 2015. Spatial skills training in introductory computing. In Proceedings of the 2015 Annual ACM International Comput- ing Education Research Conference. ACM, New York, NY, 13-20.
  9. Briana B. Morrison, Lauren E. Margulieux, and Mark Guzdial. 2015. Subgoals, context, and worked examples in learning computing problem solving. In Proceedings of the 2015 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 21-29.
  10. Michael Hewner and Shitanshu Mishra. 2016. When everyone knows CS is the best ma- jor: Decisions about CS in an Indian context. In Proceedings of the 2016 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 3-11.
  11. Miranda C. Parker, Mark Guzdial, and Shelly Engleman. 2016. Replication, validation, and use of a language independent CS1 knowledge assessment. In Proceedings of the 2016 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 93-101.
  12. Briana B. Morrison, Adrienne Decker, and Lauren E. Margulieux. 2016. Learning loops: A replication study illuminates impact of HS courses. In Proceedings of the 2016 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 221- 230.
  13. Kathryn Cunningham, Sarah Blanchard, Barbara Ericson, and Mark Guzdial. 2017. Us- ing tracing and sketching to solve programming problems: Replicating and extending an analysis of what students draw. In Proceedings of the 2017 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 164-172.
  14. Briana B. Morrison. 2017. Dual modality code explanations for novices: Unexpected re- sults. In Proceedings of the 2017 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 226-235.
  15. Eva Marinus, Zoe Powell, Rosalind Thornton, Genevieve McArthur, and Stephen Crain. 2018. Unravelling the cognition of coding in 3-to-6-year olds: The development of an assessment tool and the relation between coding ability and cognitive compiling of syn- tax in natural language. In Proceedings of the 2018 Annual ACM International Computing Education Research Conference. ACM, New York, NY, 133-141.
  16. Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2012. All syntax errors are not equal. In Proceedings of the 2012 Annual Conference on Innovation and Technology in Com- puter Science Education. ACM, New York, NY, 75-80.
  17. Ville Isomöttönen, Ville Tirronen, and Michael Cochez. 2013. Issues with a course that emphasizes self-direction. In Proceedings of the 2013 Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, NY, 111-116.
  18. Jaime Spacco, Davide Fossati, John Stamper, and Kelly Rivers. 2013. Towards improving programming habits to create better computer science course outcomes. In Proceedings of the 2013 Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, NY, 243-248.
  19. Christopher Watson and Frederick W. B. Li. 2014. Failure rates in introductory program- ming revisited. In Proceedings of the 2014 Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, NY, 39-44.
  20. Renate Thies and Jan Vahrenhold. 2016. Back to school: Computer science unplugged in the wild. In Proceedings of the 2016 Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, NY, 118-123.
  21. Juho Leinonen, Krista Longi, Arto Klami, Alireza Ahadi, and Arto Vihavainen. 2016. Typ- ing patterns and authentication in practical programming exams. In Proceedings of the 2016 Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, NY, 160-165.
  22. Jennifer Campbell, Diane Horton, and Michelle Craig. 2016. Factors for success in on- line CS1. In Proceedings of the 2016 Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, NY, 320-325.
  23. Keith Quille and Susan Bergin. 2018. Programming: Predicting student success early in CS1. A re-validation and replication study. In Proceedings of the 2018 Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, NY, 15-20.
  24. Soohyun Nam Liao, William G. Griswold, and Leo Porter. 2018. Classroom experience report on jigsaw learning. In Proceedings of the 2018 Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, NY, 302-307.
  25. Robert McCartney and Kate Sanders. 2018. ITiCSE working groups and collaboration in the computing education community. In Proceedings of the 2018 Annual Conference on Innovation and Technology in Computer Science Education. ACM, New York, NY, 332-337.
  26. Alex D. Radermacher and Gursimran S. Walia. 2011. Investigating the effective imple- mentation of pair programming: An empirical investigation. In Proceedings of the 2011 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 655-660.
  27. Alan C. Jamieson, Lindsay H. Jamieson, and Angela C. Johnson. 2012. Application of non- programming focused Treisman-style workshops in introductory computer science. In Proceedings of the 2012 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 271-276.
  28. Laurie Murphy, Renee McCauley, and Sue Fitzgerald. 2012. Explain in plain English ques- tions: Implications for teaching. In Proceedings of the 2012 ACM SIGCSE Technical Sym- posium on Computer Science Education. ACM, New York, NY, 385-390.
  29. Kuba Karpierz and Steven A. Wolfman. 2014. Misconceptions and concept inventory questions for binary search trees and hash tables. In Proceedings of the 2014 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 109-114.
  30. Neil Christopher Charles Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014. Blackbox: A large scale repository of novice programmers' activity. In Proceedings of the 2014 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 223-228.
  31. Mark Zarb, Janet Hughes, and John Richards. 2014. Evaluating industry-inspired pair programming communication guidelines with undergraduate students. In Proceedings of the 2014 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 361-366.
  32. Paul Denny. 2015. Generating practice questions as a preparation strategy for introduc- tory programming exams. In Proceedings of the 2015 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 278-283.
  33. Arto Vihavainen, Craig S. Miller, and Amber Settle. 2015. Benefits of self-explanation in introductory programming. In Proceedings of the 2015 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 284-289.
  34. Renee McCauley, Brian Hanks, Sue Fitzgerald, and Laurie Murphy. 2015. Recursion vs. iteration: An empirical study of comprehension revisited. In Proceedings of the 2015 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 350- 355.
  35. Briana B. Morrison, Lauren E. Margulieux, Barbara Ericson, and Mark Guzdial. 2016. Subgoals help students solve Parsons problems. In Proceedings of the 2016 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 42-47.
  36. Juho Leinonen, Krista Longi, Arto Klami, and Arto Vihavainen. 2016. Automatic infer- ence of programming performance and experience from typing patterns. In Proceedings of the 2016 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 132-137.
  37. Robert Deloatch, Brian P. Bailey, and Alex Kirlik. 2016. Measuring effects of modality on perceived test anxiety for computer programming exams. In Proceedings of the 2016 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 291-296.
  38. Daniel Zingaro and Leo Porter. 2016. Impact of student achievement goals on CS1 out- comes. In Proceedings of the 2016 ACM SIGCSE Technical Symposium on Computer Science Education. New York, NY, 279-296.
  39. David Weintrop and Nathan Holbert. 2017. From blocks to text and back: Programming patterns in a dual-modality environment. In Proceedings of the 2017 ACM SIGCSE Techni- cal Symposium on Computer Science Education. ACM, New York, NY, 633-638.
  40. Yingjun Cao and Leo Porter. 2017. Evaluating student learning from collaborative group tests in introductory computing. In Proceedings of the 2017 ACM SIGCSE Technical Sym- posium on Computer Science Education. ACM, New York, NY, 99-104.
  41. Christine Alvarado, Mia Minnes, and Leo Porter. 2017. Micro-classes: A structure for improving student experience in large classes. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 21-26.
  42. Breanne K. Litts, Yasmin B. Kafai, Debora Lui, Justice Walker, and Sari Widman. 2017. Understanding high school students' reading, remixing, and writing codeable circuits for electronic textiles. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 381-386.
  43. Daniel Zingaro, Michelle Craig, Leo Porter, Brett A. Becker, Yingjun Cao, Phill Conrad, Diana Cukierman, Arto Hellas, Dastyni Loksa, and Neena Thota. 2016. Achievement goals in CS1: Replication and extension. In Proceedings of the 2018 ACM SIGCSE Tech- nical Symposium on Computer Science Education. ACM, New York, NY, 687-692.
  44. Chris Wilcox and Albert Lionelle. 2018. Quantifying the benefits of prior programming experience in an introductory computer science course. In Proceedings of the 2018 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 80-85.
  45. Onni Aarne, Petrus Peltola, Juho Leinonen, and Arto Hellas. 2018. A study of pair pro- gramming enjoyment and attendance using study motivation and strategy metrics. In Proceedings of the 2018 ACM SIGCSE Technical Symposium on Computer Science Educa- tion. ACM, New York, NY, 759-764.
  46. Austin Cory Bart, Dennis Kafura, Clifford A. Shaffer, and Eli Tilevich. 2018. Reconciling the promise and pragmatics of enhancing computing pedagogy with data science. In Proceedings of the 2018 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, New York, NY, 1029-1034.
  47. Uolevi Nikula, Orlena Gotel, and Jussi Kasurinen. 2011. A motivation guided holistic re- habilitation of the first programming course. ACM Transactions on Computing Education 11, 4, 24.
  48. Cynthia Bailey Lee, Saturnino Garcia, and Leo Porter. 2013. Can peer instruction be ef- fective in upper-division computer science courses? ACM Transactions on Computing Ed- ucation 13, 3, 12.
  49. Adam S. Carter, Christopher D. Hundhausen, and Olusola Adesope. 2017. Blending mea- sures of programming and social behavior into predictive models of student achievement in early computing courses. ACM Transactions on Computing Education 17, 3, 12.
  50. Neil C. C. Brown, Sue Sentance, Tom Crick, and Simon Humphreys. 2014. Restart: The resurgence of computer science in UK schools. ACM Transactions on Computing Education 14, 2 (2014), 9.
  51. Ed Yong. 2012. Bad copy. Nature 485, 7398 (2012), 298.
  52. Association for Computing Machinery. 2019. Guide to Reviewing Papers. Retrieved July 24, 2019 from https://chi2019. acm.org/guide-to-reviewing-papers/.
  53. Laura Toma and Jan Vahrenhold. 2018. Self-efficacy, cognitive load, and emotional reactions in collaborative algo- rithms labs-A case study. In Proceedings of the 2018 ACM Conference on International Computing Education Research. ACM, New York, NY, 1-10.
  54. Andreas Stefik and Susanna Siebert. 2013. An empirical investigation into programming language syntax. ACM Trans- actions on Computing Education 13, 4 (2013), 19.
  55. Anders Berglund, Mats Daniels, and Arnold Pears. 2006. Qualitative research projects in computing education re- search: An overview. In Proceedings of the 8th Australasian Conference on Computing Education-Volume 52. 25-33.
  56. SIGCSE Technical Symposium. 2019. SIGCSE 2019-Call For Participation. Retrieved July 24, 2019 from https:// sigcse2019.sigcse.org/info/cfp.html#papers.
  57. Association for Computing Machinery. 2018. Artifact Review and Badging. Retrieved July 24, 2019 from https://www. acm.org/publications/policies/artifact-review-badging.
  58. International Computing Education Research Conference. 2018. ICER'18-Call for Participation. Retrieved July 24, 2019 from https://icer.acm.org/icer-2018/icer18-call-for-participation/.

Authors
Hao, Q., Smith IV, D. H., Iriumi, N., Tsikerdekis, M., & Ko, A. J.
Affiliations
  • Western Washington University (US)
  • Western Washington University, Bellingham, WA
  • University of Washington (US)
ORCID
Venue
ACM Transactions on Computing Education
Publisher
Association for Computing Machinery (ACM)
Published
2019-08-17
Volume
19
Issue
4
Pages
1-18
ISSN
1946-6226 (ISSN-L)
DOI
10.1145/3345328
OpenAlex
W2969260140
Cited by
51
References
35
Citations by year
2026: 4 · 2025: 2 · 2024: 9 · 2023: 7 · 2022: 12 · 2021: 7 · 2020: 9 · 2019: 1
FWCI
7.85
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
ACM Transactions on Computing Education (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
Keywords
replication rate, replication study, computing education research, research reliability, research reproducibility, science policy, research initiatives

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review