Replication in computing education research: researcher attitudes and experiences
Open accessAbstract
Replicability is a core principle of the scientific method. However, several scientific disciplines have suffered crises in confidence caused, in large part, by attitudes toward replication. This work reports on the value the computing education research community associates with studies that aim to replicate, reproduce or repeat earlier research. The results were obtained from a survey of 73 computing education researchers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This replication challenge exacerbates the existing acknowledged crises in replication studies within computing education research, which existing strategies such as editorial policies and a healthy environment of replication studies will not address.
“This exacerbates the existing acknowledged crises in replication studies within computing education research [1,24], which existing strategiessuch as editorial policies and a healthy environment of replication studieswill not address [24].”
From Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment (2024) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1× · see citing works
“For example, a recent study in psychology claimed that only 40% of studies were replicable [10].”
ResultMultiple comparisons among means
Dunn, O. J. (1961) · Journal of the American Statistical Association · doi:10.1080/01621459.1961.10482090 · Cited by 5,006 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1× · see citing works
“, biology [7], health care [19], political science [15], and computational science [37]).”
BackgroundContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · see citing works
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · see citing works
Statistical significance in psychological research.
Lykken, D. T. (1968) · Psychological bulletin · doi:10.1037/h0026141 · Cited by 1,363 · see citing works
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · see citing works
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · see citing works
The case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931 · see citing works
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 818 · see citing works
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750 · see citing works
Failure rates in introductory programming revisited
Watson, C., & Li, F. W. B. (2014) · Proceedings of the 2014 conference on Innovation & technology in computer science education - ITiCSE '14 · doi:10.1145/2591708.2591749 · Cited by 639 · see citing works
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · see citing works
Comment on “Estimating the reproducibility of psychological science.”
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016) · Science · doi:10.1126/science.aad7243 · Cited by 474 · see citing works
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · see citing works
Most published research findings are false-but a little replication goes a long way.
Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. W. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040028 · Cited by 341 · see citing works
Educational Data Mining and Learning Analytics in Programming
Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., et al. (2015) · doi:10.1145/2858796.2858798 · Cited by 332 · see citing works
Self-citationReplication research's disturbing trend
Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2007) · Journal of Business Research · doi:10.1016/j.jbusres.2006.12.003 · Cited by 271 · see citing works
Exploring Machine Learning Methods to Automatically Identify Students in Need of Assistance
Ahadi, A., Lister, R., Haapala, H., & Vihavainen, A. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787717 · Cited by 234 · see citing works
Self-citationMetascience could rescue the ‘replication crisis’
Schooler, J. W. (2014) · Nature · doi:10.1038/515009a · Cited by 226 · see citing works
Porter, L., Lee, C. B., Simon, B., & Zingaro, D. (2011) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2016911.2016923 · Cited by 197 · see citing works
Response to Comment on “Estimating the reproducibility of psychological science”
Anderson, C., Bahník, Š., Barnett‐Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., et al. (2016) · Science · doi:10.1126/science.aad9163 · Cited by 189 · see citing works
Valentine, D. W. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1028174.971391 · Cited by 163 · see citing works
Naturally occurring data as research instrument
Lister, R., Clear, T., Simon, Bouvier, D. J., Carter, P., Eckerdal, A., Jacková, J., Lopez, M., McCartney, R., Robbins, P., et al. (2010) · ACM SIGCSE Bulletin · doi:10.1145/1709424.1709460 · Cited by 127 · see citing works
Automatic Inference of Programming Performance and Experience from Typing Patterns
Leinonen, J., Longi, K., Klami, A., & Vihavainen, A. (2016) · Proceedings of the 47th ACM Technical Symposium on Computing Science Education · doi:10.1145/2839509.2844612 · Cited by 103 · see citing works
Theoretical underpinnings of computing education research
Malmi, L., Sheard, J., Simon, Bednarik, R., Helminen, J., Kinnunen, P., Korhonen, A., Myller, N., Sorva, J., & Taherkhani, A. (2014) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2632320.2632358 · Cited by 102 · see citing works
Self-citationDo we know how difficult the rainfall problem is?
Seppälä, O., Ihantola, P., Isohanni, E., Sorva, J., & Vihavainen, A. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2828963 · Cited by 93 · see citing works
Self-citationCharacterizing research in computing education
Malmi, L., Sheard, J., Simon, Bednarik, R., Helminen, J., Korhonen, A., Myller, N., Sorva, J., & Taherkhani, A. (2010) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1839594.1839597 · Cited by 89 · see citing works
Self-citationA Methodological Review of Computer Science Education Research
Randolph, J., Julnes, G., Sutinen, E., & Lehman, S. (2008) · Journal of Information Technology Education Research · doi:10.28945/183 · Cited by 88 · see citing works
A fresh look at novice programmers' performance and their teachers' expectations
Utting, I., Tew, A. E., McCracken, M., Thomas, L., Bouvier, D. J., Frye, R., Paterson, J. H., Caspersen, M. E., Kolikant, Y. B., Sorva, J., et al. (2013) · doi:10.1145/2543882.2543884 · Cited by 79 · see citing works
Muma, J. R. (1993) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3605.927 · Cited by 62 · see citing works
McCauley, R. A., Hanks, B., Fitzgerald, S., & Murphy, L. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677227 · Cited by 60 · see citing works
Introduction to the Special Section: Data, Data, Everywhere . . . Especially in My File Drawer
Spellman, B. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611432124 · Cited by 58 · see citing works
Why (or why not), when and how to replicate research
Mackey, A. (2012) · Lancaster EPrints (Lancaster University) · Cited by 46 · see citing works
An exploration of error quotient in multiple contexts
Petersen, A., Spacco, J. W., & Vihavainen, A. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2828966 · Cited by 41 · see citing works
Self-citationA (Updated) Review of Empiricism at the SIGCSE Technical Symposium
Al-Zubidy, A., Carver, J. C., Heckman, S. S., & Sherriff, M. S. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844601 · Cited by 37 · see citing works
Skill Progression in Scratch Revisited
Matias, J. N., Dasgupta, S., & Hill, B. M. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858349 · Cited by 31 · see citing works
Can first-year students program yet?
McCartney, R., Boustedt, J., Eckerdal, A., Sanders, K., & Zander, C. (2013) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2493394.2493412 · Cited by 22 · see citing works
Emergence of computing education as a research discipline
Simon, N. P. (2015) · Aaltodoc (Aalto University) · Cited by 20 · see citing works
Replication and Non-Quantitative Research
Golden, M. A. (1995) · PS Political Science & Politics · doi:10.2307/420313 · Cited by 18 · see citing works
Valuing computer science education research?
Clear, T. (2006) · doi:10.1145/1315803.1315806 · Cited by 17 · see citing works
Thomas, L., Eckerdal, A., McCartney, R., Moström, J. E., Sanders, K., & Zander, C. (2014) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2632320.2632353 · Cited by 13 · see citing works
Proceedings of the Fourth Finnish/Baltic Sea Conference on Computer Science Education
Korhonen, A., & Malmi, L. (2004) · Cited by 9 · see citing works
Self-citationPublication Bias in the Computer Science Education Research Literature
Randolph, J., & Bednarik, R. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.3217/jucs-014-04-0575 · Cited by 8 · see citing works
Replication and Non-Quantitative Research
Golden, M. A. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057723 · Cited by 6 · see citing works
Replicating a Validated CS1 Assessment (Abstract Only)
Parker, M. C., & Guzdial, M. (2016) · doi:10.1145/2839509.2850559 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- O. S. Gómez, N. Juristo, and S. Vegas. Replication, reproduction and re-analysis: Three ways for verifying experimental findings. In Proceedings of the 1st international workshop on replication in empirical software engineering research (RESER 2010), Cape Town, South Africa, 2010.
Cited by
Sorted by citations
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199 · see citing works
“5 let lst = map2 (fun (x, y) -> x + y) [1;2;3] [4;5;6]”
BackgroundInfluentialPage, M. J., Moher, D., Fidler, F. M., Higgins, J. P. T., Brennan, S., Haddaway, N., Hamilton, D. G., Kanukula, R., Karunananthan, S., Maxwell, L. J., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01670-0 · Cited by 55 · see citing works
Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment
Richards, M., Waugh, K., Slaymaker, M., Petre, M., Woodthorpe, J., & Gooch, D. (2024) · ACM Transactions on Computing Education · doi:10.1145/3633287 · Cited by 52 · see citing works
“This exacerbates the existing acknowledged crises in replication studies within computing education research [ 1 , 24 ], which existing strategies – such as editorial policies and a healthy environment of replication studies – will not address [ 24 ].”
BackgroundA Systematic Investigation of Replications in Computing Education Research
Hao, Q., Smith IV, D. H., Iriumi, N., Tsikerdekis, M., & Ko, A. J. (2019) · ACM Transactions on Computing Education · doi:10.1145/3345328 · Cited by 51 · see citing works
“Many researchers have expressed hesitation in conducting replication studies because they believed that (1) the value of original works is more significant than replication studies or (2) original works carry more weight in determining promotion [17].”
BackgroundInfluentialAchievement Goals in CS1: Replication and Extension
Zingaro, D., Craig, M., Porter, L., Becker, B. A., Cao, Y., Conrad, P., Cukierman, D., Hellas, A., Loksa, D., & Thota, N. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159452 · Cited by 35 · see citing works
“Few studies in CS education seek to replicate existing work [1].”
BackgroundSelf-citationHeckman, S. S., Carver, J. C., Sherriff, M. S., & Al-Zubidy, A. (2021) · ACM Transactions on Computing Education · doi:10.1145/3470652 · Cited by 32 · see citing works
“For a community to systematically advance knowledge, its membersmust be able to understand publishedwork thoroughly enough to perform replications, conduct meta-analyses, and build theories [4, 8, 12, 16, 19, 21, 24, 29, 41, 42, 59].”
InfluentialARE REPLICATION STUDIES INFREQUENT BECAUSE OF NEGATIVE ATTITUDES?
McManus, K. (2021) · Studies in Second Language Acquisition · doi:10.1017/s0272263121000838 · Cited by 31 · see citing works
“In a survey of 73 researchers in computer science education (Ahadi et al., 2016), respondents agreed that “original studies are more prestigious than replication studies” (p.”
BackgroundMethodA Study of Keystroke Data in Two Contexts
Edwards, J., Leinonen, J., & Hellas, A. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366863 · Cited by 24 · see citing works
“in other contexts [14], there are also issues with how replication studies are valued and consequently preferred [1].”
BackgroundSelf-citationMalmi, L., Sheard, J., Kinnunen, P., Simon, N. P., & Sinclair, J. E. (2022) · ACM Transactions on Computing Education · doi:10.1145/3530221 · Cited by 25 · see citing works
Spatial Skills and Demographic Factors in CS1
Ly, A., Parkinson, J., Cutts, Q. I., Liut, M., & Petersen, A. (2021) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3488042.3488049 · Cited by 23 · see citing works
“This study does not invalidate that work, as noted earlier, but it does highlight the need for replication in computing education [1].”
ResultSelf-citationVachon, B., Curran, J., Karunananthan, S., Brehaut, J., Graham, I. D., Moher, D., Sales, A. E., Straus, S. E., Fiander, M., Paprica, P. A., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.09.027 · Cited by 22 · see citing works
Learning programming, syntax errors and institution-specific factors
Ahadi, A., Lister, R., Lal, S., & Hellas, A. (2018) · Australasian Computing Education Conference · doi:10.1145/3160489.3160490 · Cited by 20 · see citing works
“Such an observation would be in line with the results of a recent study on attitudes towards replication, where one of the observations was that novel results are more valuable than confirming old studies [1].”
BackgroundResultSelf-citationPromoting intentions to persist in computing: an examination of six years of the EarSketch program
Wanzer, D. L., McKlin, T., Freeman, J., Magerko, B. S., & Lee, T. (2020) · Computer Science Education · doi:10.1080/08993408.2020.1714313 · Cited by 19 · see citing works
Vachon, B., Curran, J., Karunananthan, S., Brehaut, J., Graham, I. D., Moher, D., Sales, A. E., Straus, S. E., Fiander, M., Paprica, P. A., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.07.006 · Cited by 19 · see citing works
Are You Open? A Content Analysis of Transparency and Openness Guidelines in HCI Journals
Ballou, N., Warriar, V. R., & Deterding, S. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445584 · Cited by 18 · see citing works
“meated human-computer interaction (HCI) [8, 12, 55, 58] and connected fields, including computer science [7], health informatics [9], graphics and visualization [4, 30], and computing education [1].”
BackgroundBecker, B. A., Settle, A., Luxton-Reilly, A., Morrison, B. B., & Laxer, C. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3432448 · Cited by 17 · see citing works
“Others survey the community to identify interests [10], attitudes towards research [2], or to identify trends such as making learningmore appropriate from gender, diversity, accessibility and inclusion standpoints [8, 13, 19].”
BackgroundMcGill, M. M. (2019) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3364510.3364528 · Cited by 17 · see citing works
“To help fix this, there are calls for replication studies be explicitly called for in Calls for Papers of conference and journals with the acknowl-edgement of the value of these types of studies called out [1, 10].”
BackgroundResultInfluentialSurfacing Inequities and Their Broader Implications in the CS Education Research Community
McGill, M. M., Davis, S., & Reyes, J. (2022) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3501385.3543969 · Cited by 14 · see citing works
A Study of Pair Programming Enjoyment and Attendance using Study Motivation and Strategy Metrics
Aarne, O., Peltola, P., Leinonen, J., & Hellas, A. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159493 · Cited by 14 · see citing works
“Finally, the results presented in this work provide additional evidence on the recent discussion in computing education research that has highlighted the importance of replication studies [1, 16] for examining whether results from one context generalize to others”
ResultSelf-citationRovshenov, A., & Sarsar, F. (2023) · Journal of Educational Technology and Online Learning · doi:10.31681/jetol.1201010 · Cited by 12 · see citing works
“In addition, reviewing the limitations of the studies is important for the reproducibility of similar research (Ahadi, Hellas, Ihantola, Korhonen & Petersen, 2016).”
MethodSystematic reviewers' perspectives on replication of systematic reviews: A survey
Nguyen, P., McKenzie, J. E., Hamilton, D. G., Moher, D., Tugwell, P., Fidler, F. M., Haddaway, N., Higgins, J. P. T., Kanukula, R., Karunananthan, S., et al. (2023) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.12009 · Cited by 12 · see citing works
“This sentiment has been widely documented among researchers, funding agencies, and journal editors, regardless of the type of research being replicated [24, 25, 31 – 34].”
Decker, A., & McGill, M. M. (2019) · Education Sciences · doi:10.3390/educsci9020069 · Cited by 12 · see citing works
“Due to the newness of computing education, replication studies in educational research are still lacking and the data that are reported are incomplete [14,16,17].”
BackgroundiSnap: Evolution and Evaluation of a Data-Driven Hint System for Block-Based Programming
Marwan, S., & Price, T. (2022) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2022.3223577 · Cited by 11 · see citing works
“[43] published a review on replication in computing education research discussing different reasons why replication studies are needed, such as to “verify earlier work,” or to “elucidate which other factors may be relevant” or “rule out the effects of site-specific factors.”
BackgroundLaunching Registered Report Replications in Computer Science Education Research
Brown, N. C. C., Marinus, E., & Cheuoua, A. H. (2022) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3501385.3543971 · Cited by 10 · see citing works
“As it stands, computer science education researchers estimate that only 30% of work is likely to replicate [1]; we believe that with structural changes to the publication process such as introducing registered reports, we can increase this number together.”
BackgroundInfluentialWhat are We Asking our Students? A Literature Map of Student Surveys in Computer Science Education
Bernuy, A. Z., & Harrington, B. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387383 · Cited by 9 · see citing works
“Computer science education has struggled with providing validated tools for the research community [35], and use of unvalidated and biased tools could lead Session: Understanding Our Students to a loss of confidence in the research community and ultimately a replication crisis [1].”
BackgroundInvestigating the Applicability of the Normalized Programming State Model to BlueJ Programmers
Richards, B., & Hunt, A. (2018) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3279720.3279731 · Cited by 8 · see citing works
Search of the Emotional Design Effect in Programming Revised
Nurminen, M., Leppänen, L., Väätäjä, H., & Ihantola, P. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66610-5_39 · Cited by 7 · see citing works
Self-citationExploring the Interplay of Achievement Goals, Self-Efficacy, Prior Experience and Course Achievement
Pesonen, H., Leinonen, J., Haaranen, L., & Hellas, A. (2023) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3610969.3611178 · Cited by 6 · see citing works
“In essence, our study explores topics of prior studies in new contexts and with new data [1, 25].”
BackgroundSelf-citationThe Evolution of Computing Education Research: A Meta-Analytic Perspective
Malmi, L., Sinclair, J. E., Sheard, J., Simon, & Kinnunen, P. (2023) · Springer eBooks · doi:10.1007/978-3-031-25336-2_4 · Cited by 5 · see citing works
Where's the Data? Finding and Reusing Datasets in Computing Education
Kiesler, N., Impagliazzo, J., Biernacka, K., Kapoor, A., Kazmi, Z., Ramagoni, S. G., Sane, A., Tran, K., Taneja, S., & Wu, Z. (2024) · doi:10.1145/3598579.3689378 · Cited by 4 · see citing works
Training Computing Educators to Become Computing Education Researchers
Carver, J. C., Heckman, S. S., & Sherriff, M. S. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499297 · Cited by 4 · see citing works
20-Years Later: A Replication Study on Teaching CS1 Concepts
Garcia, R., & Craig, M. (2025) · ACM Transactions on Computing Education · doi:10.1145/3730405 · Cited by 3 · see citing works
Delving into gender gap perceptions of computing students: A replication study.
Lacave, C., Molina, A. I., García‐Holgado, A., & González, C. S. G. (2021) · Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'21) · doi:10.1145/3486011.3486428 · Cited by 3 · see citing works
“However, despite replication studies helps to confirm or discard expectations, there are few replication studies in computer science [13], mainly because original works are believed to have more impact, to be more prestigious, and to be more likely to be accepted for publication [1].”
BackgroundAre a Static Analysis Tool Study's Findings Static? A Replication
Liu, D., Calver, J., & Craig, M. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653545 · Cited by 2 · see citing works
Bai, G. R., Jiang, Z., Price, T., & Stolee, K. T. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671102 · Cited by 2 · see citing works
Decades of Striving for Pedagogical and Technological Alignment
Haaranen, L., Ahrenberg, L., & Hellas, A. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631809 · Cited by 2 · see citing works
Self-citationMarwan, S. (2021) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 1 · see citing works
Brown, N. C. C., Cheuoua, A. H., & Marinus, E. (2022) · Computer Science Education · doi:10.1080/08993408.2022.2090114
“A survey of computer science education researchers by Ahadi et al. (2016) confirmed that this perception is also present in our field, and that researchers believed replications to be harder to publish than original studies.”
Growing an Inclusive Community of K-12 CS Education Researchers
Davis, S., & McGill, M. M. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569725
"Medium-n studies" in computing education conferences
Guerzhoy, M. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631854
McGill, M. M., Gransbury, I., & Heckman, S. S. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893479
Meegammana, P., Suriyaarachchi, H., Denny, P. C., & Nanayakkara, S. C. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729037
“In computing education, the lack of replication studies often hinders the field’s progress, resulting in approaches that can be challenging to adopt in classroom practice across various disciplines [1, 14, 22].”
Page, M. J., Moher, D., Fidler, F. M., Higgins, J. P. T., Brennan, S. E., Haddaway, N. R., Hamilton, D. G., Kanukula, R., Karunananthan, S., Maxwell, L. J., et al. (2021) · Figshare · doi:10.6084/m9.figshare.14442918
Does Peer Review Need to Change? A Panel on Reporting Standards and Checklists in the Age of AI
Duchowski, A. T., Stefik, A., Ralph, P., Dix, A. J., Krejtz, K., Myers, B. A., Jorge, J., & Wehbe, R. R. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3790074
“Researchers have raised concerns about the quality of research in Computer Science Education [10] and specifically the lack of replication [1].”
Towards a Comprehensive Understanding of Replication in Computing Education
Garcia, R., Bernuy, A. Z., Bouvier, D. J., Heckman, S. S., Kern, B., Krause-Levy, S., Liut, M., Nasir, U., Pan, Y., & Sperling, J. (2026) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3803401.3812048
“Other studies have also investigated replication studies in CER, such as a 2016 study [2] that surveyed Computing Education researchers on their experiences and perceptions with replication studies.”
InfluentialRevisiting the Replication Study Design Used in Computing Education Research
Garcia, R., Lovellette, E., Wu, X., & Bernuy, A. Z. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811649
Marx, E., Witt, C., Engelhardt, M., Grillenberger, A., & Sprenger, M. T. (2026) · Workshop in Primary and Secondary Computing Education · doi:10.1145/3801749.3801775
“Empirical evidence indicates that such conditions are rarely met in CER; researchers estimate that only about one-third of published studies in the field would be replicable [1].”
- Authors
- Ahadi, A., Hellas, A., Ihantola, P., Korhonen, A., & Petersen, A.
- Affiliations
- University of Technology Sydney (AU)
- University of Helsinki (FI)
- Tampere University of Technology (FI)
- Aalto University (FI)
- University of Toronto (CA)
- Venue
- Proceedings of the 16th Koli Calling International Conference on Computing Education Research
- Publisher
- ACM
- Published
- 2016-11-24
- Pages
- 2-11
- OpenAlex
- W2549453003
- Cited by
- 48
- References
- 49
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 4 · 2023: 8 · 2022: 6 · 2021: 10 · 2020: 5 · 2019: 4 · 2018: 4 · 2017: 1
- FWCI
- 2.39
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- Koli Calling International Conference on Computing Education Research (publishedVersion)
- UTS ePRESS (University of Technology Sydney) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Educational Research and Methods
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- replication, reproducibility, computing education research, research credibility, scientific method
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
