The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies
Open accessAbstract
Educational data mining and learning analytics promise better understanding of student behavior and knowledge, as well as new information on the tacit factors that contribute to student actions. This knowledge can be used to inform decisions related to course and tool design and pedagogy, and to further engage students and guide those at risk of failure. This working group report provides an overview of the body of knowledge regarding the use of educational data mining and learning analytics focused on the teaching and learning of programming. In a literature survey on mining students' programming processes for 2005-2015, we observe a significant increase in work related to the field. However, the majority of the studies focus on simplistic metric analysis and are conducted within a single institution and a single course. This indicates the existence of further avenues of research and a critical need for validation and replication to better understand the various contributing factors and the reasons why certain results occur. We introduce a novel taxonomy to analyse replicating studies and discuss the importance of replicating and reproducing previous work. We describe what is the state of the art in collecting and sharing programming data. To better understand the challenges involved in replicating or reproducing existing studies, we report our experiences from three case studies using programming data. Finally, we present a discussion of future directions for the education and research community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Guidelines for conducting and reporting case study research in software engineering
Runeson, P., & Höst, M. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9102-8 · Cited by 3,862 · cited 4×
“For this purpose, we adapted the quality criteria for research proposed by Downs & Black [34] and Runeson & H¨ost [94] and added them to the data extraction form.”
MethodInfluentialMcCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · Working group reports from ITiCSE on Innovation and technology in computer science education · doi:10.1145/572133.572137 · Cited by 626
Web-CAT: automatically grading programming assignments
Edwards, S., & Pérez-Quiñones, M. A. (2008) · Annual Conference on Innovation and Technology in Computer Science Education · doi:10.1145/1597849.1384371 · Cited by 314
Self-citationMethods and tools for exploring novice compilation behaviour
Jadud, M. C. (2006) · Proceedings of the second international workshop on Computing education research · doi:10.1145/1151588.1151600 · Cited by 295
37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data
Altadmri, A., & Brown, N. C. C. (2015) · Proceedings of the 46th ACM Technical Symposium on Computer Science Education · doi:10.1145/2676723.2677258 · Cited by 261
Web-CAT: automatically grading programming assignments
Edwards, S. H., & Perez-Quinones, M. A. (2008) · Proceedings of the 13th annual conference on Innovation and technology in computer science education · doi:10.1145/1384271.1384371 · Cited by 256
Self-citationReplicability is not Reproducibility:Nor is it Good Science
Drummond, C. (2009) · Cited by 227
On automated grading of programming assignments in an academic institution
Cheang, B., Kurnia, A., Lim, A., & Oon, W.-C. (2003) · Computers & Education · doi:10.1016/s0360-1315(03)00030-7 · Cited by 227
Educational Data Mining: Applications and Trends
(2014) · Studies in Computational Intelligence · doi:10.1007/978-3-319-02738-8 · Cited by 204
Modeling how students learn to program
Piech, C., Sahami, M., Koller, D., Cooper, S., & Blikstein, P. (2012) · Proceedings of the 43rd ACM technical symposium on Computer Science Education · doi:10.1145/2157136.2157182 · Cited by 197
An analysis of patterns of debugging among novice computer science students
Ahmadzadeh, M., Elliman, D., & Higgins, C. (2005) · Proceedings of the 10th annual SIGCSE conference on Innovation and technology in computer science education · doi:10.1145/1067445.1067472 · Cited by 185
Blackbox: a large scale repository of novice programmers' activity
Brown, N. C. C., Kölling, M., McCall, D., & Utting, I. (2014) · Proceedings of the 45th ACM technical symposium on Computer science education · doi:10.1145/2538862.2538924 · Cited by 179
No tests required: comparing traditional and dynamic predictors of programming success
Watson, C., Li, F. W. B., & Godwin, J. L. (2014) · Proceedings of the 45th ACM technical symposium on Computer science education · doi:10.1145/2538862.2538930 · Cited by 175
A First Look at Novice Compilation Behaviour Using BlueJ
Jadud, M. C. (2005) · Computer Science Education · doi:10.1080/08993400500056530 · Cited by 169
Spacco, J., Hovemeyer, D., Pugh, W., Emad, F., Hollingsworth, J. K., & Padua-Perez, N. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140131 · Cited by 163
Self-citationAutomated assessment and experiences of teaching programming
Higgins, C. A., Gray, G., Symeonidis, P., & Tsintsifas, A. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163410 · Cited by 163
Enhancing syntax error messages appears ineffectual
Denny, P. C., Luxton-Reilly, A., & Carpenter, D. (2014) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2591708.2591748 · Cited by 154
All syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144
How (not) to introduce badges to online exercises
Haaranen, L., Ihantola, P., Hakulinen, L., & Korhonen, A. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538921 · Cited by 143
Self-citationAutonomously Generating Hints by Inferring Problem Solving Policies
Piech, C., Sahami, M., Huang, J., & Guibas, L. J. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2724668 · Cited by 139
UUhistle: a software tool for visual program simulation
Sorva, J., & Sirkiä, T. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930471 · Cited by 136
Understanding the syntax barrier for novices
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999807 · Cited by 133
Coarse-grained detection of student frustration in an introductory programming course
Rodrigo, M. M. T., & Baker, R. S. (2009) · doi:10.1145/1584322.1584332 · Cited by 133
Sirkiä, T., & Sorva, J. (2012) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2401796.2401799 · Cited by 128
How do students solve parsons programming problems? an analysis of interaction traces
Helminen, J., Ihantola, P., Karavirta, V., & Malmi, L. (2012) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2361276.2361300 · Cited by 117
Self-citationComparing effective and ineffective behaviors of student programmers
Edwards, S. H., Snyder, J., Pérez-Quiñones, M. A., Allevato, A., Kim, D., & Tretola, B. (2009) · doi:10.1145/1584322.1584325 · Cited by 117
Self-citationCodeWrite: supporting student-driven practice of java
Denny, P. C., Luxton-Reilly, A., Tempero, E. D., & Hendrickx, J. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953299 · Cited by 113
Affective and behavioral predictors of novice programmer achievement
Rodrigo, M. M. T., Baker, R. S., Jadud, M. C., Amarra, A. C. M., Dy, T., Espejo-Lahoz, M. B. V., Lim, S. A. L., Pascua, S. A. M. S., Sugay, J. O., & Tabanao, E. S. (2009) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1562877.1562929 · Cited by 103
Characterizing research in computing education: a preliminary analysis of the literature
Malmi, L., Sheard, J., Simon, Bednarik, R., Helminen, J., Korhonen, A., Myller, N., Sorva, J., & Taherkhani, A. (2010) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1839594.1839597 · Cited by 96
Self-citationPredicting at-risk novice Java programmers through the analysis of online protocols
Tabanao, E. S., Rodrigo, M. M. T., & Jadud, M. C. (2011) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2016911.2016930 · Cited by 95
An analysis of patterns of debugging among novice computer science students
Ahmadzadeh, M., Elliman, D., & Higgins, C. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1151954.1067472 · Cited by 93
An exploration of novice compilation behaviour in BlueJ
Jadud, M. C. (2021) · Kent Academic Repository (University of Kent) · doi:10.22024/unikent/01.02.86458 · Cited by 90
Investigating novice programming mistakes: educator beliefs vs. student data
Brown, N. C. C., & Altadmri, A. (2014) · Proceedings of the tenth annual conference on International computing education research · doi:10.1145/2632320.2632343 · Cited by 89
An open platform for managing short programming exercises
Papancea, A., Spacco, J. W., & Hovemeyer, D. (2013) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2493394.2493401 · Cited by 86
Self-citationOn the use of resubmissions in automatic assessment systems
Karavirta, V., Korhonen, A., & Malmi, L. (2006) · Computer Science Education · doi:10.1080/08993400600912426 · Cited by 84
Self-citationTowards improving programming habits to create better computer science course outcomes
Spacco, J. W., Fossati, D., Stamper, J. C., & Rivers, K. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2465594 · Cited by 81
Self-citationA fresh look at novice programmers' performance and their teachers' expectations
Utting, I., Tew, A. E., McCracken, M., Thomas, L., Bouvier, D. J., Frye, R., Paterson, J. H., Caspersen, M. E., Kolikant, Y. B., Sorva, J., et al. (2013) · doi:10.1145/2543882.2543884 · Cited by 79
Analyzing Student Work Patterns Using Programming Exercise Data
Spacco, J. W., Denny, P. C., Richards, B., Babcock, D., Hovemeyer, D., Moscola, J., & Duvall, R. C. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677297 · Cited by 79
Self-citationIncreasing Adoption of Smart Learning Content for Computer Science Education
Brusilovsky, P. L., Edwards, S. H., Kumar, A. N., Malmi, L., Benotti, L., Buck, D., Ihantola, P., Prince, R., Sirkiä, T., Александрович, С. С., et al. (2014) · doi:10.1145/2713609.2713611 · Cited by 75
Self-citationAn Empirical Study of Iterative Improvement in Programming Assignments
Pettit, R., Homer, J., Gee, R., Mengel, S. A., & Starbuck, A. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677279 · Cited by 64
Identifying challenging CS1 concepts in a large problem dataset
Cherenkova, Y., Zingaro, D., & Petersen, A. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538966 · Cited by 63
Self-citationGeneration of problems, answers, grade, and feedback---case study of a fully automated tutor
Kumar, A. N. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163408 · Cited by 60
A fast measure for identifying at-risk students in computer science
Falkner, N., & Falkner, K. E. K. (2012) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2361276.2361288 · Cited by 58
A mobile learning application for parsons problems with automatic feedback
Karavirta, V., Helminen, J., & Ihantola, P. (2012) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2401796.2401798 · Cited by 55
Self-citationURI Online Judge Academic: A tool for algorithms and programming classes
Bez, J. L., Tonin, N. A., & Rodegheri, P. R. (2014) · International Conference on Computer Science & Education (ICCSE) · doi:10.1109/iccse.2014.6926445 · Cited by 51
Impacts of adaptive feedback on teaching test-driven development
Buffardi, K., & Edwards, S. H. (2013) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2445196.2445287 · Cited by 49
Self-citationNorris, C., Barry, F. E., Fenwick, J. B., et al. (2008) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1384271.1384284 · Cited by 48
How can software metrics help novice programmers
Cardell‐Oliver, R. (2011) · UWA Profiles and Research Repository (UWA) · Cited by 47
How novices tackle their first lines of code in an IDE: analysis of programming session traces
Vihavainen, A., Helminen, J., & Ihantola, P. (2014) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2674683.2674692 · Cited by 47
Self-citationAnother look at the behaviors of novice programmers
Fenwick, J. B., Norris, C., Barry, F. E., Rountree, J., Spicer, C. J., & Cheek, S. D. (2009) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1508865.1508973 · Cited by 47
Code hunt: experience with coding contests at scale
Bishop, J., Horspool, R. N., Xie, T., Tillmann, N., & de Halleux, J. (2015) · doi:10.5555/2819009.2819072 · Cited by 45
Facilitating code-writing in PI classes
Zingaro, D., Cherenkova, Y., Karpova, O., & Petersen, A. (2013) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2445196.2445369 · Cited by 44
Self-citationIdentification of programmers from typing patterns
Longi, K., Leinonen, J., Nygren, H., Salmi, J., Arto Klami, A., & Vihavainen, A. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2828960 · Cited by 44
Self-citationKiesmüller, U. (2008) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1595356.1595361 · Cited by 42
Falkner, N., Vivian, R., Piper, D., & Falkner, K. E. K. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538896 · Cited by 41
An exploration of error quotient in multiple contexts
Petersen, A., Spacco, J. W., & Vihavainen, A. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2828966 · Cited by 41
Self-citationOn the differences between correct student solutions
Luxton-Reilly, A., Denny, P. C., Kirk, D., Tempero, E. D., & Yu, S.-Y. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2462505 · Cited by 39
Running students' software tests against each others' code
Edwards, S. H., Shams, Z., Cogswell, M., et al. (2012) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2157136.2157202 · Cited by 39
Self-citationEstimating programming knowledge with Bayesian knowledge tracing
Kasurinen, J., & Nikula, U. (2009) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1562877.1562972 · Cited by 36
SPOC-supported introduction to programming
Piccioni, M., Estler, C., & Meyer, B. (2014) · doi:10.1145/2591708.2591759 · Cited by 36
Kara, finite state machines, and the case for programming as part of general education
Hartmann, W., Nievergelt, J., & Reichert, R. (2002) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/hcc.2001.995251 · Cited by 36
An Investigation into Keystroke Latency Metrics as an Indicator of Programmer Performance
Thomas, R. C., Karahasanović, A., & Kennedy, G. E. (2005) · UWA Profiles and Research Repository (University of Western Australia) · Cited by 35
Exploring Problem Solving Paths in a Java Programming Course
Hosseini, R., Vihavainen, A., & Brusilovsky, P. L. (2014) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 34
Self-citationInvestigating Automated Student Modeling in a Java MOOC
Yudelson, M. V., Hosseini, R., Vihavainen, A., & Brusilovsky, P. L. (2014) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 33
Self-citationExploring influences on student adherence to test-driven development
Buffardi, K., & Edwards, S. H. (2012) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2325296.2325324 · Cited by 32
Self-citationA detector for non-literal Java errors
Dy, T., & Rodrigo, M. M. T. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930485 · Cited by 32
Identifying Styles and Paths toward Success in MOOCs.
Sharma, K., Jermann, P., & Dillenbourg, P. (2015) · Educational Data Mining · Cited by 29
Implementation of a smart lab for teachers of novice programmers
Alammary, A. S., Carbone, A., & Sheard, J. (2012) · Swinburne figshare (Swinburne University of Technology) · Cited by 28
Self-citationRecording and analyzing in-browser programming sessions
Helminen, J., Ihantola, P., & Karavirta, V. (2013) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2526968.2526970 · Cited by 28
Self-citationAn Incremental Hint System For Automated Programming Assignments
Antonucci, P., Estler, C., Nikolić, Đ., Piccioni, M., & Meyer, B. (2015) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2729094.2742607 · Cited by 28
Analysis of source code snapshot granularity levels
Vihavainen, A., Luukkainen, M., & Ihantola, P. (2014) · Conference on Information Technology Education · doi:10.1145/2656450.2656473 · Cited by 27
Self-citationA formative study of influences on student testing behaviors
Buffardi, K., & Edwards, S. H. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538982 · Cited by 23
Self-citationWerner, L., McDowell, C., & Denner, J. (2013) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554631 · Cited by 22
Effective and ineffective software testing behaviors by novice programmers
Buffardi, K., & Edwards, S. H. (2013) · doi:10.1145/2493394.2493406 · Cited by 22
Self-citationEvaluation of Source Code with Item Response Theory
Berges, M., & Hubwieser, P. (2015) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2729094.2742619 · Cited by 21
Beacon- and Schema-Based Method for Recognizing Algorithms from Students' Source Code
Taherkhani, A., & Malmi, L. (2013) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554635 · Cited by 21
Another look at the behaviors of novice programmers
Fenwick, J. B., Norris, C., Barry, F. E., Rountree, J., Spicer, C. J., & Cheek, S. D. (2009) · ACM SIGCSE Bulletin · doi:10.1145/1539024.1508973 · Cited by 20
A system for program visualization and problem-solving path assessment of novice programmers
Satratzemi, M., Dagdilelis, V., & Evagelidis, G. (2001) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/377435.377667 · Cited by 19
Pärtel, M., Luukkainen, M., Vihavainen, A., & Vikberg, T. (2013) · International Journal of Technology Enhanced Learning · doi:10.1504/ijtel.2013.059495 · Cited by 18
Self-citationAssistance in computer programming learning using educational data mining and learning analytics
Fernandez-Medina, C., Pérez-Pérez, J.-R., Álvarez-García, V. M., & Paule-Ruiz, M. D. P. (2013) · Annual Conference on Innovation and Technology in Computer Science Education · doi:10.1145/2462476.2462496 · Cited by 16
Matsuzawa, Y., Okada, K., & Sakai, S. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2462493 · Cited by 16
Middle school students using Alice: what can we learn from logging data?
Werner, L., McDowell, C., & Denner, J. (2013) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2445196.2445347 · Cited by 12
Data Mining for Generating Hints in a Python Tutor.
Dominguez, A. K., Yacef, K., & Curran, J. (2010) · Cited by 11
Program annotations: feedback for students learning to program
Bancroft, P., & Roe, P. (2006) · QUT ePrints (Queensland University of Technology) · Cited by 11
CS EDUCATION RESEARCHThe naughties in CSEd research: a retrospective
Lister, R. (2010) · ACM Inroads · doi:10.1145/1721933.1721942 · Cited by 10
Categorizing variations of student-implemented sorting algorithms
Taherkhani, A., Korhonen, A., & Malmi, L. (2012) · Computer Science Education · doi:10.1080/08993408.2012.692917 · Cited by 9
Self-citationString formatting considered harmful for novice programmers
Hughes, M. C., Jadud, M. C., & Rodrigo, M. M. T. (2010) · Computer Science Education · doi:10.1080/08993408.2010.507335 · Cited by 5
Cited by
The current landscape of learning analytics in higher education
Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.07.027 · Cited by 766
Introductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., et al. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295779 · Cited by 485
Self-citationPredicting academic performance: a systematic literature review
Hellas, A., Ihantola, P., Petersen, A., Ajanovski, V. V., Gutica, M., Hynninen, T., Knutas, A., Leinonen, J., Messom, C., & Liao, S. N. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295783 · Cited by 355
Self-citationExploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199
Metadata
- Authors
- Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., Rubio, M. Á., Sheard, J., Skupas, B., Spacco, J., Szabo, C., & Toll, D.
- Venue
- Proceedings of the 2015 ITiCSE on Working Group Reports
- Publisher
- ACM
- Published
- 2015-07-04
- Pages
- 41-63
- OpenAlex
- W2264893711
- Cited by
- 332
- References
- 89
- FWCI
- 14.36
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.