The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
The Effect of Achievement Badges on Students’ Behavior: An Empirical Study in a University-Level Computer Science Course
Open accessAbstract
Achievement badges are a form of gamification that are used in an attempt to increase user engagement and motivation in various systems. A badge is typically a graphical icon that appears as a reward for the user after reaching an achievement but that has no practical value. In this study, we describe and evaluate the use of achievement badges in the TRAKLA2 online learning environment where students solve interactive, automatically assessed exercises in a Data Structures and Algorithms course throughout the semester. We conducted an experiment where the students (N=281) were randomly divided into a treatment and a control group, with and without achievement badges. Students in the treatment group were awarded achievement badges, for example, for solving exercises correctly on the first attempt, doing exercises early, or solving all the exercises in a round with full points. Grading was the same for both groups, i.e. collecting badges did not affect the final grade, even though the exercise points themselves did. Students’ activity in TRAKLA2 was logged in order to find out whether the achievement badges had an effect on their behavior. We also collected numerical and open-ended feedback in order to find out students’ attitudes towards the badges. Our results show that achievement badges can be used to affect students’ behavior. Statistically significant differences were observed in the time used per exercise, number of sessions, total time, and normalized total number of badges. Furthermore, the majority of the students reported being motivated by the badges. Based on our findings, achievement badges seem to be a promising method to motivate students and to encourage desired study practices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, (Hakulinen et al., 2015) present convincing evidence that points, badges and leaderboard incorporated in online Java exercises increases the use of an open learning environment.”
From Gamifying education: what is known, what is believed and what remains uncertain: a critical review (2017) · cites this paper as Hakulinen et al., 2015
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, two papers (Hakulinen et al., 2015) and (Landers & Landers, 2015) report positive outcomes for using single game elements, but one is for badges and the other one for leaderboards.”
From Gamifying education: what is known, what is believed and what remains uncertain: a critical review (2017) · cites this paper as Hakulinen et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Intrinsic Motivation and Self-Determination in Human Behavior
Deci, E. L., & Ryan, R. M. (1985) · doi:10.1007/978-1-4899-2271-7 · Cited by 26,772
Flow: The Psychology of Optimal Experience
Buchanan, R., & Csíkszentmihályi, M. (1991) · Design Issues · doi:10.2307/1511458 · Cited by 9,449
Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification
Hamari, J., Koivisto, J., & Sarsa, H. (2014) · 2014 47th Hawaii International Conference on System Sciences · doi:10.1109/hicss.2014.377 · Cited by 5,201
Gamifying learning experiences: Practical implications and outcomes
Domínguez‐Díaz, A., Saenz-de-Navarrete, J., de‐Marcos, L., Sanz, L. F., Pagés, C., & Martínez-Herráiz, J.-J. (2013) · Computers & Education · doi:10.1016/j.compedu.2012.12.020 · Cited by 2,189
Extrinsic Rewards and Intrinsic Motivation in Education: Reconsidered Once Again
Deci, E. L., Koestner, R., & Ryan, R. M. (2001) · Review of Educational Research · doi:10.3102/00346543071001001 · Cited by 1,596
Gamification in Education: What, How, Why Bother?
Lee, J. J., & Hammer, J. (2011) · Academic exchange quarterly · Cited by 1,199
The effects of praise on children's intrinsic motivation: A review and synthesis.
Corpus, J. H., & Lepper, M. R. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.5.774 · Cited by 751
An empirical study comparing gamification and social networking on e-learning
de-Marcos, L., Domínguez, A., Saenz-de-Navarrete, J., & Pagés, C. (2014) · Computers & Education · doi:10.1016/j.compedu.2014.01.012 · Cited by 626
(2012) · Cited by 563
Engaging with massive online courses
Anderson, A., Huttenlocher, D. P., Kleinberg, J. M., & Leskovec, J. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568042 · Cited by 559
Are badges useful in education?: it depends upon the type of badge and expertise of learner
Abramovich, S., Schunn, C. D., & Higashi, R. M. (2013) · Educational Technology Research and Development · doi:10.1007/s11423-013-9289-2 · Cited by 554
The effect of virtual achievements on student engagement
Denny, P. C. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470763 · Cited by 500
Raising engagement in e-learning through gamification
Ioana, M. C. (2011) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.8133016 · Cited by 480
Raising engagement in e-learning through gamification
Ioana, M. C. (2011) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.8132981 · Cited by 478
Procrastination, participation, and performance in online learning environments
Michinov, N., Brunot, S., Bohec, O. L., Juhel, J., & Delaval, M. (2010) · Computers & Education · doi:10.1016/j.compedu.2010.07.025 · Cited by 425
Gaming science: the “Gamification” of scientific thinking
Morris, B. J., Croker, S. J., Zimmerman, C., Gill, D., & Romig, C. J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00607 · Cited by 256
Empirical Study on the Effect of Achievement Badges in TRAKLA2 Online Learning Environment
Hakulinen, L., Auvinen, T., & Korhonen, A. (2013) · doi:10.1109/latice.2013.34 · Cited by 228
Self-citationFramework for Designing and Evaluating Game Achievements
Hamari, J., & Eranti, V. (2011) · doi:10.26503/dl.v2011i1.545 · Cited by 222
Using software testing to move students from trial-and-error to reflection-in-action
Edwards, S. H. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1028174.971312 · Cited by 213
Visual Algorithm Simulation Exercise System with Automatic Assessment: TRAKLA2
Malmi, L., Karavirta, V., Korhonen, A., Nikander, J., Seppälä, O., & Silvasti, P. (2004) · Informatics in Education · doi:10.15388/infedu.2004.19 · Cited by 205
Self-citationApplying game achievement systems to enhance user experience in a photo sharing service
Montola, M., Nummenmaa, T., Lucero, A., Boberg, M., & Korhonen, H. (2009) · International Academic Mindtrek Conference (Mindtrek) · doi:10.1145/1621841.1621859 · Cited by 157
How (not) to introduce badges to online exercises
Haaranen, L., Ihantola, P., Hakulinen, L., & Korhonen, A. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538921 · Cited by 133
Self-citationComparing effective and ineffective behaviors of student programmers
Edwards, S. H., Snyder, J., Pérez-Quiñones, M. A., Allevato, A., Kim, D., & Tretola, B. (2009) · doi:10.1145/1584322.1584325 · Cited by 117
Using badges for shaping interactions in online learning environments
McDaniel, R., Lindgren, R., & Friskics, J. (2012) · IEEE International Professional Communication Conference (ProComm) · doi:10.1109/ipcc.2012.6408619 · Cited by 112
Malmi, L., Karavirta, V., Korhonen, A., & Nikander, J. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163412 · Cited by 90
Self-citationThe Presenter's Paradox: Figure 1.
Weaver, K. D., Garcia, S. M., & Schwarz, N. (2012) · Journal of Consumer Research · doi:10.1086/664497 · Cited by 85
On the use of resubmissions in automatic assessment systems
Karavirta, V., Korhonen, A., & Malmi, L. (2006) · Computer Science Education · doi:10.1080/08993400600912426 · Cited by 83
Self-citationMekler, E. D., Brühlmann, F., Opwis, K., & Tuch, A. N. (2013) · edoc (University of Basel) · doi:10.5451/unibas-ep32047 · Cited by 63
Another look at the behaviors of novice programmers
Fenwick, J. B., Norris, C., Barry, F. E., Rountree, J., Spicer, C. J., & Cheek, S. D. (2009) · ACM SIGCSE Bulletin · doi:10.1145/1539024.1508973 · Cited by 20
Software Architectures for Implementing Achievement Badges - Practical Experiences
Haaranen, L., Hakulinen, L., Ihantola, P., & Korhonen, A. (2014) · International Conference on Learning and Teaching in Computing and Engineering (LaTICE) · doi:10.1109/latice.2014.16 · Cited by 11
Self-citation
Research works printed but not linked: 4
- K. Huotari and J. Hamari, "Defining gamification: A service marketing perspective," in Proceeding of the 16th International Academic MindTrek Conference, ser. MindTrek '12. New York, NY, USA: ACM, 2012, pp. 17-22.
- S. Deterding, D. Dixon, R. Khaled, and L. Nacke, "From game design elements to gamefulness: defining "gamification"," in Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, ser. MindTrek '11. New York, NY, USA: ACM, 2011, pp. 9-15.
- A. Decker and E. L. Lawley, "Life's a game and the game of life: How making a game out of it can change student behavior," in Proceeding of the 44th ACM Technical Symposium on Computer Science Education, ser. SIGCSE '13. New York, NY, USA: ACM, 2013, pp. 233-238.
- N. J. Falkner and K. E. Falkner, "A fast measure for identifying at- risk students in computer science," in Proceedings of the ninth annual international conference on International computing education research. ACM, 2012, pp. 55-62.
Cited by
Sorted by citations
The Rise of Motivational Information Systems: A Review of Gamification Research
Koivisto, J., & Hamari, J. (2017) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.10.013 · Cited by 2,229
Hamari, J., Shernoff, D. J., Rowe, E., Coller, B., Asbell-Clarke, J., & Edwards, T. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.07.045 · Cited by 1,713
Gamifying education: what is known, what is believed and what remains uncertain: a critical review
Dichev, C., & Dicheva, D. (2017) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-017-0042-5 · Cited by 2,112
“…to behavioral and motivational (Attali & Arieli-Attali, 2015; Barrio et al., 2015; Boticki et al., 2015; Christy & Fox, 2014; Davis & Klein, 2015; Hakulinen et al., 2015; Landers & Landers, 2015; Long & Aleven, 2014; Pedro et al., 2015a; Tu et al., 2015; Tvarozek & Brza, 2014) 2 elements 8 (16”
InfluentialBai, S., Hew, K. F., & Huang, B. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100322 · Cited by 994
Groening, C., & Binnewies, C. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.02.026 · Cited by 249
Metadata
- Authors
- Hakulinen, L., Auvinen, T., & Korhonen, A.
- Affiliations
- Aalto University (FI)
- ORCID
- Korhonen, Ari
- Venue
- International Journal of Emerging Technologies in Learning (iJET)
- Published
- 2015-02-21
- Volume
- 10
- Issue
- 1
- Pages
- 18
- ISSN
- 1863-0383 (ISSN-L)
- OpenAlex
- W1994917054
- Cited by
- 201
- References
- 30
- Citations by year
- 2026: 8 · 2025: 9 · 2024: 12 · 2023: 17 · 2022: 31 · 2021: 24 · 2020: 19 · 2019: 55 · 2018: 11 · 2017: 8 · 2016: 3 · 2015: 4
- FWCI
- 21.90
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- International Journal of Emerging Technologies in Learning (iJET) (publishedVersion, cc-by)
- Aaltodoc (Aalto University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- achievement badges, gamification, student motivation, student behavior, virtual learning environments, data structures and algorithms course, computer science education, student engagement, student attitudes, automated assessment, study habits
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
