Delaying access to a problem-skipping option increases effortful practice: Application of an A/B test in large-scale online learning
Open accessPublished version of a preprint
Abstract
We report on an online double-blind randomized controlled field experiment (A/B test) in Math Garden, a computer adaptive practice system with over 150,000 active primary school children.The experiment was designed to eliminate an unforeseen opportunity to practice with minimal effort.Some children tend to skip problems that require deliberate effort, and only attempt problems that they can spontaneously answer.The intervention delayed the option to skip a problem, thereby promoting effortful practice.The results reveal an increase in the exerted effort, without being at the expense of engagement.Whether the additional effort positively affected the children's learning gains could not be concluded.Finally, in addition to these substantial results, the experiment demonstrates some of the advantages of A/B tests, such as the unique opportunity to apply truly blind randomized field experiments in educational science. 2004).In this paper, we show that large-scale experiments in online learning environments, also referred to as A/B tests, can be used to solve some of these issues.We report on a successful application of an A/B test in a large-scale online computer-adaptive practice system for Dutch primary schools (Math Garden, with over 150,000 active users).In the A/B test we delayed the option to skip a problem.This option was used by some children to skip difficult problems and practice with minimal effort, and the delay was thus aimed at promoting more effortful practice.Before describing the experimental details, we first aim to build a basic understanding of A/B tests in relation to traditional educational experiments, introduce the online practice system that is central to the experiment, and then discuss our motivation for this particular intervention.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Interventions and randomized controlled field studies have manipulated self-concept or other motivation constructs, allowing for causal inferences.
“There have also been interventions and randomized controlled field studies in which either self-concept or other motivation constructs were manipulated (e.g., Savi et al., 2018;Vansteenkiste et al., 2004), thereby allowing for causal inferences.”
From Motivation-Achievement Cycles in Learning: a Literature Review and Research Agenda (2022) · cites this paper as Savi et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447 · see citing works
The role of motivation in promoting and sustaining self-regulated learning
Pintrich, P. R. (1999) · International Journal of Educational Research · doi:10.1016/s0883-0355(99)00015-4 · Cited by 1,987 · see citing works
Understanding and misunderstanding randomized controlled trials
Deaton, A., & Cartwright, N. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.005 · Cited by 1,841 · see citing works
Self-Regulated Learning: Beliefs, Techniques, and Illusions
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013) · Annual Review of Psychology · doi:10.1146/annurev-psych-113011-143823 · Cited by 1,468 · see citing works
The State of Applied Econometrics: Causality and Policy Evaluation
Athey, S. C., & Imbens, G. W. (2017) · The Journal of Economic Perspectives · doi:10.1257/jep.31.2.3 · Cited by 1,406 · see citing works
Evidence-Based Education Policies: Transforming Educational Practice and Research
Slavin, R. E. (2002) · Educational Researcher · doi:10.3102/0013189x031007015 · Cited by 1,107 · see citing works
Heffernan, N. T., & Heffernan, C. L. (2014) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0024-x · Cited by 616 · see citing works
Governing Education in a Complex World
OECD (2016) · Educational research and innovation · doi:10.1787/9789264255364-en · Cited by 303 · see citing works
Klinkenberg, S., Straatemeier, M., & van der Maas, H. L. J. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.02.003 · Cited by 261 · see citing works
Speed-Accuracy Response Models: Scoring Rules based on Response Time and Accuracy
Maris, G., & van der Maas, H. L. J. (2012) · Psychometrika · doi:10.1007/s11336-012-9288-y · Cited by 129 · see citing works
Self-citationUnderstanding and Misunderstanding Randomized Controlled Trials
Deaton, A., & Cartwright, N. D. (2016) · National Bureau of Economic Research · doi:10.3386/w22595 · Cited by 127 · see citing works
Toward Measuring and Maintaining the Zone of Proximal Development in Adaptive Instructional Systems
Murray, T., & Arroyo, I. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47987-2_75 · Cited by 119 · see citing works
The Triumph of Hope Over Experience in the Search for “What Works”: A Response to Slavin
Olson, D. R. (2004) · Educational Researcher · doi:10.3102/0013189x033001024 · Cited by 115 · see citing works
Non-formal mechanisms in mathematical cognitive development: The case of arithmetic
Braithwaite, D. W., Goldstone, R. L., van der Maas, H. L. J., & Landy, D. H. (2016) · Cognition · doi:10.1016/j.cognition.2016.01.004 · Cited by 44 · see citing works
Cerri, S. A., Paraguaçu, F. B., & Gouardères, G. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47987-2 · Cited by 42 · see citing works
van der Ven, S. V. D., Klaiber, J. D., & van der Maas, H. L. J. (2017) · Educational Psychology · doi:10.1080/01443410.2016.1150421 · Cited by 30 · see citing works
Set as an Instance of a Real‐World Visual‐Cognitive Task
Nyamsuren, E., & Taatgen, N. A. (2013) · Cognitive Science · doi:10.1111/cogs.12001 · Cited by 25 · see citing works
Self-adapting the success rate when practicing math
Jansen, B. R. J., Hofman, A. D., Savi, A. O., Visser, I., & van der Maas, H. L. J. (2016) · Learning and Individual Differences · doi:10.1016/j.lindif.2016.08.027 · Cited by 24 · see citing works
Self-citationWilliams, J. J., Li, N., Kim, J., Whitehill, J., Maldonado, S., Pechenizkiy, M., Chu, L. F., & Heffernan, N. T. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2523265 · Cited by 21 · see citing works
Tracing the Development of Typewriting Skills in an Adaptive E-Learning Environment
van den Bergh, M., Schmittmann, V. D., Hofman, A. D., & van der Maas, H. L. J. (2015) · Perceptual and Motor Skills · doi:10.2466/23.25.pms.121c26x6 · Cited by 8 · see citing works
Experimentalism in Dutch education policy
Borghans, L. (2016) · Educational research and innovation · doi:10.1787/9789264255364-11-en · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 23 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Vygotskii, L. S. (1978). Mind in society. Harvard University Press.
Cited by
Sorted by citations
Motivation-Achievement Cycles in Learning: a Literature Review and Research Agenda
Vu, T., Magis-Weinberg, L., Jansen, B. R. J., van Atteveldt, N., Janssen, T. W. P., Lee, N. C., van der Maas, H. L. J., Raijmakers, M. E. J., Sachisthal, M. S. M., & Meeter, M. (2022) · Educational Psychology Review · doi:10.1007/s10648-021-09616-7 · Cited by 224 · see citing works
Schoors, R. V., Elen, J., Raes, A., & Depaepe, F. (2021) · British Journal of Educational Technology · doi:10.1111/bjet.13148 · Cited by 101 · see citing works
Learning As It Happens: A Decade of Analyzing and Shaping a Large-Scale Online Learning System
Brinkhuis, M. J. S., Savi, A. O., Hofman, A. D., Coomans, F., van der Maas, H. L. J., & Maris, G. (2018) · Journal of Learning Analytics · doi:10.18608/jla.2018.52.3 · Cited by 42 · see citing works
Self-citationBeing accurate about accuracy in verbal deception detection
Kleinberg, B., Arntz, A. R., & Verschuère, B. (2019) · PLoS ONE · doi:10.1371/journal.pone.0220228 · Cited by 29 · see citing works
Hrastinski, S., Stenbom, S., Benjaminsson, S., & Jansson, M. (2019) · Interactive Learning Environments · doi:10.1080/10494820.2019.1583674 · Cited by 29 · see citing works
Monferrer, L., Valentín, G. L., & Villanueva, M. S. (2022) · Education Sciences · doi:10.3390/educsci12010047 · Cited by 13 · see citing works
Value-added digital game-based learning in K-12 science education: a systematic literature review
Hussein, M. H., Sulaiman, N. L., Noroozi, O., Elaish, M. M., & Alhasan, A. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2450652 · Cited by 11 · see citing works
Omer, S. M. S., Evers, K., Wang, C.-Y., & Chen, S. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05475-7 · Cited by 28 · see citing works
Zhang, S., Bergner, Y., DiTrapani, J., & Jeon, M. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106847 · Cited by 9 · see citing works
Governing games Adaptive game selection in the Math Garden
Brinkhuis, M. J. S., Cordes, W., & Hofman, A. D. (2020) · ITM Web of Conferences · doi:10.1051/itmconf/20203303003 · Cited by 9 · see citing works
Predicting and Reducing Quitting in Online LearningPreprint
ten Broeke, N., Hofman, A. D., Kruis, J., de Mooij, S. M. M., & van der Maas, H. L. J. (2022) · doi:10.31219/osf.io/htzvm · Cited by 7 · see citing works
Can A/B Testing at Scale Accelerate Learning Outcomes in Low- and Middle-Income Environments?
Friedberg, A. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-36336-8_119 · Cited by 5 · see citing works
Psychometrics of an Elo-based large-scale online learning system
Vermeiren, H., Kruis, J., Bolsinova, M. A., van der Maas, H. L. J., & Hofman, A. D. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100376 · Cited by 5 · see citing works
Dynamic Knowledge Tracing Models for Large-Scale Adaptive Learning Environments
Sapountzi, A., Bhulai, S., Cornelisz, I., & van Klaveren, C. (2019) · Data Archiving and Networked Services (DANS) · Cited by 5 · see citing works
Daltry, R., Hinks, J., Sun, C., Major, L., Otieno, M. A., & Otieno, K. (2025) · Education Sciences · doi:10.3390/educsci15060698 · Cited by 4 · see citing works
Being accurate about verbal credibility assessmentPreprint
Kleinberg, B., Arntz, A. R., & Verschuère, B. (2019) · doi:10.31234/osf.io/h6pxt · Cited by 4 · see citing works
Adaptive Learning Systems and Interference in Causal InferencePreprint
Savi, A. O., ten Broeke, N., & Hofman, A. D. (2021) · doi:10.31234/osf.io/4ax25 · Cited by 4 · see citing works
Self-citationThree Strikes and Who is Out? Individual Differences in Error-Induced QuittingPreprint
Johansson, A., Savi, A. O., van der Maas, H. L. J., & Hofman, A. D. (2024) · doi:10.31234/osf.io/6tduv · Cited by 3 · see citing works
Self-citationSavi, A. O., van Klaveren, C., & Cornelisz, I. (2023) · Computers & Education · doi:10.1016/j.compedu.2023.104908 · Cited by 2 · see citing works
Self-citationAssessing Reliability in AI-Powered Learning Systems with A/A Tests
Ye, L., Savi, A. O., & Hofman, A. D. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729553 · Cited by 1 · see citing works
Self-citationEnacting an emergency EFL course in an Indonesian disadvantaged condition
Khotimah, K., Apgrianto, K., Mustofa, M., Ubaidillah, M. F., & Amalia, S. (2021) · JEES (Journal of English Educators Society) · doi:10.21070/jees.v6i2.1384 · Cited by 1 · see citing works
Towards an Idiographic EducationPreprint
Savi, A. O. (2019) · doi:10.31237/osf.io/7gqhz
Self-citationBrezack, N., Lee, M. S. C., Collins, K., Chan, W., & Feng, M. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3733951
Sun, C., Major, L., Daltry, R., Obradovic, L., & Friedberg, A. (2026) · Educational Technology Research and Development · doi:10.1007/s11423-026-10703-1 · Cited by 1 · see citing works
- Authors
- Savi, A. O., Ruijs, N. M., Maris, G. K. J., & van der Maas, H. L. J.
- Affiliations
- University of Amsterdam (NL)
- Cito (Netherlands) (NL)
- Corresponding author
- Savi, Alexander O.
- Venue
- Computers & Education
- Publisher
- Elsevier BV
- Published
- 2018-04
- Volume
- 119
- Pages
- 84-94
- ISSN
- 0360-1315 (ISSN-L)
- OpenAlex
- W2781352358
- Cited by
- 25
- References
- 23
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 2 · 2023: 2 · 2022: 2 · 2021: 5 · 2020: 1 · 2019: 5 · 2018: 1
- FWCI
- 1.14
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Computers & Education (publishedVersion)
- VU Research Portal (publishedVersion, other-oa)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- SocArXiv (OSF Preprints) (submittedVersion)
- SocArXiv (OSF Preprints) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Netherlands Organisation for Scientific Research (314-99-107)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Keywords
- A/B testing, online learning, primary school mathematics, learning gains, student engagement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
