CoChat
Sign Up
Cited by 16Psychometric Methodologies and TestingEducation, Achievement, and GiftednessEducational Technology and Assessment

How do test-takers rate their effort? A comparative analysis of self-report and log file data

Open access
Csányi, Róbert, Molnár, Gyöngyvér·2023·Learning and Individual Differences
DOI 10.1016/j.lindif.2023.102340
Open this paper in CoChat — free account

Abstract

The present study investigates students' test-taking effort by integrating and comparing traditional self-report questionnaire data and students' test-taking behavior, based on log data analyses. Previous studies have shown that different methods often lead to different results. A computer-based measure of complex problem-solving in uncertain situations was used to minimize the influence of factual knowledge on test performance. K-means cluster analysis was used to build groups of students differing in test-taking effort, resulting in 3 distinct groups. The correlation between students' test-taking effort and test performance proved to be weaker based on the self-reported questionnaire data than on their actual test-taking behavior. Both the self-report questionnaire and the log data showed a decrease in test-taking effort during the test. The number of clicks played the largest role in predicting performance. Results suggest that (1) self-report questionnaire data are not consistent with students' actual test-taking behavior and (2) it's not necessary to make the maximum effort to obtain valid test results, but a certain level of effort is needed.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.lindif.2023.102340

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Methods based on log data are more suitable than self-report-based assessment tools to objectively assess student behavior.

    “However, according to Csányi and Molnár (2023), since students' self-reported data are not always consistent with students' actual behavior, methods based on log data are more suitable than self-report-based assessment tools to objectively assess student behavior.”

    From You have a notification: the role of push notifications in shaping students’ engagement, self-regulation and academic procrastination (2025) · cites this paper as Csányi and Molnár (2023)

    The Educational relevance statement (p. 1) says self-reports are not always consistent with actual test-taking behavior and that log-data methods are more appropriate for investigating test-taking effort. The claim broadens that conclusion to objectively assessing student behavior generally; the paper's comparison concerns test-taking effort. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Self-reported data are not always consistent with students' actual behavior.

    “However, according to Csányi and Molnár (2023), since students' self-reported data are not always consistent with students' actual behavior, methods based on log data are more suitable than self-report-based assessment tools to objectively assess student behavior.”

    From You have a notification: the role of push notifications in shaping students’ engagement, self-regulation and academic procrastination (2025) · cites this paper as Csányi and Molnár (2023)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 57 · Cited by 16

Research works printed but not linked: 41

  1. AlZoubi, O., Fossati, D., Di Eugenio, B., Green, N., & CHEN, L. (2013). Predicting Students ' performance and problem solving behavior from iList log data. In Proceedings of the 21st international conference on computers in education, ICCE 2013 (pp. 1 -6). http://refhub.elsevier.com/S1041-6080(2300084-5/rf0005)
  2. Asseburg, R., & Frey, A. (2013). Too hard, too easy, or just right? The relationship between effort or boredom and ability-difficulty fit. Psychological Test and Assessment Modeling, 55 , 92 -104. http://refhub.elsevier.com/S1041-6080(2300084-5/rf0010)
  3. Attali, Y. (2016). Effort in low-stakes assessments: What does it take to perform as well as in a high-stakes setting? Educational and Psychological Measurement, 76 (6), 1045 -1058. https://doi.org/10.1177/0013164416634789 · DOI
  4. Barry, C. L., & Finney, S. J. (2016). Modeling change in effort across a low-stakes testing session: A latent growth curve modeling approach. Applied Measurement in Education, 29 (1), 46 -64. https://doi.org/10.1080/08957347.2015.1102914 · DOI
  5. Buchholz, J., Cignetti, M., & Piacentini, M. (2022). Developing measures of engagement in PISA. 279. Doi:Doi:https://doi.org/10.1787/2d9a73ca-en. · DOI
  6. Butler, J., & Adams, R. J. (2007). The impact of differential investment of student effort on the outcomes of international studies. Journal of Applied Measurement, 8 (3), 279 -304. http://refhub.elsevier.com/S1041-6080(2300084-5/rf0030)
  7. Csap ´ o, B., & Moln ´ ar, G. (2019). Online diagnostic assessment in support of personalized teaching and learning: The eDia system. Frontiers in Psychology, 10 . https://doi.org/ 10.3389/fpsyg.2019.01522 · DOI
  8. DeMars, C. E. (2000). Test stakes and item format interactions. Applied Measurement in Education, 13 (1), 55 -77. https://doi.org/10.1207/s15324818ame1301\_3 · DOI
  9. Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53 (1), 109 -132. https://doi.org/10.1146/annurev. psych.53.100901.135153 · DOI
  10. Eichmann, B., Greiff, S., Naumann, J., Brandhuber, L., & Goldhammer, F. (2020). Exploring behavioural patterns during complex problem-solving. Journal of Computer Assisted Learning, 36 (6), 933 -956. https://doi.org/10.1111/jcal.12451 · DOI
  11. Eitel, A., Endres, T., & Renkl, A. (2020). Self-management as a bridge between cognitive load and self-regulated learning: The illustrative case of seductive details. Educational Psychology Review, 32 (4), 1073 -1087. https://doi.org/10.1007/s10648020-09559-5 · DOI
  12. Finn, B. (2015). Measuring motivation in low-stakes assessments. ETS Research Report Series, 2015 (2), 1 -17. https://doi.org/10.1002/ets2.12067 · DOI
  13. Funke, J. (2014). Analysis of minimal complex systems and complex problem solving require different forms of causal cognition. Frontiers in Psychology, 5 (JUL), 1 -3. https://doi.org/10.3389/fpsyg.2014.00739 · DOI
  14. Gignac, G. E., Bartulovich, A., & Salleo, E. (2019). Maximum effort may not be required for valid intelligence test score interpretations. Intelligence, 75 , 73 -84. https://doi. org/10.1016/j.intell.2019.04.007 · DOI
  15. Goldhammer, F., Martens, T., Christoph, G., & Lüdtke, O. (2016). Test-taking engagement in PIAAC. In , 133 . OECD Education Working Papers (pp. 0 -67). https:// doi.org/10.1787/5jlzfl6fhxs2-en · DOI
  16. Goldhammer, F., Naumann, J., R ¨ olke, H., Stelter, A., & T ´ oth, K. (2017). Relating product data to process data from computer-based competency assessment. In Competence assessment in education: Research, models and instruments (pp. 407 -425). https://doi. org/10.1007/978-3-319-50030-0 · DOI
  17. Goldhammer, F., Naumann, J., Stelter, A., T ´ oth, K., R ¨ olke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106 (3), 608 -626. https://doi.org/10.1037/a0034716 · DOI
  18. Greiff, S., Moln ´ ar, G., Martin, R., Zimmermann, J., & Csap ´ o, B. (2018). Students ' exploration strategies in computer-simulated complex problem environments: A latent class approach. Computers and Education, 126 (February), 248 -263. https:// doi.org/10.1016/j.compedu.2018.07.013 · DOI
  19. Greiff, S., Niepel, C., Scherer, R., & Martin, R. (2016). Understanding students ' performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log fi les. Computers in Human Behavior, 61 , 36 -46. https://doi.org/10.1016/j.chb.2016.02.095 · DOI
  20. Greiff, S., Wüstenberg, S., Moln ´ ar, G., Fischer, A., Funke, J., & Csap ´ o, B. (2013). Complex problem solving in educational contexts -Something beyond g: Concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology, 105 (2), 364 -379. https://doi.org/10.1037/a0031856 · DOI
  21. Hauser, C., & Kingsbury, G. G. (2009). Individual score validity in a modest-stakes adaptive educational testing setting (The Annual Meeting of the National Council on Measurement in Education). http://refhub.elsevier.com/S1041-6080(2300084-5/rf0110)
  22. Hofverberg, A., Ekl ¨ of, H., & Lindfors, M. (2022). Who makes an effort? A person-centered examination of motivation and beliefs as predictors of Students ' effort and performance on the PISA 2015 science assessment. Frontiers in Education, 6 . https:// doi.org/10.3389/feduc.2021.791599 · DOI
  23. Inzlicht, M., Schmeichel, B. J., & Macrae, C. N. (2014). Why self-control seems (but may not be) limited. Trends in Cognitive Sciences, 18 (3), 127 -133. https://doi.org/ 10.1016/j.tics.2013.12.009 · DOI
  24. Khorramdel, L., von Davier, M., Gonzalez, E., & Yamamoto, K. (2020). Plausible values: Principles of item response theory and multiple imputations. In D. B. Maehler, & B. Rammstedt (Eds.), Large-Scale Cognitive Assessment: Analyzing PIAAC Data (pp. 27 -47). Springer International Publishing. https://doi.org/10.1007/978-3-03047515-4\_3. · DOI
  25. Kriegbaum, K., Jansen, M., & Spinath, B. (2014). Motivation: A predictor of PISA ' s mathematical competence beyond intelligence and prior test achievement. Learning and Individual Differences, 43 , 140 -148. https://doi.org/10.1016/j. lindif.2015.08.026 · DOI
  26. Lindner, C., Lindner, M. A., & Retelsdorf, J. (2019). Die 5-Item-Skala zur Messung der momentan verfügbaren Selbstkontrollkapazit ¨ at (SMS-5) im Lern- und Leistungskontext. Diagnostica, 65 (4), 228 -242. https://doi.org/10.1026/00121924/a000230 · DOI
  27. Lindner, C., Nagy, G., Ramos Arhuis, W. A., & Retelsdorf, J. (2017). A new perspective on the interplay between self-control and cognitive performance: Modeling progressive depletion patterns. PLoS One, 12 (6), Article e0180149. https://doi.org/10.1371/ journal.pone.0180149
  28. Lindner, C., Nagy, G., & Retelsdorf, J. (2018). The need for self-control in achievement tests: Changes in students ' state self-control capacity and effort investment. Social Psychology of Education, 21 (5), 1113 -1131. https://doi.org/10.1007/s11218-0189455-9 · DOI
  29. Lindner, C., & Retelsdorf, J. (2019). Perceived -And not manipulated -Self-control depletion predicts students ' achievement outcomes in foreign language assessments. Educational Psychology, 40 (4), 490 -508. https://doi.org/10.1080/ 01443410.2019.1661975
  30. Lindner, M. A. (2020). Representational and decorative pictures in science and mathematics tests: Do they make a difference? Learning and Instruction, 68 (September 2019), Article 101345. https://doi.org/10.1016/j. learninstruc.2020.101345 · DOI
  31. Lindner, M. A., Lüdtke, O., Grund, S., & K ¨ oller, O. (2017). The merits of representational pictures in educational assessment: Evidence for cognitive and motivational effects in a time-on-task analysis. Contemporary Educational Psychology, 51 , 482 -492. https://doi.org/10.1016/j.cedpsych.2017.09.009 · DOI
  32. Lundgren, E., & Ekl ¨ of, H. (2020). Within-item response processes as indicators of testtaking effort and motivation. Educational Research and Evaluation, 26 (5 -6), 275 -301. https://doi.org/10.1080/13803611.2021.1963940 · DOI
  33. Moln ´ ar, G., & Csap ´ o, B. (2018). The efficacy and development of students ' problemsolving strategies during compulsory schooling: Logfile analyses. Frontiers in Psychology, 9 (MAR), 1 -17. https://doi.org/10.3389/fpsyg.2018.00302 · DOI
  34. Nuutila, K., Tapola, A., Tuominen, H., Moln ´ ar, G., & Niemivirta, M. (2021). Mutual relationships between the levels of and changes in interest, self-efficacy, and perceived difficulty during task engagement. Learning and Individual Differences, 92 (December 2020). https://doi.org/10.1016/j.lindif.2021.102090 · DOI · Paper page
  35. Penk, C., & Richter, D. (2017). Change in test-taking motivation and its relationship to test performance in low-stakes assessments. Educational Assessment, Evaluation and Accountability, 29 (1), 55 -79. https://doi.org/10.1007/s11092-016-9248-7 · DOI
  36. Qiao, X., & Jiao, H. (2018). Data mining techniques in analyzing process data: A didactic. Frontiers in Psychology, 9 , 2231. https://doi.org/10.3389/fpsyg.2018.02231 · DOI
  37. Rios, J. A. (2021). Improving test-taking effort in low-stakes group-based educational testing: A Meta-analysis of interventions (pp. 1 -22). March: Applied Measurement in Education. https://doi.org/10.1080/08957347.2021.1890741 · DOI
  38. Rios, J. A., Liu, O. L., & Bridgeman, B. (2014). Identifying low-effort examinees on student learning outcomes assessment: A comparison of two approaches. New Directions for Institutional Research, 2014 (161), 69 -82. https://doi.org/10.1002/ ir.20068
  39. Sahin, F., & Colvin, K. F. (2020). Enhancing response time thresholds with response behaviors for detecting disengaged examinees. Large-Scale Assessments in Education, 8 (1), 5. https://doi.org/10.1186/s40536-020-00082-1 · DOI
  40. Schüttpelz-Brauns, K., Kadmon, M., Kiessling, C., Karay, Y., Gestmann, M., & K ¨ ammer, J. E. (2018). Identifying low test-taking effort during low-stakes tests with the new Test-taking Effort Short Scale (TESS) -Development and psychometrics. BMC Medical Education, 18 (1), 101. https://doi.org/10.1186/s12909-018-1196-0 · DOI
  41. Shi, C., Wei, B., Wei, S., Wang, W., Liu, H., & Liu, J. (2021). A quantitative discriminant method of elbow point for the optimal number of clusters in clustering algorithm. EURASIP Journal on Wireless Communications and Networking, 2021 (31). https://doi. org/10.1186/s13638-021-01910-w · DOI

Authors
Csányi, R., & Molnár, G.
Affiliations
University of Szeged (HU)
Corresponding author
Csányi, Róbert
ORCID
Venue
Learning and Individual Differences
Publisher
Elsevier BV
Published
2023-08
Volume
106
Pages
102340
ISSN
1041-6080 (ISSN-L)
DOI
10.1016/j.lindif.2023.102340
OpenAlex
W4385541418
Cited by
16
References
57
Citations by year
2026: 4 · 2025: 9 · 2024: 3
FWCI
2.95
Type
journal article
Language
en
Open access
Licence
Locations
  • Learning and Individual Differences (publishedVersion, cc-by-nc-nd)
  • SZTE Publicatio Repozitórium (University of Szeged) (acceptedVersion, other-oa)
  • Repository of the Academy's Library (Library of the Hungarian Academy of Sciences) (acceptedVersion)
Funders
  • OTKA (K135727)
  • Innovációs és Technológiai Minisztérium
  • Nemzeti Kutatási, Fejlesztési és Innovaciós Alap
  • Hungarian Scientific Research Fund
  • Magyar Tudományos Akadémia (KOZOKT2021-16)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Education, Achievement, and Giftedness
  • Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
Keywords
test-taking effort, self-report questionnaire, log file data, computer-based testing, complex problem solving, test performance, effort-performance relationship

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review