The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Embracing Trustworthiness and Authenticity in the Validation of Learning Analytics Systems
Abstract
Learning analytics sits in the middle space between learning theory and data analytics. The inherent diversity of learning analytics manifests itself in an epistemology that strikes a balance between positivism and interpretivism, and knowledge that is sourced from theory and practice. In this paper, we argue that validation approaches for learning analytics systems should be cognisant of these diverse foundations. Through a systematic review of learning analytics validation research, we find that there is currently an over-reliance on positivistic validity criteria. Researchers tend to ignore interpretivistic criteria such as trustworthiness and authenticity. In the 38 papers we analysed, researchers covered positivistic validity criteria 221 times, whereas interpretivistic criteria were mentioned 37 times. We motivate that learning analytics can only move forward with holistic validation strategies that incorporate “thick descriptions” of educational experiences. We conclude by outlining a planned validation study using argument-based validation, which we believe will yield meaningful insights by considering a diverse spectrum of validity criteria.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The large-scale collection of learner data raises concerns regarding ethics, privacy, fairness, and trustworthiness [7,22].”
From Federated Learning Analytics: Investigating the Privacy-Performance Trade-Off in Machine Learning for Educational Analytics (2024) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Design Science in Information Systems Research1
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004) · MIS Quarterly · doi:10.2307/25148625 · Cited by 10,707 · cited 3×
“Research is relevant when it addresses “the problems faced and the opportunities afforded by the interaction of people, organisations, and information technology” [25].”
BackgroundInfluentialBut is it rigorous? Trustworthiness and authenticity in naturalistic evaluation
Lincoln, Y. S., & Guba, E. G. (1986) · New Directions for Program Evaluation · doi:10.1002/ev.1427 · Cited by 3,668
Validating the interpretations and uses of test scores
Kane, M. T. (2013) · Journal of Educational Measurement · doi:10.1111/jedm.12000 · Cited by 2,110
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017) · Computers & Education · doi:10.1016/j.compedu.2016.10.001 · Cited by 1,003
Time will tell: The role of mobile learning analytics in self-regulated learning
Tabuenca, B., Kalz, M., Drachsler, H., & Specht, M. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.08.004 · Cited by 242
SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing
van Haastrecht, M., Sarhan, I., Yigit Ozkan, B., Brinkhuis, M., & Spruit, M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.685591 · Cited by 80
Self-citationUsing trace data to enhance Students' self-regulation: A learning analytics perspective
Ye, D., & Pennisi, S. (2022) · The Internet and Higher Education · doi:10.1016/j.iheduc.2022.100855 · Cited by 69
Counting Clicks is Not Enough: Validating a Theorized Model of Engagement in Learning Analytics
Fincham, E., Whitelock-Wainwright, A., Kovanović, V., Joksimović, S., van Staalduinen, J.-P., & Gašević, D. (2019) · Proceedings of the 9th International Conference on Learning Analytics & Knowledge · doi:10.1145/3303772.3303775 · Cited by 57
Research works printed but not linked: 53
- 10.1080/0969594X.2020.1843136 · DOI
- 10.1016/j.compedu.2011.08.030 · DOI · Paper page
- 10.1016/j.chb.2019.05.036 · DOI
- 10.1186/s12909-018-1322-z · DOI
- 10.1007/978-3-030-63007-2_44 · DOI
- 10.1186/s41239-020-00217-y · DOI
- 10.3390/app10010163 · DOI
- 10.3390/s21082863 · DOI
- John Dewey. 1938. Logic: The Theory of Inquiry. Henry Holt & Company, New York, NY, USA.
- John Dewey and Arthur Fisher Bentley. 1949. Knowing and the Known. Beacon Press, Boston, MA, USA.
- 10.1016/j.compedu.2020.103829 · DOI
- 10.1145/3448139.3448187 · DOI
- 10.1145/3448139.3448140 · DOI
- 10.1007/s11409-022-09304-z · DOI
- 10.18608/jla.2019.63.8 · DOI
- Joy Galaige, Geraldine Torrisi-Steele, Sebastian Binnewies, and Kewen Wang. 2018. What Is Important in Student-Facing Learning Analytics? A User-Centered Design Approach. In Proceedings of the 22nd Pacific Asia Conference on Information Systems(PACIS’18). AIS, Yokohoma, Japan, 1248–1261.
- 10.1108/IJWIS-12-2016-0074 · DOI
- 10.1016/j.iheduc.2015.10.002 · DOI · Paper page
- 10.1016/j.chb.2022.107304 · DOI · Paper page
- Clifford Geertz. 1973. Thick Description: Toward an Interpretive Theory of Culture. In The Interpretation Of Cultures. Basic Books, New York, NY, USA, 3–30.
- 10.19173/irrodl.v16i1.1976 · DOI · Paper page
- 10.1007/BF02766777 · DOI · Paper page
- 10.1111/jedm.12234 · DOI
- 10.1016/j.chb.2018.07.021 · DOI
- 10.1111/jedm.12334 · DOI
- 10.1007/s10758-014-9226-4 · DOI · Paper page
- 10.1145/2567574.2567616 · DOI
- 10.1007/s10758-020-09448-4 · DOI
- 10.1016/j.iheduc.2016.03.002 · DOI
- 10.1145/3170358.3170413 · DOI
- 10.18608/jla.2014.12.3 · DOI · Paper page
- Thomas S. Kuhn. 1962. The Structure of Scientific Revolutions. University of Chicago Press, Chicago, IL, USA.
- 10.1016/j.chb.2017.11.011 · DOI · Paper page
- 10.1145/3375462.3375534 · DOI
- 10.1145/3170358.3170391 · DOI
- 10.17705/1jais.00594 · DOI
- 10.3390/electronics9081194 · DOI
- 10.1145/2723576.2723625 · DOI
- 10.1007/s11423-019-09693-0 · DOI
- 10.1080/0020739X.2020.1782492 · DOI
- 10.1353/eac.2007.0011 · doi:10.1353/eac.2007.0011
- 10.1371/journal.pone.0203590 · DOI
- 10.1007/s11412-021-09356-4 · DOI
- Mohammed Saqr Olga Viberg and Henriikka Vartiainen. 2020. Capturing the Participation and Social Dimensions of Computer-Supported Collaborative Learning through Social Network Analysis: Which Method and Measures Matter?International Journal of Computer-Supported Collaborative Learning 15 2(2020) 227–248. https://doi.org/10.1007/s11412-020-09322-6 · DOI
- George Siemens. 2004. Connectivism: A Learning Theory for the Digital Age. International Journal of Instructional Technology and Distance Learning 2, 1(2004), 1–8.
- 10.3115/v1/W14-4102 · DOI · Paper page
- 10.1007/s11423-021-09998-z · DOI
- 10.1111/jcal.12401 · DOI
- 10.1016/j.chb.2020.106457 · DOI
- 10.24059/olj.v20i2.783 · DOI
- 10.1016/j.iheduc.2006.10.005 · DOI · Paper page
- Mina Yoo and Sung-Hee Jin. 2020. Development and Evaluation of Learning Analytics Dashboards to Support Online Discussion Activities. Educational Technology & Society 23, 2 (2020), 1–18.
- 10.1111/bjet.13156 · DOI
Cited by
van Haastrecht, M., Brinkhuis, M., & Spruit, M. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-64299-9_5 · Cited by 13
Self-citation
Metadata
- Authors
- van Haastrecht, M., Brinkhuis, M., Peichl, J., Remmele, B., & Spruit, M.
- Affiliations
- Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Netherlands
- Department of Information and Computing Sciences, Utrecht University, Netherlands
- University of Education Freiburg, Germany
- Department of Public Health and Primary Care, Leiden University Medical Center (LUMC), Netherlands and Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Netherlands
- Venue
- LAK23: 13th International Learning Analytics and Knowledge Conference
- Publisher
- ACM
- Published
- 2023-03-13
- Pages
- 552-558
- Cited by
- 3
- References
- 8
- Funders
- Erasmus+ (2022-1-DE02-KA220-VET-000087221)
- Horizon 2020 Framework Programme (883588)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.