Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 3

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Embracing Trustworthiness and Authenticity in the Validation of Learning Analytics Systems

van Haastrecht, Max, Brinkhuis, Matthieu, Peichl, Jessica, Remmele, Bernd, Spruit, Marco · 2023 · LAK23: 13th International Learning Analytics and Knowledge Conference

Abstract

Learning analytics sits in the middle space between learning theory and data analytics. The inherent diversity of learning analytics manifests itself in an epistemology that strikes a balance between positivism and interpretivism, and knowledge that is sourced from theory and practice. In this paper, we argue that validation approaches for learning analytics systems should be cognisant of these diverse foundations. Through a systematic review of learning analytics validation research, we find that there is currently an over-reliance on positivistic validity criteria. Researchers tend to ignore interpretivistic criteria such as trustworthiness and authenticity. In the 38 papers we analysed, researchers covered positivistic validity criteria 221 times, whereas interpretivistic criteria were mentioned 37 times. We motivate that learning analytics can only move forward with holistic validation strategies that incorporate “thick descriptions” of educational experiences. We conclude by outlining a planned validation study using argument-based validation, which we believe will yield meaningful insights by considering a diverse spectrum of validity criteria.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3576050.3576060

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 53

  1. 10.1080/0969594X.2020.1843136 · DOI
  2. 10.1016/j.compedu.2011.08.030 · DOI · Paper page
  3. 10.1016/j.chb.2019.05.036 · DOI
  4. 10.1186/s12909-018-1322-z · DOI
  5. 10.1007/978-3-030-63007-2_44 · DOI
  6. 10.1186/s41239-020-00217-y · DOI
  7. 10.3390/app10010163 · DOI
  8. 10.3390/s21082863 · DOI
  9. John Dewey. 1938. Logic: The Theory of Inquiry. Henry Holt & Company, New York, NY, USA.
  10. John Dewey and Arthur Fisher Bentley. 1949. Knowing and the Known. Beacon Press, Boston, MA, USA.
  11. 10.1016/j.compedu.2020.103829 · DOI
  12. 10.1145/3448139.3448187 · DOI
  13. 10.1145/3448139.3448140 · DOI
  14. 10.1007/s11409-022-09304-z · DOI
  15. 10.18608/jla.2019.63.8 · DOI
  16. Joy Galaige, Geraldine Torrisi-Steele, Sebastian Binnewies, and Kewen Wang. 2018. What Is Important in Student-Facing Learning Analytics? A User-Centered Design Approach. In Proceedings of the 22nd Pacific Asia Conference on Information Systems(PACIS’18). AIS, Yokohoma, Japan, 1248–1261.
  17. 10.1108/IJWIS-12-2016-0074 · DOI
  18. 10.1016/j.iheduc.2015.10.002 · DOI · Paper page
  19. 10.1016/j.chb.2022.107304 · DOI · Paper page
  20. Clifford Geertz. 1973. Thick Description: Toward an Interpretive Theory of Culture. In The Interpretation Of Cultures. Basic Books, New York, NY, USA, 3–30.
  21. 10.19173/irrodl.v16i1.1976 · DOI · Paper page
  22. 10.1007/BF02766777 · DOI · Paper page
  23. 10.1111/jedm.12234 · DOI
  24. 10.1016/j.chb.2018.07.021 · DOI
  25. 10.1111/jedm.12334 · DOI
  26. 10.1007/s10758-014-9226-4 · DOI · Paper page
  27. 10.1145/2567574.2567616 · DOI
  28. 10.1007/s10758-020-09448-4 · DOI
  29. 10.1016/j.iheduc.2016.03.002 · DOI
  30. 10.1145/3170358.3170413 · DOI
  31. 10.18608/jla.2014.12.3 · DOI · Paper page
  32. Thomas S. Kuhn. 1962. The Structure of Scientific Revolutions. University of Chicago Press, Chicago, IL, USA.
  33. 10.1016/j.chb.2017.11.011 · DOI · Paper page
  34. 10.1145/3375462.3375534 · DOI
  35. 10.1145/3170358.3170391 · DOI
  36. 10.17705/1jais.00594 · DOI
  37. 10.3390/electronics9081194 · DOI
  38. 10.1145/2723576.2723625 · DOI
  39. 10.1007/s11423-019-09693-0 · DOI
  40. 10.1080/0020739X.2020.1782492 · DOI
  41. 10.1353/eac.2007.0011 · doi:10.1353/eac.2007.0011
  42. 10.1371/journal.pone.0203590 · DOI
  43. 10.1007/s11412-021-09356-4 · DOI
  44. Mohammed Saqr Olga Viberg and Henriikka Vartiainen. 2020. Capturing the Participation and Social Dimensions of Computer-Supported Collaborative Learning through Social Network Analysis: Which Method and Measures Matter?International Journal of Computer-Supported Collaborative Learning 15 2(2020) 227–248. https://doi.org/10.1007/s11412-020-09322-6 · DOI
  45. George Siemens. 2004. Connectivism: A Learning Theory for the Digital Age. International Journal of Instructional Technology and Distance Learning 2, 1(2004), 1–8.
  46. 10.3115/v1/W14-4102 · DOI · Paper page
  47. 10.1007/s11423-021-09998-z · DOI
  48. 10.1111/jcal.12401 · DOI
  49. 10.1016/j.chb.2020.106457 · DOI
  50. 10.24059/olj.v20i2.783 · DOI
  51. 10.1016/j.iheduc.2006.10.005 · DOI · Paper page
  52. Mina Yoo and Sung-Hee Jin. 2020. Development and Evaluation of Learning Analytics Dashboards to Support Online Discussion Activities. Educational Technology & Society 23, 2 (2020), 1–18.
  53. 10.1111/bjet.13156 · DOI

Cited by

See all 3

Metadata

Authors
van Haastrecht, M., Brinkhuis, M., Peichl, J., Remmele, B., & Spruit, M.
Affiliations
  • Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Netherlands
  • Department of Information and Computing Sciences, Utrecht University, Netherlands
  • University of Education Freiburg, Germany
  • Department of Public Health and Primary Care, Leiden University Medical Center (LUMC), Netherlands and Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Netherlands
Venue
LAK23: 13th International Learning Analytics and Knowledge Conference
Publisher
ACM
Published
2023-03-13
Pages
552-558
Cited by
3
References
8
Funders
  • Erasmus+ (2022-1-DE02-KA220-VET-000087221)
  • Horizon 2020 Framework Programme (883588)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.