CoChat
Sign Up
Cited by 27

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Validation of newly developed tasks for the assessment of generic Critical Online Reasoning (COR) of university students and graduates

Nagel, Marie-Theres, Zlatkin-Troitschanskaia, Olga, Fischer, Jennifer·2022·Frontiers in Education
DOI 10.3389/feduc.2022.914857
Open this paper in CoChat — free account

Abstract

In recent decades, the acquisition of information has evolved substantially and fundamentally affects students’ use of information, so that the Internet has become one of the most important sources of information for learning. However, learning with freely accessible online resources also poses challenges, such as vast amounts of partially unstructured, untrustworthy, or biased information. To successfully learn by using the Internet, students therefore require specific skills for selecting, processing, and evaluating the online information, e.g., to distinguish trustworthy from distorted or biased information and for judging its relevance with regard to the topic and task at hand. Despite the central importance of these skills, their assessment in higher education is still an emerging field. In this paper, we present the newly defined theoretical-conceptual framework Critical Online Reasoning (COR). Based on this framework, a corresponding performance assessment, Critical Online Reasoning Assessment (CORA), was newly developed and underwent first steps of validation in accordance with the Standards for Educational and Psychological Testing. We first provide an overview of the previous validation results and then expand them by including further analyses of the validity aspects “internal test structure” and “relations with other variables”. To investigate the internal test structure, we conducted variance component analyses based on the generalizability theory with a sample of 125 students and investigated the relations with other variables by means of correlation analyses. The results show correlations with external criteria as expected and confirm that the CORA scores reflect the different test performances of the participants and are not significantly biased by modalities of the assessment. With these new analyses, this study substantially contributes to previous research by providing comprehensive evidence for the validity of this new performance assessment that validly assesses the complex multifaceted construct of critical online reasoning among university students and graduates. CORA results provide unique insights into the interplay between features of online information acquisition and processing, learning environments, and the cognitive and metacognitive requirements for critically reasoning from online information in university students and young professionals.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/feduc.2022.914857

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 5 references

Research works printed but not linked: 73

  1. 2014, Standards for Educational and Psychological Testing
  2. Ali, 2020, Online and remote learning in higher education institutes: a necessity in light of COVID-19 pandemic, High. Educ. Stud., 10, 16, 10.5539/hes.v10n3p16 · DOI · Paper page
  3. Aspegren, 2005, Which basic communication skills in medicine are learnt spontaneously and which need to be taught and trained?, Med. Teach., 27, 539, 10.1080/01421590500136501 · DOI
  4. Banerjee, 2020, Narratives and their impact on students’ information seeking and critical online reasoning in higher education economics and medicine, Front. Educ., 5, 570625, 10.3389/feduc.2020.570625 · DOI
  5. Bates, 2015, Fitting linear mixed-effects models using lme4, J. Stat. Softw., 67, 1, 10.18637/jss.v067.i01 · DOI · Paper page
  6. Binkley, 2012, Defining twenty-first century skills, 17, 10.1007/978-94-007-2324-5_2 · DOI · Paper page
  7. Boh Podgornik, 2016, Development, testing, and validation of an information literacy test (ILT) for higher education, J. Assoc. Inf. Sci. Technol., 67, 2420, 10.1002/asi.23586 · DOI
  8. Braasch, 2018, Handbook of Multiple Source Use, 10.4324/9781315627496 · DOI
  9. Brand-Gruwel, 2009, A descriptive model of information problem solving while using internet, Comput. Educ., 53, 1207, 10.1016/j.compedu.2009.06.004 · DOI
  10. Braun, 2021, Performance-based assessment of students’ communication skills, Int. J. Chin. Educ., 10, 221258682110062, 10.1177/22125868211006202 · DOI
  11. Braun, 2018, Erfassung praxisbezogener Anforderungen und Tätigkeiten von Hochschulabsolvent(inn)en (PAnTHoa), Zeitschrift für Hochschulentwicklung, 209, 10.3217/zfhe-13-01/11 · DOI
  12. Braun, 2020, Performance assessment of critical thinking: conceptualization, design, and implementation, Front. Educ., 5, 156, 10.3389/feduc.2020.00156 · DOI
  13. Campbell, 1959, Convergent and discriminant validation by the multitrait-multimethod matrix, Psychol. Bull., 56, 81, 10.1037/h0046016 · DOI · Paper page
  14. Chan, 2017, A review of literature on challenges in the development and implementation of generic competencies in higher education curriculum, Int. J. Educ. Dev., 57, 1, 10.1016/j.ijedudev.2017.08.010 · DOI
  15. Ciampaglia, 2018, The digital misinformation pipeline–proposal for a research agenda, 413
  16. Cronbach, 1972, The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles
  17. Desai, 2019, Comparing the use of open and closed questions for web-based measures of the continued-influence effect, Behav. Res. Methods, 51, 1426, 10.3758/s13428-018-1066-z · DOI
  18. Flanagin, 2017, Digital Media and Perceptions of Source Credibility in Political Communication. The Oxford Handbook of Political Communication
  19. Goldhammer, 2013, Assessing individual differences in basic computer skills, Eur. J. Psychol. Assess., 29, 263, 10.1027/1015-5759/a000153 · DOI
  20. Goldman, 2018, Learning from multiple sources in a digital society, 86, 10.4324/9781315617572-9 · DOI
  21. Guath, 2022, Civic online reasoning among adults: an empirical evaluation of a prescriptive theory and its correlates, Front. Educ., 7, 721731, 10.3389/feduc.2022.721731 · DOI
  22. Hahnel, 2019, The role of cognitive load in university students’ comprehension of multiple documents, Zeitschrift für pädagogische Psychologie, 33, 105, 10.1024/1010-0652/a000238 · DOI
  23. Harrison, 2019, Experts, knowledge and criticality in the age of ‘alternative facts’: reexamining the contribution of higher education, Teach. High. Educ., 24, 259, 10.1080/13562517.2019.1578577 · DOI
  24. Herrero-Diz, 2019, The credibility of online news: an evaluation of the information by university students, Cult. Educ., 31, 407, 10.1080/11356405.2019.1601937 · DOI
  25. Huang, 2019, Exploring patterns in undergraduate students’ information problem solving: a cross-case comparison study, Knowledge Manag. E-Learn., 11, 428, 10.34105/j.kmel.2019.11.023 · DOI
  26. Jiang, 2018, Using the linear mixed-effect model framework to estimate generalizability variance components in R, Methodology, 14, 133, 10.1027/1614-2241/a000149 · DOI
  27. Johnson, 2016, Students’ approaches to the evaluation of digital information: insights from their trust judgments, Br. J. Educ. Technol., 47, 1243, 10.1111/bjet.12306 · DOI
  28. Koltay, 2011, The media and the literacies: media literacy, information literacy, digital literacy, Media Cult. Soc., 33, 211, 10.1177/0163443710393382 · DOI
  29. Korn, 2004, Teaching talking: Oral communication skills in a Law course, J. Leg. Educ., 54, 588
  30. Ku, 2009, Assessing students’ critical thinking performance: urging for measurements using multi-response format, Think. Skills Creat., 4, 70, 10.1016/j.tsc.2009.02.001 · DOI
  31. Leeder, 2019, How college students evaluate and share “fake news” stories, Libr. Inf. Sci. Res., 41, 100967, 10.1016/j.lisr.2019.100967 · DOI
  32. Leighton, 2017, Using Think-Aloud Interviews and Cognitive Labs in Educational Research, 10.1093/acprof:oso/9780199372904.001.0001 · DOI
  33. Limberg, 2012, Three theoretical perspectives on information literacy, Hum. IT, 11, 93
  34. Liu, 2014, Assessing critical thinking in higher education: current state and directions for next-generation assessment, ETS Res. Rep. Ser., 2014, 1, 10.1002/ets2.12009 · DOI · Paper page
  35. Maurer, 2018, Media effects on positive and negative learning, 197
  36. Maurer, 2020, Positive and negative media effects on university students’ learning: Preliminary findings and a research program, 109
  37. Mayer, 2009, Multimedia Learning, 10.1017/CBO9780511811678 · DOI · Paper page
  38. McGrew, 2017, The challenge that’s bigger than fake news. Civic reasoning in a social media environment, Am. Educ., 41, 4
  39. Mislevy, 2017, Sociocognitive Foundations of Educational Measurement
  40. Mislevy, 2012, Design and discovery in educational assessment: Evidence-centered design, psychometrics, and educational data mining, Journal of Educational Data Mining, 4, 11
  41. Molerov, 2020, Assessing university students’ critical online reasoning ability: a conceptual and assessment framework with preliminary evidence, Front. Educ., 5, 577843, 10.3389/feduc.2020.577843 · DOI · Paper page
  42. Nagel, 2020, How do University students’ web search behavior, website characteristics, and the interaction of Both influence students’ critical online reasoning?, Front. Educ., 5, 565062, 10.3389/feduc.2020.565062 · DOI
  43. 2012, Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century
  44. Naumann, 2001, Validierung des INCOBI anhand eines Vergleichs von Anwendungsexperten und Anwendungsnovizen, Zeitschrift für Pädagogische Psychologie, 15, 219, 10.1024//1010-0652.15.34.219 · DOI
  45. Nygren, 2020, Students evaluating and corroborating digital news, Scand. J. Educ. Res., 66, 549, 10.1080/00313831.2021.1897876 · DOI
  46. Osborne, 2022, Science Education in an Age of Misinformation
  47. Oser, 2020, A three-level model for critical thinking: Critical alertness, critical reflection, and critical analysis, 89
  48. Park, 2021, A Scientometric study of digital literacy, ICT literacy, information literacy, and media literacy, J. Data Info. Sci., 6, 116, 10.2478/jdis-2021-0001 · DOI
  49. Qiu, 2017, Limited individual attention and online virality of low-quality information, Nat. Hum. Behav., 1, 1, 10.1038/s41562-017-0132 · DOI
  50. Rammstedt, 2013, Grundlegende Kompetenzen Erwachsener im internationalen Vergleich: Ergebnisse von PIAAC 2012
  51. Reddy, 2020, Digital literacy: a review of literature, Int. J. Technoethics, 11, 65, 10.4018/IJT.20200701.oa1 · DOI · Paper page
  52. Rowley, 2015, Students’ trust judgements in online health information seeking, Health Informatics J., 21, 316, 10.1177/1460458214546772 · DOI
  53. Sanders, 2015, Information behaviors and information literacy skills of LIS students: an international perspective, J. Educ. Libr. Inf. Sci., 56, 80, 10.12783/issn.2328-2967/56/S1/9 · DOI
  54. Schlebusch, 2018, Computer anxiety, computer self-efficacy and attitudes toward the internet of first year students at a south African University of Technology, Africa Educ. Rev., 15, 72, 10.1080/18146627.2017.1341291 · DOI
  55. Schmidt, 2020, Undergraduate students’ critical online reasoning: process mining analysis, Front. Psychol., 11, 576273, 10.3389/fpsyg.2020.576273 · DOI
  56. Selwyn, 2016, Students’ use of Wikipedia as an academic resource—patterns of use and perceptions of usefulness, Internet High. Educ., 28, 28, 10.1016/j.iheduc.2015.08.004 · DOI
  57. Sendurur, 2018, Students as information consumers: a focus on online decision making process, Educ. Inf. Technol., 23, 3007, 10.1007/s10639-018-9756-9 · DOI
  58. Shavelson, 1981, Generalizability theory: 1973–1980, Br. J. Math. Stat. Psychol., 34, 133, 10.1111/j.2044-8317.1981.tb00625.x · DOI
  59. Shavelson, 1989, Generalizability theory, American Psychologist, 44, 922, 10.1037/0003-066X.44.6.922 · DOI
  60. Shavelson, 2019, Assessment of university students’ critical thinking: next generation performance assessment, Int. J. Test., 19, 337, 10.1080/15305058.2018.1543309 · DOI
  61. Shavelson, 2018, International Performance Assessment of Learning in Higher Education (iPAL): Research and Development, Assessment of Learning Outcomes in Higher Education: Cross-National Comparisons and Perspectives, 193, 10.1007/978-3-319-74338-7_10 · DOI
  62. Solano-Flores, 2020, Boolean analysis of Interobserver agreement: formal and functional evidence sampling in complex coding endeavors, Educ. Meas. Issues Pract., 40, 26, 10.1111/emip.12409 · DOI
  63. 2021, Stata Statistical Software: Release 17
  64. Virtanen, 2018, Factors explaining the learning of generic skills: a study of university students’ experiences, Teaching in Higher Education, 24, 880, 10.1080/13562517.2018.1515195 · DOI
  65. Walton, 2020, Information literacy and the societal imperative of information discernment, 149
  66. Wathen, 2001, Believe it or not: factors influencing credibility on the web, J. Am. Soc. Inf. Sci. Technol., 53, 134, 10.1002/asi.10016 · DOI
  67. Weber, 2019, Information-seeking behaviour and academic success in higher education: which search strategies matter for grade differences among university students and how does this relevance differ by field of study?, High. Educ., 77, 657, 10.1007/s10734-018-0296-4 · DOI
  68. Whitelock-Wainwright, 2020, Exploring student information problem solving behaviour using fine-grained concept map and search tool data, Comput. Educ., 145, 103731, 10.1016/j.compedu.2019.103731 · DOI
  69. Wineburg, 2018, Why Google can’t save us. The challenges of our post-Gutenberg moment, 221
  70. Wineburg, 2016, Why students can’t google their way to the truth: fact-checkers and students approach websites differently, Educ. Week, 36, 22
  71. Zieky, 2014, An introduction to the use of evidence-centered design in test development, Psicol. Educ., 20, 79, 10.1016/j.pse.2014.11.003 · DOI
  72. Zlatkin-Troitschanskaia, Performance assessment and digital training framework for young professionals’ generic and domain-specific online reasoning in law, medicine, and teacher practice, J. Supranat. Pol. Educ., 13, 9, 10.15366/jospoe2021.13.001 · DOI
  73. Zlatkin-Troitschanskaia, Students’ online information use and learning progress in higher education – a critical literature review, Stud. High. Educ., 46, 1996, 10.1080/03075079.2021.1953336 · DOI

Cited by

Sorted by citations

See all 27

Metadata

Authors
Nagel, M.-T., Zlatkin-Troitschanskaia, O., & Fischer, J.
Venue
Frontiers in Education
Publisher
Frontiers Media SA
Published
2022-11-07
Volume
7
ISSN
2504-284X
DOI
10.3389/feduc.2022.914857
Cited by
27
References
5
Licence
creativecommons.org/licenses/by/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review