Source evaluation of domain experts and novices during Web search
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Nowadays, almost everyone uses the World Wide Web (WWW) to search for information of any kind. In education, students frequently use the WWW for selecting information to accomplish assignments such as writing an essay or preparing a presentation. The evaluation of sources and information is an important sub‐skill in this process. But many students have not yet optimally developed this skill. On the basis of verbal reports, eye‐tracking data and navigation logs, this study investigated how novices in the domain of psychology evaluate Internet sources as compared to domain experts. In addition, two different verbal reporting techniques, namely thinking aloud and cued retrospective reporting, were compared in order to examine students' evaluation behaviour. Results revealed that domain expertise has an impact on individuals' evaluation behaviour during Web search, such that domain experts showed a more sophisticated use of evaluation criteria to judge the reliability of sources and information and selected more reliable information than domain novices. Furthermore, the different verbal reporting techniques did not lead to different conclusions on criteria use in relation to domain expertise, although in general more utterances concerning evaluation of sources and information were expressed during cued retrospective reporting. Lay Description: What is already known about this topic: When searching the Web, students assess the reliability of information and sources in a spontaneous manner. Students tend to use superficial criteria when judging the trustworthiness of web‐based information and sources. Little is known about the differences between domain experts and novices in terms of the ways they spontaneously evaluate web‐based information and sources. What this paper adds: Domain knowledge influences evaluation behaviour, with greater domain knowledge leading to more reliable selection of sources. A comparison of techniques for measuring evaluation behaviour when searching the Web for information reveals that different techniques, such as thinking aloud and cued retrospective reporting, have different pros and cons whose implications depend on the research questions being addressed. Implications for practice and/or policy: In formal education, more attention should be paid to equipping students with the requisite skills in searching for and evaluating web‐based information. Embedded process‐oriented instructional designs are needed that foster the appropriate evaluation behaviours.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537
Ericsson, K. A., & Kintsch, W. (1995) · Psychological Review · doi:10.1037/0033-295x.102.2.211 · Cited by 3,498
Information Seeking in Electronic Environments
Marchionini, G. (1995) · doi:10.1017/cbo9780511626388 · Cited by 833
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715 · cited 6×
“To analyse how exhaustively participants considered the different search results presented on the SERP before accessing the fi rst Website, the percentage of search results visually scanned before the fi rst click , that is, before they accessed the fi rst Website, was determined (cf. Kammerer & Gerjets, 2014; Pan et al.…”
InfluentialWhat are you looking for?: an eye-tracking study of information usage in web search
Cutrell, E., & Guan, Z. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240690 · Cited by 460 · cited 1×
“The minimum fi xation duration was set to 100 ms with a fi xation radius of 30 pixels (cf. Cutrell & Guan, 2007; Gerjets et al., 2011).”
College student Web use, perceptions of information credibility, and verification behavior
Metzger, M. J., Flanagin, A. J., & Zwarun, L. (2003) · Computers & Education · doi:10.1016/s0360-1315(03)00049-6 · Cited by 448 · cited 1×
“…Web to search for a variety of information nowadays, and students frequently use search engines to fi nd information that will help them accomplish assignments such as writing an essay or preparing a presentation (e.g. Metzger, Flanagin, & Zwarun, 2003; Purcell, Heaps, Buchanan, & Friedrich, 2013).”
Source Evaluation, Comprehension, and Learning in Internet Science Inquiry Tasks
Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009) · American Educational Research Journal · doi:10.3102/0002831209333183 · Cited by 419 · cited 2×
“This fi nding is in line with research of Goldman et al. (2012) and Wiley et al. (2009) who also found a relation between source evaluation behaviour and learning outcomes.”
A Comparison of Concurrent and Retrospective Verbal Protocol Analysis
Kuusela, H., & Paul, P. (2000) · The American Journal of Psychology · doi:10.2307/1423365 · Cited by 408
Relevance and Goal-Focusing in Text Processing
McCrudden, M. T., & Schraw, G. (2007) · Educational Psychology Review · doi:10.1007/s10648-006-9010-7 · Cited by 372 · cited 1×
“This is in line with the theory of relevance in text processing (McCrudden & Schraw, 2007) and with research of Braasch et al. (2009).”
Rouet, J.-F. (2006) · doi:10.4324/9780203820094 · Cited by 337 · cited 1×
“Therefore, when searching the Web, the critical evaluation of sources and information is a very important sub-skill in information problem solving (Brand-Gruwel & Stadtler, 2011; Kammerer & Gerjets, 2012; Rouet, 2006).”
Cited by
Wineburg, S., & McGrew, S. (2019) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1177/016146811912101102 · Cited by 394
Lateral reading: Reading less and learning more when evaluating digital information
Wineburg, S., & McGrew, S. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3048994 · Cited by 198
Groothuijsen, S., van den Beemt, A., Remmers, J. C., & van Meeuwen, L. W. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100290 · Cited by 92
Tsai, M.-J., Wu, A.-H., Bråten, I., & Wang, C.-Y. (2022) · Computers & Education · doi:10.1016/j.compedu.2022.104544 · Cited by 47
Lang, F., Kammerer, Y., Stürmer, K., & Gerjets, P. (2021) · European Journal of Psychology of Education · doi:10.1007/s10212-020-00471-8 · Cited by 12
“After participants had completed the task, for each controversy, their own test-taking behavior was played back to them at 50% speed (Brand-Gruwel et al. 2017; Kammerer et al. 2013; van Gog et al. 2005) as a so-called gaze replay, a screen-recording video with their eye movements superimposed.”
BackgroundSelf-citation
Metadata
- Authors
- Brand‐Gruwel, S., Kammerer, Y., van Meeuwen, L., & van Gog, T.
- Venue
- Journal of Computer Assisted Learning
- Publisher
- Wiley
- Published
- 2017-01-05
- Volume
- 33
- Issue
- 3
- Pages
- 234-251
- ISSN
- 0266-4909 (ISSN-L)
- 1365-2729
- OpenAlex
- W2590842578
- Cited by
- 115
- References
- 11
- FWCI
- 18.99
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Source evaluation of domain experts and novices during Web search. https://app.cochat.ai/papers/10.1111/jcal.12162
- The paper
- Brand‐Gruwel, S., Kammerer, Y., van Meeuwen, L., & van Gog, T. (2017). Source evaluation of domain experts and novices during Web search. Journal of Computer Assisted Learning. https://doi.org/10.1111/jcal.12162
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.