Relevance versus big numbers: Students’ criteria for selecting scholarly references online.
Abstract
This study examined the effect of the number of citations attributed to documents on third year psychology students' selection of bibliographical references. Our main assumption was that students would take high numbers of citations as accessible relevance cues and use them heuristically to facilitate decision making, potentially bypassing deeper relevance assessment based on semantic processing. Experiment 1 presented the students with a reference selection task while manipulating the number of citations attributed to references, and found that the number of citations had a strong impact on reference selection. Moreover, the effect was independent from topic familiarity and even from students' prior knowledge of what the number of citations meant. Experiment 2 used eye-tracking data to show that this "big number" effect was contingent upon the participants fixating the numbers of citations attributed to documents. Experiment 3 manipulated the semantic relevance of references to the search topic, and demonstrated that the less relevant references were 3 times more likely to be selected when they came with a high number of citations. Overall, the study shows that the number of citations significantly influences students' selections, competing with the semantic relevance of references. Implications for the teaching of online search skills are discussed. (PsycINFO Database Record (c) 2018 APA, all rights reserved).
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
(2011) · Cited by 14,317 · see citing works
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328 · see citing works
Eye movements in reading and information processing: 20 years of research
Rayner, K. (1998) · Psychological Bulletin · doi:10.1037/0033-2909.124.3.372 · Cited by 7,979 · see citing works
(1998) · Cited by 4,875 · see citing works
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373 · see citing works
Social and Heuristic Approaches to Credibility Evaluation Online
Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010) · Journal of Communication · doi:10.1111/j.1460-2466.2010.01488.x · Cited by 1,394 · see citing works
Attention and choice: A review on eye movements in decision making
Orquin, J. L., & Loose, S. M. (2013) · Acta Psychologica · doi:10.1016/j.actpsy.2013.06.003 · Cited by 952 · see citing works
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715 · see citing works
Collective knowledge systems: Where the Social Web meets the Semantic Web
Gruber, T. (2007) · Journal of Web Semantics · doi:10.1016/j.websem.2007.11.011 · Cited by 581 · see citing works
Conceptualizing sources in online news
Sundar, S. S., & Nass, C. (2001) · Journal of Communication · doi:10.1111/j.1460-2466.2001.tb02872.x · Cited by 533 · see citing works
News cues: Information scent and cognitive heuristics
Sundar, S. S., Knobloch‐Westerwick, S., & Hastall, M. R. (2006) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20511 · Cited by 303 · see citing works
Karmarkar, U. R., & Tormala, Z. L. (2009) · Journal of Consumer Research · doi:10.1086/648381 · Cited by 294 · see citing works
Eye tracking and pupillometry are indicators of dissociable latent decision processes.
Cavanagh, J. F., Wiecki, T. V., Kochar, A., & Frank, M. J. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0035813 · Cited by 289 · see citing works
The skills of document use: from text comprehension to Web-based learning
Rouet, J. (2007) · Choice Reviews Online · doi:10.5860/choice.44-5174 · Cited by 260 · see citing works
Self-citationThe limitations of term co-occurrence data for query expansion in document retrieval systems
Peat, H. J., & Willett, P. (1991) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199106)42:5 · Cited by 254 · see citing works
How users assess Web pages for information seeking
Tombros, A., Ruthven, I., & Jose, J. M. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20106 · Cited by 202 · see citing works
Impact of Popularity Indications on Readers' Selective Exposure to Online News
Knobloch‐Westerwick, S., Sharma, N., Hansen, D. L., & Alter, S. M. (2005) · Journal of Broadcasting & Electronic Media · doi:10.1207/s15506878jobem4903_3 · Cited by 191 · see citing works
Rouet, J., Ros, C., Goumi, A., Macedo‐Rouet, M., & Dinet, J. (2010) · Learning and Instruction · doi:10.1016/j.learninstruc.2010.02.007 · Cited by 129 · see citing works
Self-citationMaglaughlin, K. L., & Sonnenwald, D. H. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10049 · Cited by 119 · see citing works
The time course of gaze bias in visual decision tasks
Glaholt, M. G., & Reingold, E. M. (2008) · Visual Cognition · doi:10.1080/13506280802362962 · Cited by 112 · see citing works
Fu, W., & Sim, C. (2011) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21641 · Cited by 97 · see citing works
Keselman, H. J., Algina, J. J., Kowalchuk, R. K., & Wolfinger, R. D. (1999) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929908832460 · Cited by 96 · see citing works
Literacy as Multidimensional: Locating Information and Reading Comprehension
Guthrie, J. T., & Mosenthal, P. B. (1987) · Educational Psychologist · doi:10.1080/00461520.1987.9653053 · Cited by 96 · see citing works
Eye movements in the visual search of word lists
Ojanpää, H., Näsänen, R., & Kojo, I. (2002) · Vision Research · doi:10.1016/s0042-6989(02)00077-9 · Cited by 84 · see citing works
Document representations and clues to document relevance
Barry, C. L. (1998) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(1998)49:14 · Cited by 74 · see citing works
Information resource selection of undergraduate students in academic search tasks
Lee, J. Y., Paik, W., & Joo, S. (2012) · Cited by 71 · see citing works
Modeling Semantic and Structural Knowledge in Web Navigation
Juvina, I., & van Oostendorp, H. (2008) · Discourse Processes · doi:10.1080/01638530802145205 · Cited by 65 · see citing works
Tips and Strategies for Mixed Modeling with SAS/STAT ® Procedures
Kiernan, K., Tao, J., & Gibbs, P. (2012) · Cited by 62 · see citing works
Keil, F. C., & Kominsky, J. F. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067777 · Cited by 26 · see citing works
Flagged first, then by citations
2 of the 31 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Information search behavior in fragile and conflict-affected learning contexts
AlDahdouh, A. (2021) · The Internet and Higher Education · doi:10.1016/j.iheduc.2021.100808 · Cited by 20 · see citing works
I forgot that you existed: Role of memory accessibility in the gender citation gap.
Yan, V. X., Arndt, A., Muenks, K. M., & Henderson, M. D. (2024) · American Psychologist · doi:10.1037/amp0001299 · Cited by 8 · see citing works
Kiili, C., Brante, E. W., Räikkönen, E., & Coiro, J. (2020) · Journal for the Study of Education and Development Infancia y Aprendizaje · doi:10.1080/02103702.2019.1690839 · Cited by 8 · see citing works
“Las investigaciones previas sugieren que algunos adolescentes utilizan estrategias superficiales para seleccionar y evaluar los textos en línea (Coiro et al., 2015; Macedo-Rouet et al., 2019; Rouet et al., 2018).”
BackgroundSevryugina, Y. V., & Vargas, D. (2026) · Journal of Chemical Education · doi:10.1021/acs.jchemed.5c01620 · Cited by 4 · see citing works
Londra, F., Politti, M. E., & Saux, G. (2020) · PSICOLOGÍA UNEMI · doi:10.29076/issn.2602-8379vol4iss7.2020pp40-52p · Cited by 5 · see citing works
“En otro experimento, Rouet et al. (2018) solicitaron a sus participantes seleccionar uno de dos documentos presentados en un resultado de búsqueda tipo Google, con la consigna de realizar un trabajo ficticio para la universidad.”
BackgroundLondra, F., & Saux, G. (2023) · Reading Psychology · doi:10.1080/02702711.2023.2179144 · Cited by 3 · see citing works
“…lean to more shallow cues such as the looks of a website (Wineburg & McGrew, 2019), personal preferences of search engines (Hargittai et al., 2010), number of citations (Rouet et al., 2018), and be influenced by their beliefs about knowledge (Barzilai & Eshet-Alkalai, 2015), among other factors.”
Background
- Authors
- Rouet, J.-F., Skov, O., De Pereyra, G., Ros, C., Le Bigot, L., & Vibert, N.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Venue
- Journal of Experimental Psychology: Applied
- Published
- 2018-12
- Volume
- 24
- Issue
- 4
- Pages
- 476-489
- ISSN
- 1076-898X (ISSN-L)
- 1939-2192
- OpenAlex
- W2895993948
- Semantic Scholar
- 343853c6b2534e044f55080bca06ff21e86d99f1
- Cited by
- 6
- References
- 31
- Citations by year
- 2026: 1 · 2024: 1 · 2023: 1 · 2021: 1 · 2020: 2
- FWCI
- 7.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology Applied (publishedVersion)
- PubMed (publishedVersion)
- HAL (Le Centre pour la Communication Scientifique Directe) (submittedVersion)
- University of Southern Denmark Research Portal (University of Southern Denmark) (submittedVersion)
- Funders
- French National Research Agency (ANR-11-IDFI-0028)
- Topics
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Psychology › Developmental and Educational Psychology › Educational Strategies and Epistemologies
- Keywords
- citation counts, reference selection, heuristic decision making, semantic relevance, bibliographic databases, eye tracking, psychology students, information seeking
- MeSH terms
- Heuristics, Adolescent, Adult, Bibliographies as Topic, Decision Making, Female, Humans, Male, Middle Aged, Students, Knowledge, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
