An eye‐tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine
Abstract
Eye movement data can provide an in‐depth view of human reasoning and the decision‐making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk‐aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the G oogle search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (T itle, S ummary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Book Review: Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,361 · see citing works
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537 · see citing works
Rayner, K. (2009) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210902816461 · Cited by 2,682 · see citing works
Eye Tracking Methodology: Theory and Practice
Duchowski, A. T. (2003) · Springer eBooks · doi:10.1007/978-1-4471-3750-4 · Cited by 2,247 · see citing works
Eye Tracking Methodology; Theory and Practice
(2007) · Qualitative Market Research An International Journal · doi:10.1108/13522750710740862 · Cited by 1,318 · see citing works
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715 · see citing works
Eye-tracking analysis of user behavior in WWW search
Granka, L. A., Joachims, T., & Gay, G. K. (2004) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1008992.1009079 · Cited by 701 · see citing works
Judgment of information quality and cognitive authority in the Web
Rieh, S. Y. (2001) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10017 · Cited by 615 · see citing works
Judgment of information quality and cognitive authority in the Web
Rieh, S. Y. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10017.abs · Cited by 582 · see citing works
Serendipity and information seeking: an empirical study
Foster, A. E., & Ford, N. J. (2003) · Journal of Documentation · doi:10.1108/00220410310472518 · Cited by 523 · see citing works
The concept of relevance in IR
Borlund, P. (2003) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10286 · Cited by 489 · see citing works
What are you looking for?: an eye-tracking study of information usage in web search
Cutrell, E., & Guan, Z. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240690 · Cited by 460 · see citing works
A re-examination of relevance: toward a dynamic, situational definition∗
Schamber, L., Eisenberg, M. B., & Nilan, M. S. (1990) · Information Processing & Management · doi:10.1016/0306-4573(90)90050-c · Cited by 429 · see citing works
Eye tracking in web search tasks
Goldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N. G., & Wichansky, A. M. (2002) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/507072.507082 · Cited by 384 · see citing works
Users' criteria for relevance evaluation: A cross-situational comparison
Barry, C. L., & Schamber, L. (1998) · Information Processing & Management · doi:10.1016/s0306-4573(97)00078-2 · Cited by 276 · see citing works
What do you see when you're surfing?
Buscher, G., Cutrell, E. B., & Morris, M. R. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518705 · Cited by 254 · see citing works
The influence of task and gender on search and evaluation behavior using Google
Lorigo, L., Pan, B., Hembrooke, H. A., Joachims, T., Granka, L. A., & Gay, G. K. (2005) · Information Processing & Management · doi:10.1016/j.ipm.2005.10.001 · Cited by 252 · see citing works
Spink, A. H., Greisdorf, H., & Bateman, J. (1998) · Information Processing & Management · doi:10.1016/s0306-4573(98)00025-9 · Cited by 242 · see citing works
The development of a method for the evaluation of interactive information retrieval systems
Borlund, P., & Ingwersen, P. (1997) · Journal of Documentation · doi:10.1108/eum0000000007198 · Cited by 240 · see citing works
How users assess Web pages for information seeking
Tombros, A., Ruthven, I., & Jose, J. M. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20106 · Cited by 202 · see citing works
Self-citationEye tracking and online search: Lessons learned and challenges ahead
Lorigo, L., Haridasan, M., Brynjarsdóttir, H., Ling, X., Joachims, T., Gay, G. K., Granka, L. A., Pellacini, F., & Pan, B. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20794 · Cited by 188 · see citing works
Gerjets, P. H., Kammerer, Y., & Werner, B. (2010) · Learning and Instruction · doi:10.1016/j.learninstruc.2010.02.005 · Cited by 186 · see citing works
Eye-mouse coordination patterns on web search results pages
Rodden, K., Fu, X., Aula, A., & Spiro, I. (2008) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1358628.1358797 · Cited by 167 · see citing works
Eye-Tracking Reveals the Personal Styles for Search Result Evaluation
Aula, A., Majaranta, P., & Räihä, K. (2005) · Lecture notes in computer science · doi:10.1007/11555261_104 · Cited by 112 · see citing works
Individual differences in gaze patterns for web search
Dumais, S., Buscher, G., & Cutrell, E. B. (2010) · Information Interaction in Context Symposium (IIiX) · doi:10.1145/1840784.1840812 · Cited by 104 · see citing works
Interactive information retrieval
Ruthven, I. G. (2008) · Annual Review of Information Science and Technology · doi:10.1002/aris.2008.1440420109 · Cited by 88 · see citing works
Self-citationEisenberg, M. B. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90042-8 · Cited by 79 · see citing works
User‐defined relevance criteria in web searching
Savolainen, R., & Kari, J. (2006) · Journal of Documentation · doi:10.1108/00220410610714921 · Cited by 67 · see citing works
Text or Pictures? An Eyetracking Study of How People View Digital Video Surrogates
Hughes, A., Wilkens, T., Wildemuth, B. M., et al. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45113-7_27 · Cited by 63 · see citing works
Relevance criteria identified by health information users during Web searches
Crystal, A., & Greenberg, J. (2006) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20436 · Cited by 62 · see citing works
Tang, R., & Solomon, P. J. (2001) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.1116 · Cited by 53 · see citing works
Using Eye Tracking to Evaluate Alternative Search Results Interfaces
Rele, R. S., & Duchowski, A. T. (2005) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120504901508 · Cited by 42 · see citing works
Electronic news delivery project
Watters, C. R., Shepherd, M. A., & Burkowski, F. J. (1998) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(1998)49:2 · Cited by 30 · see citing works
An evaluation framework of user interaction with metadata surrogates
Balatsoukas, P., Morris, A., & O’Brien, A. (2009) · Journal of Information Science · doi:10.1177/0165551508099090 · Cited by 19 · see citing works
Self-citationSearchers' relevance judgments and criteria in evaluating web pages in a learning style perspective
Papaeconomou, C., Zijlema, A., & Ingwersen, P. (2008) · Information Interaction in Context Symposium (IIiX) · doi:10.1145/1414694.1414722 · Cited by 18 · see citing works
A user‐centered functional metadata evaluation of moving image collections
Zhang, Y., & Li, Y. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20839 · Cited by 14 · see citing works
The use of relevance criteria during predictive judgment: An eye tracking approach
Balatsoukas, P., & Ruthven, I. G. (2010) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14504701145 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
4 of the 42 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Hughes A., 2003, Proceedings of the Second International Conference CIVR 2003 Urbana‐Champaign, IL, July 24–25, 271
- Saracevic T., 1989, Modeling and measuring user‐intermediary computer interaction in online searching: Design of a study, Proceedings of the Annual Meeting of the American Society for Information Science and Technology, 26, 75
- 10.1002/(sici)1097-4571(199802)49:2<115::aid-asi3>3.0.co;2-t · DOI
Cited by
Sorted by citations
Characterizing relevance with eye-tracking measures
Jacek Gwizdka, A. (2014) · Information Interaction in Context Symposium (IIiX) · doi:10.1145/2637002.2637011 · Cited by 94 · see citing works
“Balatsoukas & Ruthven [2] describe an extensive study of relevance and eye-tracking measures.”
BackgroundLewandowski, D., & Kammerer, Y. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1761450 · Cited by 84 · see citing works
“Balatsoukas and Ruthven (2012) asked participants to conduct a Web search on a self-chosen topic and analysed participants ’ total fi xation times and the number of fi xations on titles, descriptions, and URLs of search results.”
BackgroundTemporal dynamics of eye‐tracking and EEG during reading and relevance decisions
Jacek Gwizdka, A., Hosseini, R., Cole, M., et al. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23904 · Cited by 82 · see citing works
“In a think-aloud study by Balatsoukas and Ruthven (2012), users made more frequent and longer fixations on nonrelevant document surrogates.”
BackgroundUndergraduate Students’ Justifications for Source Selection in a Digital Academic Context
List, A., Grossnickle, E. M., & Alexander, P. A. (2015) · Journal of Educational Computing Research · doi:10.1177/0735633115606659 · Cited by 74 · see citing works
“Research on the justifications students offer for source selection is limited (Bilal & Kirby, 2002) and has largely focused on nonepistemic dimensions of source selection, such as source relevance (e.g., Balatsoukas & Ruthven, 2012; Gerjets et al., 2011; Lee et al., 2012; Rouet et al., 2011).”
BackgroundSaračević, T. (2016) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00723ed1v01y201607icr050 · Cited by 65 · see citing works
The effects of credibility cues on the selection of search engine results
Unkel, J., & Haas, A. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23820 · Cited by 59 · see citing works
“The ranking furthermore influences time afforded to and the quality of these subjective judgments (Balatsoukas & Ruthven, 2012).”
BackgroundGeorgas, H. (2014) · portal Libraries and the Academy · doi:10.1353/pla.2014.0034 · Cited by 55 · see citing works
Sixth graders’ evaluation strategies when reading Internet search results: an eye-tracking study
Hautala, J., Kiili, C., Kammerer, Y., Loberg, O., Hokkanen, S., & Leppänen, P. H. T. (2018) · Behaviour and Information Technology · doi:10.1080/0144929x.2018.1477992 · Cited by 52 · see citing works
“In addition to evaluating the semantic relevance of a search result, the expected quality of information (or credibility of an information source) can also play a role in a user ’ s selection decisions (e.g. Rieh 2002; Balatsoukas and Ruthven 2012; Kammerer and Gerjets 2014).”
BackgroundMultidimensional relevance modeling via psychometrics and crowdsourcing
Zhang, Y., Zhang, J., Lease, M., & Jacek Gwizdka, A. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2609577 · Cited by 49 · see citing works
Effects of task complexity on online search behavior of adolescents
Walhout, J., Oomen, P., Jarodzka, H. M., & Brand‐Gruwel, S. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23782 · Cited by 48 · see citing works
“The viewing and result selection patterns were in line with what was reported by previous studies (Balatsoukas & Ruthven, 2012; H € ochst € otter & Lewandowski, 2009; Pan et al., 2007; Saven-kov, Braslavski, & Lebedev, 2011): The top search results were viewed and selected more often than the ones…”
ResultCharacteristics of human perception and their relevance when studying information behavior
Lueg, C. (2014) · Journal of Documentation · doi:10.1108/jd-05-2012-0064 · Cited by 44 · see citing works
Eye tracking in library and information science: a literature review
Lund, H. (2016) · Library Hi Tech · doi:10.1108/lht-07-2016-0085 · Cited by 44 · see citing works
Motivation determines Facebook viewing strategy: An eye movement analysis
Scott, G. G., & Hand, C. J. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.11.029 · Cited by 43 · see citing works
Eye-Tracking Studies of Web Search Engines: A Systematic Literature Review
Strzelecki, A. (2020) · Information · doi:10.3390/info11060300 · Cited by 38 · see citing works
“Balatsoukas and Ruthven (2012) [75] 24 Google Students Tobii T60 Desktop Organic Temporal, count Regular Informational English Table”
BackgroundMethodInfluentialThe missing link: Information behavior research and its estranged relationship with embodiment
Lueg, C. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23441 · Cited by 34 · see citing works
“…such as gaze-trackers and psycho-physiological monitoring technologies coming from cognate fields including psychology, neuroscience, and ubiquitous computing are making their way into information science research (e.g., Balatsoukas & Ruthven, 2012; Erdelez, Miwa, Lund, & Gwizdka, 2009).”
BackgroundDetecting Relevance during Decision-Making from Eye Movements for UI Adaptation
Feit, A. M., Vordemann, L., Park, S., et al. (2020) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3379155.3391321 · Cited by 25 · see citing works
“[Kajan et al. 2016; Klami 2010; Nguyen and Liu 2016; Salojärvi et al. 2005] Verification SFX Sum of Fixations The total number of fixations within an AOI. A high SFX value indicates high relevance. [Balatsoukas and Ruthven 2012; Kajan et al. 2016; Klami 2010] ODT Overall Dwell Time The total time spent looking at an A…”
Salminen, J. O., Jansen, B. J. J., An, J., Jung, S., Nielsen, L., & Kwak, H. (2018) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3176349.3176391 · Cited by 23 · see citing works
Display advertising: the role of context and advertising appeals from a resistance perspective
Tsiotsou, R. H., Hatzithomas, L., & Wetzels, M. G. M. (2023) · Journal of Research in Interactive Marketing · doi:10.1108/jrim-09-2022-0302 · Cited by 22 · see citing works
Sanchiz, M., Amadieu, F., & Chevalier, A. (2020) · Human-computer interaction series · doi:10.1007/978-3-030-38825-6_5 · Cited by 18 · see citing works
Muntinga, T., & Taylor, G. (2017) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2017.1368949 · Cited by 19 · see citing works
Mariussen, A. (2014) · The International Journal of Digital Accounting Research · doi:10.4192/1577-8517-v14_6 · Cited by 17 · see citing works
“…they had sufficient computer and Internet experience (Jordan et al., 2013; Pan and Fesenmaier, 2006), did not have any visual impairments that could disturb eye-tracking (Balatsoukas and Ruthven, 2012) and had not previously visited the sample destination of Morocco (Pan and Fesenmaier, 2006).”
BackgroundMethodPian, W., Khoo, C. S. G., & Chi, J. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8354 · Cited by 16 · see citing works
“Compared with user browsing patterns such as reading time and mouse clicks [9-14], eye movement data provide more insights on user’s relevance decision on whether health information is relevant [4,7].”
BackgroundNews stories relevance effects on eye-movements
Jacek Gwizdka, A. (2014) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/2578153.2578198 · Cited by 15 · see citing works
Pian, W., Khoo, C. S. G., & Chang, Y. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5513 · Cited by 15 · see citing works
Lowe, M. S., Maxson, B. K., Stone, S. M., Miller, W. M., Snajdr, E. A., & Hanna, K. (2018) · College & Research Libraries · doi:10.5860/crl.79.4.517 · Cited by 17 · see citing works
Selecting Fiction in Library Catalogs: A Gaze Tracking Study
Pöntinen, J., & Vakkari, P. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40501-3_8 · Cited by 13 · see citing works
Users’ relevance criteria for video in leisure contexts
Albassam, S. A. A., & Ruthven, I. G. (2017) · Journal of Documentation · doi:10.1108/jd-06-2017-0081 · Cited by 13 · see citing works
Self-citationHienert, D., Kern, D., Mitsui, M. R., Shah, C. S., & Belkin, N. J. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298921 · Cited by 12 · see citing works
“In (I)IR, interesting findings between gaze behavior and concepts such as relevance [2, 18], user interest [1, 16], knowledge level [13] or task types [15] has been found.”
BackgroundRelationship between the Metadata and Relevance Criteria of Scientific Data
Zhang, G., Wang, J., Liu, J., & Pan, Y. (2021) · Data Science Journal · doi:10.5334/dsj-2021-005 · Cited by 9 · see citing works
“The study used eye tracker collecting eye movement data, such as the number and length of fixations, which could reveal a more accurate picture of the cognitive effort spent by users during the relevance judgment process (Balatsoukas, 2012).”
BackgroundMethodInfluentialLi, Q., Wang, P., Liu, C., Li, X., & Hou, J. (2024) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24961 · Cited by 9 · see citing works
“Prior research on users' information relevance evaluation in the context of web search engines has primarily adopted two perspectives: cognitive and behavioral (Balatsoukas & Ruthven, 2012).”
Exploring information interactions in the context of G oogle
Johnson, F., Rowley, J., & Sbaffi, L. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23443 · Cited by 7 · see citing works
“Balatsoukas and Ruthven (2012) observed participants’ eye movements and relevance judgments in the context of web search and confirmed topicality as the core criterion but also identified scope, tangibility, and a new dimension, serendipity, or serendipitious judgments, being purposefully used when participants had no…”
BackgroundInfluentialRousidis, D., Garoufallou, E., Balatsoukas, P., Paraskeuopoulos, K., Asderi, S., & Koutsomiha, D. (2013) · Communications in computer and information science · doi:10.1007/978-3-319-03437-9_25 · Cited by 5 · see citing works
Self-citationBetter Understanding Procedural Search Tasks: Perceptions, Behaviors, and Challenges
Choi, B., Casteel, S., Arguello, J., & Capra, R. (2023) · ACM Transactions on Information Systems · doi:10.1145/3630004 · Cited by 4 · see citing works
“For example, topicality, reputation, depth, scope, and understandability of the information are important relevance criteria that have been documented in prior work [4, 5, 17, 38, 42, 51].”
BackgroundExplore eye movement patterns in search result evaluation and individual document evaluation
Wang, Y., Xie, I., & Lee, S. (2015) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2015.1450520100144 · Cited by 4 · see citing works
“According to past research, the minimum length of fixations was set at 200 milliseconds (Balatsoukas and Ruthven, 2012).”
BackgroundMethodSearch transition as a measure of effort in information retrieval interaction
Nordlie, R., & Pharo, N. (2013) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14505001044 · Cited by 4 · see citing works
“…defined the measurement definitions range widely, from “pupil dilation” (Lorigo et al., 2006) or number of eye fixations and time spent on fixation (Balatsoukas & Ruthven, 2012) in eye-tracking studies of search and evaluation behavior to “number of iterations, i.e. queries in a search” ( Belkin…”
BackgroundInformation Relevance Factors of Argument Quality for E-Commerce Consumer Review
Lah, N. S. B. C., Hussin, A. R. C., & Dahlan, H. M. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-99007-1_81 · Cited by 4 · see citing works
How We Perceive Search Engines
Penna, L., & Quaresma, M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20889-3_8 · Cited by 4 · see citing works
“Fixation in the selection criteria, according to the position (Source: [10], p.”
BackgroundInfluentialVisualizing Searcher Gaze Patterns
Abualsaud, M., Smucker, M. D., & Clarke, C. L. A. (2021) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446041 · Cited by 3 · see citing works
“Balatsoukas and Ruthven [4] overlay SERPs with fixations and other information, similar to heatmaps.”
BackgroundA Versatile Dataset of Mouse and Eye Movements on Search Engine Results Pages
Latifzadeh, K., Jacek Gwizdka, A., & Leiva, L. A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730325 · Cited by 3 · see citing works
Looking for query terms on search engine results pages
Dickerhoof, A. M., & Smith, C. L. (2014) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.2014.14505101156 · Cited by 3 · see citing works
Scientific Data Relevance Criteria Classification and Usage
Zhang, G., Wang, J., Zhou, G., Liu, J., & Wei, C. (2018) · International Conference on Computer Science and Application Engineering (CSAE) · doi:10.1145/3207677.3278010 · Cited by 3 · see citing works
“In terms of the use of relevance criteria, the most frequently used criteria are topicality, followed by quality, which is consistent with previous studies [12,23].”
BackgroundResultEye Gaze Relevance Feedback Indicators for Information Retrieval
Akuma, S. (2022) · International Journal of Intelligent Systems and Applications · doi:10.5815/ijisa.2022.01.05 · Cited by 3 · see citing works
“Balatsoukas and Ruthven [28] produced a contrary result from previous studies, they found that users spent a longer time fixating on non-relevant document surrogates.”
ResultBehnert, C. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298978 · Cited by 2 · see citing works
“…Zijlema and Ingwersen [15], participants’ eye movements were recorded to explore the “relevance hot spots” of Web pages applied during the judgment process, while Balatsoukas and Ruthven [1] investigate relevance criteria in judgment process on Google also applying an eye-tracking approach.”
MethodSearch Results Viewing Behavior vis-à-vis Relevance Criteria
Jacek Gwizdka, A., & Chang, Y. (2019) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-28144-1_20 · Cited by 2 · see citing works
GazBy: Gaze-Based BERT Model to Incorporate Human Attention in Neural Information Retrieval
Dong, S., Goldstein, J., & Yang, G. H. (2022) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3539813.3545129 · Cited by 2 · see citing works
“The use of gaze has been explored in the IR literature, too [3, 19, 35, 37, 58].”
AdSight: Scalable and Accurate Quantification of User Attention in Multi-Slot Sponsored Search
Villaizán-Vallelado, M., Salvatori, M., Latifzadeh, K., et al. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729891 · Cited by 1 · see citing works
“To ensure data quality, events lasting less than 100ms are excluded from both metrics, following recommendations from prior studies [11, 49].”
Connaway, L. S., Faniel, I. M., Narayan, B., et al. (2019) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.80 · Cited by 1 · see citing works
“…research published between 2012 and 2014, Greifeneder (2014) reported that other methods, such as Delphi studies (Poirier & Robinson, 2014), eye-tracking (e.g. Balatsoukas & Ruthven, 2012; Wildemuth, 2009), log fi le analysis (Jiang, 2014), or participatory designs were used fewer than fi ve times.”
MethodRelevance judgment when browsing a health discussion forum : content analysis of eye fixations
Pian, W., Khoo, C. S. G., & Chang, Y. (2014) · Library and Information Science Research E-Journal · doi:10.32655/libres.2014.2.6 · Cited by 1 · see citing works
Anúncios Orgânicos e Patrocinados: estudo sobre intenção de compra online e comportamento visual
de Souza, M. T. (2018) · doi:10.11606/d.96.2018.tde-28032018-153527 · Cited by 1 · see citing works
Reading Protocol: Understanding what has been Read in Interactive Information Retrieval TasksPreprint
Hienert, D., Kern, D., Mitsui, M. R., Shah, C. S., & Belkin, N. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04262 · Cited by 1 · see citing works
“In (I)IR, interesting findings between gaze behavior and concepts such as relevance [2, 18], user interest [1, 16], knowledge level [13] or task types [15] has been found.”
BackgroundPharo, N., & Nordlie, R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-41057-4_5
“Eye-tracking software can also log the searchers’ eye movements over the screen [9].”
BackgroundRelevance judgment during health information seeking on a discussion forum
Pian, W. (2016) · doi:10.32657/10356/67016
“Other scholars also conducted studies on relevance judgment and quality judgment by tracking people’s eye fixations and compute the fixation duration for online searching results (Google surrogates) (Balatsoukas & Ruthven, 2012).”
BackgroundPalla, P. J., & Zotos, Y. C. (2016) · Springer eBooks · doi:10.1007/978-3-658-15220-8_6
Katse kaunokirjallisuuteen. Kaunokirjallisuuden valinta verkkopalvelun avulla.
Pöntinen, J. (2013) · Tampere University Institutional Repository (Tampere University)
The Changing Nature of Information Behaviour
Rowley, J. (2014) · Advances in information quality and management · doi:10.4018/978-1-4666-5888-2.ch389
Do our eyes mirror our information experience
Arvanitaki, K. J. (2016) · Utrecht University Repository (Utrecht University)
Jasiewicz, J., Kisilowska, M., & Jupowicz-Ginalska, A. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-13472-3_3
Considerations for Eye Tracking Experiments in Information RetrievalPreprint
Ortiz, M. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.12665
“relevant, partial, not) and the relationship to length of eye-fixations [5] while Marcos et al.”
BackgroundBrennan, K. P. J. (2019) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/dkvv-yv68
Lah, N. S. B. C., Hussin, A. R. C., Jalil, N. A., et al. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-030-98741-1_50
(2022) · doi:10.1632/eczy4186
Investigating Persona Viewing Behavior: An Eye-Tracking Study on Portrait-Format Persona Profile
Salminen, J. O., Jung, S., & Jansen, B. J. J. (2024) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3679318.3685376
“Eye tracking can reveal interaction patterns toward navigational and content elements and provide design recommendations [4, 5, 16].”
Investigating exploratory book search through methodological triangulation
Kuhar, M. (2026) · Aslib Journal of Information Management · doi:10.1108/ajim-06-2025-0434
- Authors
- Balatsoukas, P., & Ruthven, I.
- Affiliations
- University of Strathclyde (GB)
- Corresponding author
- Balatsoukas, Panos
- Ruthven, Ian
- Venue
- Journal of the American Society for Information Science and Technology
- Publisher
- Wiley
- Published
- 2012-08-17
- Volume
- 63
- Issue
- 9
- Pages
- 1728-1746
- ISSN
- 1532-2882 (ISSN-L)
- 1532-2890
- OpenAlex
- W1982065914
- Semantic Scholar
- dae64696e305764b3f8fdc6d8fc4a6d52a8681ac
- Cited by
- 66
- References
- 42
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 2 · 2023: 3 · 2022: 4 · 2021: 2 · 2020: 6 · 2019: 8 · 2018: 6 · 2017: 6 · 2016: 6 · 2015: 6 · 2014: 9 · 2013: 5
- FWCI
- 17.41
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the American Society for Information Science and Technology (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Strathprints: The University of Strathclyde institutional repository (University of Strathclyde) (submittedVersion)
- Funders
- Arts and Humanities Research Council (AH/G013233/1)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- eye tracking, relevance assessment, relevance criteria, Google search, visual behavior, search result ranking, cognitive effort, information retrieval, user decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
