The DOI matches its Crossref record; 241 of the 241 references it cites were found in a registry or an index.
Methods for Evaluating Interactive Information Retrieval Systems with Users
Abstract
This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users. The primary goal of this article is to catalog and compile material related to this topic into a single source. This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies. This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR). Finally, this article concludes with a discussion of outstanding challenges and future research directions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.5.879 · Cited by 82,194
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
The Discovery of Grounded Theory: Strategies for Qualitative Research.
Wagner, H. R., Glaser, B. G., & Strauss, A. L. (1968) · Social Forces · doi:10.2307/2575405 · Cited by 46,361
Nonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,245
Basics of Qualitative Research: Grounded Theory Procedures and Techniques.
Ellis, C., Strauss, A., & Corbin, J. (1992) · Contemporary Sociology · doi:10.2307/2074814 · Cited by 29,249
Handbook of Qualitative Research
Crozier, G., Denzin, N., & Lincoln, Y. (1994) · British Journal of Educational Studies · doi:10.2307/3121684 · Cited by 26,666
(1963) · Medical Entomology and Zoology · Cited by 15,085
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
The Practice of Social Research
Neustadtl, A., & Babbie, E. (1989) · Teaching Sociology · doi:10.2307/1318433 · Cited by 14,398
(1996) · Cited by 11,584
Qualitative Data Analysis: A Sourcebook of New Methods
Lotto, L. S., Miles, M. B., & Huberman, A. M. (1986) · Educational Evaluation and Policy Analysis · doi:10.2307/1163741 · Cited by 10,315
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 1×
“At its inception, it was intended as a quantitative method, although there are now a number of variations, interpretations and uses of content analysis [201].”
MethodPicard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Computer Self-Efficacy: Development of a Measure and Initial Test
Compeau, D. R., & Higgins, C. A. (1995) · MIS Quarterly · doi:10.2307/249688 · Cited by 6,553
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
The Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734
Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,347
Measurement, Design, and Analysis: An Integrated Approach.
Funk, S. G., Pedhazur, E. J., & Schmelkin, L. P. (1992) · Journal of the American Statistical Association · doi:10.2307/2290248 · Cited by 4,018
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537
Finding flow: the psychology of engagement with everyday life
(1997) · Choice Reviews Online · doi:10.5860/choice.35-1828 · Cited by 3,522
Fundamental Statistics in Psychology and Education.
Smith, B. B., & Guilford, J. P. (1951) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2980908 · Cited by 3,455
Theories of Human Communication
W.Littlejohn, S. (1978) · Cited by 3,011
Validity and reliability of the Experience-Sampling Method
CSIKSZENTMIHALYI, M., & LARSON, R. (1987) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-198709000-00004 · Cited by 2,544
Robertson, S. E., Walker, S., Jones, S., Hancock‐Beaulieu, M., & Gatford, M. (1994) · Text REtrieval Conference · Cited by 2,212
Qualitative Interviewing and Grounded Theory Analysis
Charmaz, K., & Belgrave, L. L. (2012) · The SAGE Handbook of Interview Research: The Complexity of the Craft · doi:10.4135/9781452218403.n25 · Cited by 1,997
Development of an instrument measuring user satisfaction of the human-computer interface
Chin, J. P., Diehl, V. A., & Norman, L. K. (1988) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '88 · doi:10.1145/57167.57203 · Cited by 1,609
Eye Tracking in Human-Computer Interaction and Usability Research
Jacob, R. J. K., & Karn, K. S. (2003) · The Mind's Eye · doi:10.1016/b978-044451020-4/50031-1 · Cited by 1,567
ON USER STUDIES AND INFORMATION NEEDS
WILSON, T. D. (1981) · Journal of Documentation · doi:10.1108/eb026702 · Cited by 1,565
Looking for Information: A Survey of Research on Information Seeking, Needs, and Behavior
(2016) · Studies in Information · doi:10.1108/s2055-53772016 · Cited by 1,540
Seeking Meaning: A Process Approach to Library and Information Services
Durrance, J., & Kuhlthau, C. C. (1994) · Journal of Education for Library and Information Science · doi:10.2307/40323160 · Cited by 1,536
What we talk about when we talk about context
Dourish, P. (2004) · Personal and Ubiquitous Computing · doi:10.1007/s00779-003-0253-8 · Cited by 1,529
Exploratory search: from finding to understanding
Marchionini, G. (2006) · Communications of the ACM · doi:10.1145/1121949.1121979 · Cited by 1,506
The vocabulary problem in human-system communication
Furnas, G. W., Landauer, T. K., Gomez, L. M., & Dumais, S. T. (1987) · Communications of the ACM · doi:10.1145/32206.32212 · Cited by 1,504
IR evaluation methods for retrieving highly relevant documents
Järvelin, K., & Kekäläinen, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130374 · Cited by 1,444
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
What is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397
Cognitive Styles—an overview and integration
Riding, R., & Cheema, I. (1991) · Educational Psychology · doi:10.1080/0144341910110301 · Cited by 1,181
Current practice in measuring usability: Challenges to usability studies and research
Hornbæk, K. (2006) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2005.06.002 · Cited by 1,123
Applications of Social Research Methods to Questions in Information and Library Science
(2016) · Libraries Unlimited eBooks · doi:10.5040/9798400613739 · Cited by 1,107
Statistics for behavioral sciences
Gravetter, F. J., & Wallnau, L. B. (1985) · Cited by 1,104
THE PROBABILITY RANKING PRINCIPLE IN IR
ROBERTSON, S. E. (1977) · Journal of Documentation · doi:10.1108/eb026647 · Cited by 1,103
Improving web search ranking by incorporating user behavior information
Agichtein, E., Brill, E., & Dumais, S. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148177 · Cited by 1,041
Everyday life information seeking: Approaching information seeking in the context of “way of life”
Savolainen, R. (1995) · Library & Information Science Research · doi:10.1016/0740-8188(95)90048-9 · Cited by 1,032
Task Characteristics and the Experience of Optimal Flow in Human—Computer Interaction
Ghani, J. A., & Deshpande, S. P. (1994) · The Journal of Psychology · doi:10.1080/00223980.1994.9712742 · Cited by 1,005
Placing search in context: the concept revisited.
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., & Ruppin, E. (2002) · Cited by 961
Hancock, P. A., & Meshkati, N. (1988) · Cited by 954
Task complexity affects information seeking and use
Byström, K., & Järvelin, K. (1995) · Information Processing & Management · doi:10.1016/0306-4573(94)00041-z · Cited by 885
(2005) · The Information Retrieval Series · doi:10.1007/1-4020-3851-8 · Cited by 864
RELEVANCE: A review of and a framework for the thinking on the notion in information science
Saracevic, T. (1975) · Journal of the American Society for Information Science · doi:10.1002/asi.4630260604 · Cited by 863
Psychometrics: An Introduction.
Furr, R. M., & Bacharach, V. R. (2007) · Cited by 846
Fundamental Statistics in Psychology and Education
Parish, L., & Guilford, J. P. (1957) · British Journal of Educational Studies · doi:10.2307/3118885 · Cited by 839
TREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
A diary study of task switching and interruptions
Czerwinski, M., Horvitz, E., & Wilhite, S. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985715 · Cited by 830
How are we searching the World Wide Web? A comparison of nine search engine transaction logs
Jansen, B. J., & Spink, A. (2006) · Information Processing & Management · doi:10.1016/j.ipm.2004.10.007 · Cited by 801
Personalizing search via automated analysis of interests and activities
Teevan, J., Dumais, S. T., & Horvitz, E. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076111 · Cited by 800
Wizard of Oz studies: why and how
Dahlbäck, N., Jönsson, A., & Ahrenberg, L. (1993) · Proceedings of the 1st international conference on Intelligent user interfaces - IUI '93 · doi:10.1145/169891.169968 · Cited by 742
A Business Intelligence System
Luhn, H. P. (1958) · IBM Journal of Research and Development · doi:10.1147/rd.24.0314 · Cited by 728
Question-Negotiation and Information Seeking in Libraries
Taylor, R. S. (2015) · College & Research Libraries · doi:10.5860/crl.76.3.251 · Cited by 723
Question-Negotiation and Information Seeking in Libraries
Taylor, R. S. (1968) · College & Research Libraries · doi:10.5860/crl_29_03_178 · Cited by 721
COGNITIVE PERSPECTIVES OF INFORMATION RETRIEVAL INTERACTION: ELEMENTS OF A COGNITIVE IR THEORY
INGWERSEN, P. (1996) · Journal of Documentation · doi:10.1108/eb026960 · Cited by 717 · cited 2×
“Another way to develop information needs is to work with experts who would either be assigning such information needs to target users as work tasks, or have the same kinds of information needs [138].”
BackgroundStuff I've Seen: A System for Personal Information Retrieval and Re-Use
Dumais, S., Cutrell, E., Cadiz, J. J., Jancke, G., Sarin, R., & Robbins, D. C. (2016) · ACM SIGIR Forum · doi:10.1145/2888422.2888425 · Cited by 691
The impoverished life-world of outsiders
Chatman, E. A. (1996) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199603)47:3<193::aid-asi3>3.0.co;2-t · Cited by 681
Information Retrieval Interaction
Ingwersen, P. (1992) · Cited by 634
Judgment of information quality and cognitive authority in the Web
Rieh, S. Y. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10017 · Cited by 612
Judgment of information quality and cognitive authority in the Web
Rieh, S. Y. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10017.abs · Cited by 585
Mizzaro, S. (1997) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199709)48:9<810::aid-asi6>3.0.co;2-u · Cited by 573
The perfect search engine is not enough: a study of orienteering behavior in directed search
Teevan, J., Alvarado, C., Ackerman, M. S., & Karger, D. R. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985745 · Cited by 548
The CRANFIELD TESTS ON INDEX LANGUAGE DEVICES
CLEVERDON, C. (1967) · Aslib Proceedings · doi:10.1108/eb050097 · Cited by 509
The concept of relevance in IR
Borlund, P. (2003) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10286 · Cited by 482
Statistics As Principled Argument
Abelson, R. P. (2012) · Psychology Press eBooks · doi:10.4324/9781410601155 · Cited by 465
Katona, G., & Payne, S. L. (1951) · Econometrica · doi:10.2307/1907475 · Cited by 461
Theories of information behavior
(2006) · Choice Reviews Online · doi:10.5860/choice.43-4366 · Cited by 458
Measuring usability: preference vs. performance
Nielsen, J., & Levy, J. (1994) · Communications of the ACM · doi:10.1145/175276.175282 · Cited by 458
Qualitative Research Traditions: A Review
Jacob, E. (1987) · Review of Educational Research · doi:10.2307/1170355 · Cited by 430
Bates, M. J. (1979) · Journal of the American Society for Information Science · doi:10.1002/asi.4630300406 · Cited by 428
Placing search in context: the concept revisited
(2002) · ACM Transactions on Information Systems · doi:10.1145/503104.503110 · Cited by 425
SearchTogether: an interface for collaborative web search
Morris, M. R., & Horvitz, E. (2007) · Proceedings of the 20th annual ACM symposium on User interface software and technology · doi:10.1145/1294211.1294215 · Cited by 423
Hedonic and ergonomic quality aspects determine a software's appeal
Hassenzahl, M., Platz, A., Burmester, M., & Lehner, K. (2000) · Proceedings of the SIGCHI conference on Human Factors in Computing Systems · doi:10.1145/332040.332432 · Cited by 422
Handbook of Tables for Probability and Statistics.
Jones, D. A., & Beyer, W. E. (1968) · Applied Statistics · doi:10.2307/2985692 · Cited by 421
Factors determining the performance of indexing systems
Cleverdon, C. W., Mills, J., & Keen, M. (1966) · Medical Entomology and Zoology · Cited by 414
The IIR evaluation model: a framework for evaluation of interactive information retrieval systems
Borlund, P. (2003) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 403
Voorhees, E. M., & Harman, D. K. (2005) · The MIT Press eBooks · Cited by 392
User performance versus precision measures for simple search tasks
Turpin, A., & Scholer, F. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148176 · Cited by 389
Zucker, D. R. (2014) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat06924 · Cited by 388
Task‐based information searching
Vakkari, P. (2003) · Annual Review of Information Science and Technology · doi:10.1002/aris.1440370110 · Cited by 366
Formative design evaluation of superbook
Egan, D. E., Remde, J. R., Gomez, L. M., Landauer, T. K., Eberhardt, J., & Lochbaum, C. C. (1989) · ACM Transactions on Information Systems · doi:10.1145/64789.64790 · Cited by 342
A faceted approach to conceptualizing tasks in information seeking
Li, Y., & Belkin, N. J. (2008) · Information Processing & Management · doi:10.1016/j.ipm.2008.07.005 · Cited by 336
Methodological Issues and Practices in Qualitative Research
(1993) · The Library Quarterly · doi:10.1086/602620 · Cited by 333
Hsieh-Yee, I. (1993) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199304)44:3<161::aid-asi5>3.0.co;2-8 · Cited by 333
Crowdsourcing for relevance evaluation
Alonso, O., Rose, D. E., & Stewart, B. (2008) · ACM SIGIR Forum · doi:10.1145/1480506.1480508 · Cited by 329
Information Retrieval Experiment
Jones, K. S. (1971) · Cited by 299
Exploring Web users’ optimal flow experiences
Chen, H., Wigand, R. T., & Nilan, M. (2000) · Information Technology & People · doi:10.1108/09593840010359473 · Cited by 296
Search log analysis: What it is, what's been done, how to do it
Jansen, B. J. (2006) · Library & Information Science Research · doi:10.1016/j.lisr.2006.06.005 · Cited by 284
Ingwersen, P., & Järvelin, K. (2005) · Springer eBooks · Cited by 283
Measurement Error and Research Design
Viswanathan, M. (2005) · Sage eBooks · doi:10.4135/9781412984935 · Cited by 278
Meta-analysis of correlations among usability measures
Hornbæk, K., & Law, E. L.-C. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240722 · Cited by 276
The Art of Asking Questions: Studies in Public Opinion, 3
Payne, S. L. B. (1980) · Princeton University Press eBooks · doi:10.1515/9781400858064 · Cited by 275
The experience of flow in computer-mediated and in face-to-face groups
Ghani, J. A., Supnick, R. M., & Rooney, P. S. (1991) · Journal of the Association for Information Systems · Cited by 268
Experimental components for the evaluation of interactive information retrieval systems
Borlund, P. (2000) · Journal of Documentation · doi:10.1108/eum0000000007110 · Cited by 267
Cooper, W. S. (1968) · American Documentation · doi:10.1002/asi.5090190108 · Cited by 257
User interactions with everyday applications as context for just-in-time information access
Budzik, J., & Hammond, K. J. (2000) · Proceedings of the 5th international conference on Intelligent user interfaces · doi:10.1145/325737.325776 · Cited by 252
The influence of task and gender on search and evaluation behavior using Google
Lorigo, L., Pan, B., Hembrooke, H., Joachims, T., Granka, L., & Gay, G. (2006) · Information Processing & Management · doi:10.1016/j.ipm.2005.10.001 · Cited by 251
Using terminological feedback for web search refinement: a log-based study
Anick, P. (2003) · Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval · doi:10.1145/860435.860453 · Cited by 249
Information and information sources in tasks of varying complexity
Byström, K. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10064 · Cited by 242
Using graded relevance assessments in IR evaluation
Kekäläinen, J., & Järvelin, K. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10137 · Cited by 242
The development of a method for the evaluation of interactive information retrieval systems
Borlund, P., & Ingwersen, P. (1997) · Journal of Documentation · doi:10.1108/eum0000000007198 · Cited by 240
A study of information seeking and retrieving. I. Background and methodology
Saracevic, T., Kantor, P., Chamis, A. Y., & Trivison, D. (1988) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198805)39:3<161::aid-asi2>3.0.co;2-0 · Cited by 239
Given a context by any other name: methodological tools for taming the unruly beast
Dervin, B. L. (1997) · Cited by 238
Research ethics in Internet-enabled research: Human subjects issues and methodological myopia
Walther, J. B. (2002) · Ethics and Information Technology · doi:10.1023/a:1021368426115 · Cited by 237
Changes in relevance criteria and problem stages in task performance
Vakkari, P., & Hakala, N. (2000) · Journal of Documentation · doi:10.1108/eum0000000007127 · Cited by 237
Saracevic, T. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20682 · Cited by 227
Evaluation of evaluation in information retrieval
Saracevic, T. (1995) · Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '95 · doi:10.1145/215206.215351 · Cited by 225
The pragmatics of information retrieval experimentation, revisited
Tague-Sutcliffe, J. (1992) · Information Processing & Management · doi:10.1016/0306-4573(92)90005-k · Cited by 224
On selecting a measure of retrieval effectiveness
Cooper, W. S. (1973) · Journal of the American Society for Information Science · doi:10.1002/asi.4630240204 · Cited by 224
A study of information seeking and retrieving. III. Searchers, searches, and overlap
Saracevic, T., & Kantor, P. (1988) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198805)39:3<197::aid-asi4>3.0.co;2-a · Cited by 222
Conceptual framework for tasks in information studies
Byström, K., & Hansen, P. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20197 · Cited by 220
The role of individual differences in Internet searching: An empirical study
Ford, N., Miller, D., & Moss, N. (2001) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.1165 · Cited by 219
Cognitive and task influences on Web searching behavior
Kim, K., & Allen, B. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10014 · Cited by 211
Helping people find what they don't know
Belkin, N. J. (2000) · Communications of the ACM · doi:10.1145/345124.345143 · Cited by 211
Norvaišaitė, M. (2006) · Information Research · Cited by 208
User analysis in HCI — the historical lessons from individual differences research
Dillon, A., & Watson, C. (1996) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1996.0071 · Cited by 205
Saracevic, T. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20681 · Cited by 205
How users assess Web pages for information seeking
Tombros, A., Ruthven, I., & Jose, J. M. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20106 · Cited by 202
Studying the use of popular destinations to enhance web search interaction
White, R. W., Bilenko, M., & Cucerzan, S. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277771 · Cited by 197
The history and development of transaction log analysis
Peters, T. A. (1993) · Library Hi Tech · doi:10.1108/eb047884 · Cited by 193
End-User Privacy in Human-Computer Interaction
Iachello, G., & Hong, J. (2007) · Foundations and Trends® in Human-Computer Interaction · doi:10.1561/1100000004 · Cited by 193
End-User Privacy in Human-Computer Interaction
Iachello, G., & Hong, J. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601980779 · Cited by 192
Borgman, C. L. (1989) · Information Processing & Management · doi:10.1016/0306-4573(89)90042-3 · Cited by 190
Eye tracking and online search: Lessons learned and challenges ahead
Lorigo, L., Haridasan, M., Brynjarsdóttir, H., Xia, L., Joachims, T., Gay, G., Granka, L., Pellacini, F., & Pan, B. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20794 · Cited by 188
Relevance assessment: are judges exchangeable and does it matter
Bailey, P., Craswell, N., Soboroff, I., Thomas, P., de Vries, A. P., & Yilmaz, E. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390447 · Cited by 180
Information‐seeking and mediated searching. Part 1. Theoretical framework and research design
Spink, A., Wilson, T. D., Ford, N., Foster, A., & Ellis, D. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10081 · Cited by 180
Conceptual framework for tasks in information studies
ByströmKatriina, & Hansen, P. (2005) · Journal of the American Society for Information Science and Technology · doi:10.5555/1072003.1072010 · Cited by 178
Qualitative methods in information retrieval research.
Fidel, R. (1993) · Library & Information Science Research · Cited by 170
Online searching: Measures that discriminate among users with different types of experiences
Fenichel, C. H. (1981) · Journal of the American Society for Information Science · doi:10.1002/asi.4630320104 · Cited by 168
Liberal relevance criteria of TREC -: counting on negligible documents?
Sormunen, E. (2002) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/564376.564433 · Cited by 164
Multitasking information seeking and searching processes
Spink, A., Ozmutlu, H. C., & Ozmutlu, S. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10124 · Cited by 163
Ethical Dilemmas in Research on Internet Communities
Flicker, S., Haans, D., & Skinner, H. A. (2004) · Qualitative Health Research · doi:10.1177/1049732303259842 · Cited by 163
Why batch and user evaluations do not give the same results
Turpin, A. H., & Hersh, W. (2001) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/383952.383992 · Cited by 161
Bates, M. J. (1990) · Taylor Graham Publishing eBooks · Cited by 160
Flow experiences in information technology use
Pilke, E. M. (2004) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2004.01.004 · Cited by 159
The Stratified Model of Information Retrieval Interaction: Extension and Applications
Saračević, T. (1997) · Proceedings of the American Society for Information Science and Technology · Cited by 157
Reasoning With Statistics: How To Read Quantitative Research
Williams, F., & Monge, P. R. (1986) · Medical Entomology and Zoology · Cited by 155
formative Evaluation for Educational Technologies
Flagg, B. N., & Flagg, B. N. (2013) · Routledge eBooks · doi:10.4324/9780203052181 · Cited by 154
On the evaluation of IR systems
Robertson, S. E., & Hancock‐Beaulieu, M. (1992) · Information Processing & Management · doi:10.1016/0306-4573(92)90004-j · Cited by 154
Debowski, S., Wood, R. E., & Bandura, A. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.6.1129 · Cited by 151
Information seeking and mediated searching. Part 4. Cognitive styles in information seeking
Ford, N. J., Wilson, T. D., Foster, A. E., Ellis, D. J., & Spink, A. H. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10084 · Cited by 148
Query length in interactive information retrieval
Belkin, N. J., Kelly, D., Kim, G., Kim, J.-Y., Lee, H.-J., Mureşan, G., Tang, M., Yuan, X., & Cool, C. (2003) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/860435.860474 · Cited by 148
Self-citationMining the search trails of surfing crowds: identifying relevant websites from user activity
Bilenko, M., & White, R. W. (2008) · International World Wide Web Conference (WWW) · doi:10.1145/1367497.1367505 · Cited by 147
A field study characterizing Web‐based information‐seeking tasks
Kellar, M., Watters, C. R., & Shepherd, M. A. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20590 · Cited by 145
Handbook of Tables for Probability and Statistics
F., W. T., & Beyer, W. H. (1967) · Biometrics · doi:10.2307/2528023 · Cited by 140
Evaluation of information retrieval systems : Approaches, issues, and methods
Harter, S. P., & Hert, C. A. (1997) · Cited by 133
Robertson, S. E. (2006) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1183614.1183630 · Cited by 132
Do batch and user evaluations give the same results?
Hersh, W., Turpin, A. H., Price, S., Chan, B. K. S., Kramer, D., Sacherek, L., & Olson, D. (2000) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/345508.345539 · Cited by 132
Evaluating implicit feedback models using searcher simulations
White, R. W., Ruthven, I. G., Jose, J. M., & van Rijsbergen, C. J. (2005) · ACM Transactions on Information Systems · doi:10.1145/1080343.1080347 · Cited by 131
Cookies and Web browser design
Millett, L. I., Friedman, B., & Felten, E. W. (2001) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365034 · Cited by 131
The state of retrieval system evaluation
Salton, G. (1992) · Information Processing & Management · doi:10.1016/0306-4573(92)90002-h · Cited by 129
The validity of the stimulated retrospective think-aloud method as measured by eye tracking
Guan, Z., Lee, S., Cuddihy, E., & Ramey, J. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124961 · Cited by 123
Collaborative information seeking and retrieval
Foster, J. (2006) · Annual Review of Information Science and Technology · doi:10.1002/aris.1440400115 · Cited by 123
Evaluation measures for interactive information retrieval
Su, L. T. (1992) · Information Processing & Management · doi:10.1016/0306-4573(92)90007-m · Cited by 122
Towards task-based personal information management evaluations
Elsweiler, D., & Ruthven, I. G. (2007) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1277741.1277748 · Cited by 121
User 4XXXXX9: Anonymizing Query Logs
Adar, E. (2007) · Cited by 120
Vakkari, P., Savolainen, R., & Dervin, B. L. (1997) · Cited by 118
Discovering information in context
Solomon, P. J. (2002) · Annual Review of Information Science and Technology · doi:10.1002/aris.1440360106 · Cited by 118
On the history of evaluation in IR
Robertson, S. E. (2008) · Journal of Information Science · doi:10.1177/0165551507086989 · Cited by 115
Affective feedback: an investigation into the role of emotions in the information seeking process
Arapakis, I., Jose, J. M., & Gray, P. (2008) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1390334.1390403 · Cited by 114
Ford, N. J., Miller, D., & Moss, N. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20168 · Cited by 114
Discounted Cumulated Gain based Evaluation of Multiple-Query IR Sessions
Järvelin, K., Price, S., Delcambre, L., & Lykke Nielsen, M. (2008) · Tampere University Institutional Repository (Tampere University) · Cited by 113
Searcher’s Assessments of Task Complexity for Web Searching
Bell, D. J., & Ruthven, I. G. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24752-4_5 · Cited by 111
Mackay, W. E. (1995) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/223904.223922 · Cited by 111
What Makes a Good Answer? The Role of Context in Question Answering.
Lin, J., Quan, D., Sinha, V., Bakshi, K., Huynh, D. F., Katz, B., & Karger, D. R. (2003) · Cited by 102
The Concept of Situation in Information Science.
Cool, C. (2001) · Cited by 102
What is special about the ethical issues in online research?
Elgesem, D. (2002) · Ethics and Information Technology · doi:10.1023/a:1021320510186 · Cited by 102
Belkin, N. J., & Vickery, A. (1985) · Medical Entomology and Zoology · Cited by 101
Measures of relative relevance and ranked half-life
Borlund, P., & Ingwersen, P. (1998) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/290941.291019 · Cited by 96
Evaluating hypermedia and learning: methods and results from the Perseus Project
Marchionini, G., & Crane, G. (1994) · ACM Transactions on Information Systems · doi:10.1145/174608.174609 · Cited by 95
User adaptation: good results from poor systems
Smith, C. L., & Kantor, P. B. (2008) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1390334.1390362 · Cited by 93
Interactive information retrieval
Ruthven, I. G. (2008) · Annual Review of Information Science and Technology · doi:10.1002/aris.2008.1440420109 · Cited by 88
Internet Research: An Opportunity to Revisit Classic Ethical Problems in Behavioral Research
Pittenger, D. J. (2003) · Ethics & Behavior · doi:10.1207/s15327019eb1301_08 · Cited by 86
Lagergren, E. S., & Over, P. (1998) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/290941.290986 · Cited by 84
Interfaces for staying in the flow
Bederson, B. B. (2004) · doi:10.1145/1074068.1074069 · Cited by 83
Query logs alone are not enough
Grimes, C., Tang, D., & Russell, D. M. (2007) · Cited by 83
Evaluation by comparing result sets in context
Thomas, P., & Hawking, D. (2006) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1183614.1183632 · Cited by 83
Eisenberg, M. B. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90042-8 · Cited by 79
Callan, J., Allan, J., Clarke, C. L. A., Dumais, S., Evans, D. A., Sanderson, M., & Zhai, C. (2007) · ACM SIGIR Forum · doi:10.1145/1328964.1328967 · Cited by 78
Improving Web Search Ranking by Incorporating User Behavior
Agichtein, E., Brill, E., & Dumais, S. (2006) · Cited by 76
Towards the identification of the optimal number of relevance categories
Tang, R., William M. Shaw, & Vevea, J. L. (1999) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(1999)50:3<254::aid-asi8>3.0.co;2-y · Cited by 75
Regions and levels: Measuring and mapping users' relevance judgments
Spink, A. H., & Greisdorf, H. (2001) · Journal of the American Society for Information Science and Technology · doi:10.1002/1097-4571(2000)9999:9999<::aid-asi1564>3.0.co;2-l · Cited by 75
Affective issues in library and information science systems work: A content analysis
Julien, H., McKechnie, L. E. F., & Hart, S. (2005) · Library & Information Science Research · doi:10.1016/j.lisr.2005.08.004 · Cited by 73
Relationships between categories of relevance criteria and stage in task completion
Taylor, A. R., Cool, C., Belkin, N. J., & Amadio, W. J. (2006) · Information Processing & Management · doi:10.1016/j.ipm.2006.09.008 · Cited by 72
Feedback in Information Retrieval.
Spink, A. H., & Losee, R. M. (1996) · Cited by 70
Exploring How Mouse Movements Relate to Eye Movements on Web Search Results Pages
Rodden, K., & Fu, X. (2007) · Cited by 70
Online searching styles: A case‐study‐based model of searching behavior
Fidel, R. (1984) · Journal of the American Society for Information Science · doi:10.1002/asi.4630350404 · Cited by 69
Information needs: a person-in-situation approach
Allen, B. (1997) · Cited by 68
Selecting and measuring task characteristics as independent variables
Kim, S., & Soergel, D. (2005) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14504201111 · Cited by 68
Subjective Duration Assessment: An Implicit Probe for Software Usability
Czerwinski, M. P., Horvitz, E., & Cutrell, E. B. (2001) · Cited by 66
Elicitation of term relevance feedback
Kelly, D., & Fu, X. (2006) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1148170.1148249 · Cited by 65
Self-citationWeb searching: A process‐oriented experimental study of three interactive search paradigms
Dennis, S. J., Bruza, P., & McArthur, R. (2001) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10015 · Cited by 63
Interaction in information searching and retrieval
Beaulieu, M. (2000) · Journal of Documentation · doi:10.1108/eum0000000007122 · Cited by 63
Vakkari, P. (2001) · Information Retrieval · doi:10.1023/a:1016089224008 · Cited by 62
Research works printed but not linked: 133
- Adar, 2007, Proceedings of the Workshop on Query Log Analysis: Social and Technological Challenges, at the 16th International World Wide Web Conference
- AgichteinBrillDumais, 2006, Proceedings of the 29th Annual ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’06), 3
- Allan, 2004, TREC2003, Proceedings of the 12th Text Retrieval Conference
- Allan, 2006, TREC2005, Proceedings of the l4th Text Retrieval Conference
- Allen, 1997, Information Seeking in Context: Proceedings of an International Conference on Research in Information Needs, Seeking and Use in Different Contexts, 111
- Amdur, 2003, Institutional Review Board Member Handbook
- American Psychological Association, 2001, Publication Manual of the American Psychological Association
- AmitayMurrayTeevan, 2007, Proceedings of the 16th International World Wide Web Conference
- Anick, 2003, Proceedings of the 26th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ‘03), 88
- ArapakisJose, 2008, Proceedings of the 31st Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ‘08), 395
- BaileyCraswellSoboroffThomasde VriesYilmaz, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’08), 667
- BeaulieuGatfordHuangRobertsonWalkerWilliams, 1997, Proceedings of the 5th Text Retrieval Conference (TREC-5), 143
- Belkin, 1980, “Anomalous states of knowledge as a basis for information retrieval”, Canadian Journal of Information Science, 5, 133
- BelkinCabezasCoolKimNgParkPressmanRiehSavageXie, 1997, Proceedings of the 5th Text Retrieval Conference (TREC-5), 257
- BelkinKellyKimKimLeeMuresanTangYuanCool, 2003, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 205
- BellRuthven, 2004, Proceedings of the 26th Annual International European Conference on Information Retrieval (ECIR 200fi), 57
- Bennett, 1972, “The user interface in interactive systems”, Annual Review of Information Science and Technology, 7, 159
- Bernal, 1948, “Preliminary analysis of pilot questionnaires on the use of scientific literature”, The Royal Society Scientific Information Conference, 589
- BilenkoWhite, 2008, Proceedings of the 17th International Conference on the World Wide Web (WWW ’08), 51
- Borgman, 1983, Proceedings of the 6th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’83), 162
- BorlundIngwersen, 1998, Proceedings of the 21st ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR ’98), 324
- BorlundRuthven, 2008, “Introduction to the special issue on evaluating interactive information retrieval systems”, Information Processing and Management, 44, 1
- BorlundSchneiderLaimasTombrosFeatherKellyde VriesAzzopardi, 2008, Proceedings of the 2nd International Symposium on Information Interaction in Context
- BoyceMeadowKraft, 1994, Measurement in Information Science
- BudzikHammond, 2000, Proceedings of the 5th International Conference on Intelligent User Interfaces (IUI TO), 44
- Capra, 2008, Personal Information Management, CHI 2008 Workshop
- Charmaz, 2002, Handbook of Interview Research: Context and Method
- Cleverdon, 1997, Readings in Information Retrieval, 173
- CowleyHaackLittlefieldHampson, 2006, Proceedings of the 2nd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE ’05), 13
- CzerwinskiHorvitzCutrell, 2001, Proceedings of IHM-HCI 2001 Conference, 167
- CzerwinskiHorvitzWilhite, 2004, Proceedings of ACM Human Factors in Computing Systems Conference (CHI ’Of), 175
- DahlbäckJonssonAhrenberg, 1993, Proceedings for the 1st International Conference on Intelligent User Interfaces (IUI ’93), 193
- DangKellyLin, 2008, TREC2007, Proceedings of the 16th Text Retrieval Conference
- De Mey, 1977, CC77: International Workshop on the Cognitive Viewpoint, xvi
- Dervin, 1992, Qualitative Research in Information Management, 61
- Dervin, 1996, Information Seeking in Context: Proceedings of an International Conference on Research in Information Needs, Seeking and Use in Different Contexts, 13
- DominickPenniman, 1979, Proceedings of the 2nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’79), 2
- DumaisCutrellCadizJanckeSarinRobbins, 2003, Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR ’03), 72
- Dunlop, 1997, Proceedings of the 20th ACM Conference on Research and Development in Information Retrieval (SIGIR ’97), 206
- ElsweilerRuthven, 2007, Proceedings of the 30th ACM Conference on Research and Development in Information Retrieval (SIGIR ’03), 22
- FenstermacherGinsburg, 2003, “Client-side monitoring for Web mining”, Journal of the American Society for Information Science and Technoloqy, 54, 625, 10.1002/asi.10255 · doi:10.1002/asi.10255
- Fiske, 1982, Forms of Validity in Research, 77
- FittsJonesMilton, 1950, “Eye movements of aircraft pilots during instrument-landing approaches”, Aeronautical Engineering Review, 9, 24
- GhaniSupnickRooney, 1991, Proceedings of International Conference on Information Systems (JCIS 1991), 229
- GrimesTangRussell, 2007, Proceedings of the Workshop on Query Log Analysis: Social and Technology Challenges at the 16th International World Wide Web Conference
- GuanLeeCuddihyRamey, 2006, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 1253
- HaradaNaamanSongWangPaepcke, 2004, Proceedings of the 4th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL T4), 325
- Harman, 2005, TREC: Experiment and Evaluation in Information Retrieval, 21
- HartStaveland, 1988, Human Mental Workload, 139
- HawkingBaileyCraswell, 2000, “Efficient and flexible search using text and metadata”, CSIRO Mathematical and Information Sciences Tech Report, 2000-83
- Hersh, 2006, Advances in Open Domain Question Answering, 431
- HershOver, 2001, “Introduction to a special issue on interactivity at the Text Retrieval Conference (TREC)”, Information Processing and Management, 37, 365, 10.1016/S0306-4573(00)00052-2 · doi:10.1016/s0306-4573(00)00052-2
- HershTurpinPriceChanKramerSacherekOlson, 2000, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR TO), 17
- HershElliotHickamWolfMolnarLeichtenstein, 1995, Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’95), 164
- HirshDinkelacker, 2004, “Seeking information in order to produce information: An empirical study at Hewlett Packard Labs”, Journal of the American Society for Information Science and Technology, 55, 807, 10.1002/asi.20024 · doi:10.1002/asi.20024
- HuangWang, 2004, “The influence of document presentation order and number of documents judged on users’ judgments of relevance”, Journal of the American Society for Information Science, 55, 970, 10.1002/asi.20047 · doi:10.1002/asi.20047
- IngwersenWillett, 1995, “An introduction to algorithmic and cognitive approaches for information retrieval”, Libri, 45, 160
- International Standards Office, 1998, Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs): Part II, Guidance on Usability
- JacobKarn, 2003, The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, 573
- JansenRamadossZhangZang, 2006, Workshop on Evaluating Exploratory Search Systems at the 29th Annual International ACM SIGIR Conference on Research and Development on Information Retrieval (SIGIR ‘06)
- JärvelinKekäläinen, 2000, Proceedings of the 23rd ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR ’00), 41
- JärvelinPriceDelcambreNielsen, 2008, Proceedings of the 30th European Conference on Information Retrieval
- Jex, 1988, Human Mental Workload, 5
- JoachimsGrankaPanHembrookeGay, 2005, Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’05), 154
- JohoHannahJose, 2008, Proceedings of the 2nd IHX Symposium on Information Interaction in Context, 89
- JonesTeevan, 2007, Personal Information Management
- KäkiAula, 2008, “Controlling the complexity in comparing search user interfaces via user studies”, Information Processing and Management, 44, 82, 10.1016/j.ipm.2007.02.006 · doi:10.1016/j.ipm.2007.02.006
- KantorKazaiMilic-FraylingWilkinson, 2008, Proceedings of the Conference on Information and Knowledge Management (CIKM ’08)
- Kantor, 1982, “Evaluation of and feedback in information storage and retrieval systems”, Annual Review of Information Science and Technology, 17, 99
- KellyFu, 2006, Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’06), 453
- KellyKantorMorseScholtzSun, 2006, Proceedings of the Workshop on Interactive Question Answering at the Human Language Technology Conference (HLT-NAACL ’06)
- KellyLin, 2007, “Overview of the TREC 2006 ciQA task”, SIGIR Forum, 41, 107, 10.1145/1273221.1273231 · doi:10.1145/1273221.1273231
- KellyShahSugimotoBaileyClemensIrvineJohnsonKeOhPoljakovaRodriguezvan NoordZhang, 2008, Proceedings of the 2nd Symposium on Information Interaction in Context (IIiX), 75
- KellyWacholderRittmanSunKantorSmallStrzalkowski, 2007, “Using interview data to identify evaluation criteria for interactive, analytical question answering systems”, Journal of the American Society for Information Science and Technology, 58, 1032, 10.1002/asi.20575 · doi:10.1002/asi.20575
- King, 1968, “Design and evaluation of information systems”, Annual Review of Information Science and Technology, 3, 61
- LarsenMalikTombros, 2006, INEX 2005, 398
- Likert, 1932, “A technique for the measurement of attitudes”, Archives of Psychology, 140, 1
- LinQuanSinhaBakshiHuynhKatzKarger, 2003, Proceedings of the 9th IFIP TCI3 International Conference on Human-Computer Interaction (INTERACT 2003)
- LinSmucker, 2008, Proceedings of the 31st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 08), 19
- LinBelkin, 2000, Proceedings of the Annual Conference of the American Society for Information Science (ASIS TO), 133
- Losee, 1996, “Evaluating retrieval performance given database and query characteristics: Analytical determination of performance surfaces”, Journal of the American Society for Information Science, 47, 95, 10.1002/(SICI)1097-4571(199601)47:1<95::AID-ASI9>3.0.CO;2-Y · doi:10.1002/(sici)1097-4571(199601)47:1<95::aid-asi9>3.0.co;2-y
- Luhn, 1958, H. P. Luhn: Pioneer of Infor-mation Science. Selected Works, 132
- Lund, 2001, “Measuring usability with the USE Questionnaire”, Usability and User Experience, 8
- Mackay, 1995, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 138
- Marchionini, 2008, “Toward human-Computer information retrieval”, Bulletin of the American Society for Information Science
- Michel, 1994, “What is used during cognitive processing in information retrieval and library searching? Eleven sources of search information”, Journal of the American Society for Information Science, 45, 498, 10.1002/(SICI)1097-4571(199408)45:7<498::AID-ASI5>3.0.CO;2-T · doi:10.1002/(sici)1097-4571(199408)45:7<498::aid-asi5>3.0.co;2-t
- MillettFriedmanFelten, 2001, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 46
- Morahan-Martin, 1998, Psychology and the Internet: Intrapersonal, Interpersonal, and Transpersonal Implications, 169
- MorrisHorvitz, 2007, Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST ’07), 3
- MyersWell, 2003, Research Design and Statistical Analysis
- Nielsen, 2008, “Usability 101: Introduction to usability”, Jakob Nielsen’s Alertbox
- Oard, 2000, Proceedings of the 1st Cross-Language Evaluation Forum
- O’BrienTomsKellowayKelley, 2008, Proceedings of the American Society for Information Science and Technology
- PetersKurthFlahertySandoreKaske, 1993, “An introduction to the special section on transaction log analysis”, Library Hi Tech, 11, 38, 10.1108/eb047883 · doi:10.1108/eb047883
- Petrelli, 2008, “On the role of user-centred evaluation in the advancement of interactive information retrieval”, Information Processing and Management, 44, 22, 10.1016/j.ipm.2007.01.024 · doi:10.1016/j.ipm.2007.01.024
- Robertson, 2006, Proceedings of the 15th ACM international Conference on information and Knowledge Management (CIKM’06), 78
- RoddenFu, 2007, Proceedings of the Web Information-Seeking and Interaction ( WIST) Workshop at the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’Of), 29
- Rorvig, 1988, “Psychometric measurement and information retrieval”, Annual Review of Information Science and Technology, 23, 157
- Ruthven, 2006, Proceedings of the 1st International Conference on Information Interaction in Context, 10.1145/1164820 · doi:10.1145/1164820
- Saracevic, 1967, Proceedings of the Annual Meeting of the American Documentation Institute, 4, 100
- Saracevic, 1995, Proceedings of the 18th Annual ACM SIGIR Conference on Research and Development of Information Retrieval, 138
- SaracevicKantor, 1988, “A study of information seeking and retrieving. II. Users, questions and effectiveness”, Journal of the American Society for Information Science, 39, 177, 10.1002/(SICI)1097-4571(198805)39:3<177::AID-ASI3>3.0.CO;2-F · doi:10.1002/(sici)1097-4571(198805)39:3<177::aid-asi3>3.0.co;2-f
- Savage-KnepshieldBelkin, 1999, “Interaction in information retrieval: Trends over time”, Journal of the American Society for Information Science, 50, 1067, 10.1002/(SICI)1097-4571(1999)50:12<1067::AID-ASI6>3.0.CO;2-4 · doi:10.1002/(sici)1097-4571(1999)50:12<1067::aid-asi6>3.0.co;2-4
- Siatri, 1999, “The evolution of user studies”, Libri, 49, 132, 10.1515/libr.1999.49.3.132 · doi:10.1515/libr.1999.49.3.132
- SmithKantor, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’08), 147
- Sormunen, 2002, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’02), 324
- Spärck-Jones, 1981, Information Retrieval Experiment
- Spink, 1996, “Multiple search session model of end-user behavior: An exploratory study”, Journal of the American Society for Information Science, 47, 603, 10.1002/(SICI)1097-4571(199608)47:8<603::AID-ASI4>3.0.CO;2-X · doi:10.1002/(sici)1097-4571(199608)47:8<603::aid-asi4>3.0.co;2-x
- Spink, 1997, “Study of interactive feedback during mediated information retrieval”, Journal of the American Society for Information Science, 48, 382, 10.1002/(SICI)1097-4571(199705)48:5<382::AID-ASI2>3.0.CO;2-R · doi:10.1002/(sici)1097-4571(199705)48:5<382::aid-asi2>3.0.co;2-r
- Stern, 2003, “Encountering distressing information in online research: A consideration of legal and ethical responsibilities”, New Media and Society, 5, 249, 10.1177/1461444803005002006 · doi:10.1177/1461444803005002006
- Sugar, 1995, “User-centered perspectives of information retrieval research and analysis methods”, Annual Review of Information Science and Technology, 30, 77
- SunKantor, 2006, “Cross-evaluation: A new model for information system evaluation”, Journal of American Society for Information Science and Technology, 56, 614
- Tague, 1987, “Informativeness as an ordinal utility function for information retrieval”, SIGIR Forum, 21, 10, 10.1145/30075.30077 · doi:10.1145/30075.30077
- Tague, 1981, Information Retrieval Experiment, 59
- Tague-Sutcliffe, 1992, Proceedings of the 15th Annual AGM International Conference on Research and Development in Information Retrieval (SIGIR ’92), 23
- TeevanAlvaradoAckermanKarger, 2004, Proceedings of the AGM SIGCHI Conference on Human Factors in Computing Systems (SIGCHI ’04), 415
- TeevanDumaisHorvitz, 2005, Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’05), 449
- ThomasHawking, 2006, Proceedings of the 15th Annual Conference on Information and Knowledge Management (CIKM ’06), 94
- TurpinHersh, 2001, Proceedings of the 24th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’01), 225
- TurpinScholer, 2006, Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’06), 11
- TurtlePennimanHickey, 1981, “Data entry/display devices for interactive information retrieval”, Annual Review of Information Science and Technology, 16, 55
- Urquhart, 1948, “The distribution and use of scientific and technical information”, The Royal Society Scientific Information Conference, 408
- VakkariJärvelin, 2005, New Directions in Cognitive Information Retrieval, 113
- VeerasamyBelkin, 1996, Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 85
- VeerasamyHeikes, 1997, “Effectiveness of a graphical display of retrieval results”, SIGIR Forum, 31, 236, 10.1145/278459.258580 · doi:10.1145/278459.258580
- VoorheesHarman, 2005, TREC: Experiment and Evaluation in Information Retrieval
- Wang, 1999, “Methodologies and methods for user behavior research”, Annual Review of Information Science and Technology, 34, 53
- WenRuthvenBorlund, 2006, Proceedings of the 28th European Conference in Information Retrieval (ECIR 2006’)
- WhiteBilenkoCucerzan, 2007, Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’07), 159
- XiongAgichtein, 2007, Proceedings of the Workshop on Query Log Analysis: Social and Technology Challenges at the 16th International World Wide Web Conference
- YuanMeadow, 1999, “A study of the use of variables in information retrieval user studies”, Journal of the American Society for Information Science, 50, 140, 10.1002/(SICI)1097-4571(1999)50:2<140::AID-ASI5>3.0.CO;2-P · doi:10.1002/(sici)1097-4571(1999)50:2<140::aid-asi5>3.0.co;2-p
- ZhangPlettenbergKlavansOardSoergel, 2007, Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries (JCDL ’07), 117
- Zhang, 2002, “Collaborative relevance judgments: A group consensus method for evaluating user search performance”, Journal of the American Society for Information Science and Technology, 53, 220, 10.1002/asi.10036 · doi:10.1002/asi.10036
Cited by
Crowdsourcing for relevance evaluation
Alonso, O., Rose, D. E., & Stewart, B. (2008) · ACM SIGIR Forum · doi:10.1145/1480506.1480508 · Cited by 329
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317
Mehrotra, R., McInerney, J., Bouchard, H., Lalmas, M., & Diaz, F. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3272027 · Cited by 279
Lalmas, M., O’Brien, H., & Yom-Tov, E. (2015) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02289-0 · Cited by 192
Kelly, D., Arguello, J., Edwards, A., & Wu, W.-C. (2015) · Proceedings of the 2015 International Conference on The Theory of Information Retrieval · doi:10.1145/2808194.2809465 · Cited by 155
Self-citation
Metadata
- Authors
- Kelly, D.
- Venue
- Foundations and Trends® in Information Retrieval
- Publisher
- Emerald
- Published
- 2009-05-27
- Volume
- 3
- Issue
- 1-2
- Pages
- 1-224
- ISSN
- 1554-0669 (ISSN-L)
- 1554-0677
- OpenAlex
- W2035569891
- Cited by
- 633
- References
- 241
- FWCI
- 60.91
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.