The DOI matches its Crossref record; 241 of the 241 references it cites were found in a registry or an index.
Methods for Evaluating Interactive Information Retrieval Systems with Users
Abstract
This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users. The primary goal of this article is to catalog and compile material related to this topic into a single source. This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies. This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR). Finally, this article concludes with a discussion of outstanding challenges and future research directions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–241 of 241
Some perspectives on the evaluation of information retrieval systems
Tague‐Sutcliffe, J. (1996) · Journal of the American Society for Information Science · Cited by 61
Questionnaire mode effects in interactive information retrieval experiments
Kelly, D., Harper, D. J., & Landau, B. J. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.02.007 · Cited by 60
Self-citationHarada, S., Naaman, M., Song, Y. J., Wang, Q., & Paepcke, A. (2004) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/996350.996425 · Cited by 60
The influence of relevance levels on the effectiveness of interactive information retrieval
Vakkari, P., & Sormunen, E. (2004) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20046 · Cited by 59
Harman, D. K. (2005) · Cited by 58
Written versus spoken queries: A qualitative and quantitative comparative analysis
Crestani, F., & Du, H. (2006) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20350 · Cited by 58
Evaluation of a mobile information system in context
Göker, A., & Myrhaug, H. I. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.03.011 · Cited by 57
Time, relevance and interaction modelling for information retrieval
Dunlop, M. D. (1997) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/258525.258569 · Cited by 54
Measuring information : an information services perspective
Tague‐Sutcliffe, J. (1995) · Cited by 53
Task Effects on Interactive Search: The Query Factor
Toms, E. G., O'Brien, H. L., Mackenzie, T., et al. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85902-4_31 · Cited by 53
Lopatovska, I., & Mokros, H. B. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.01.020 · Cited by 51
Evaluation issues in information retrieval
Harman, D. K. (1992) · Information Processing & Management · doi:10.1016/0306-4573(92)90001-g · Cited by 44
BLANDFORD, A. E., Adams, A., Attfield, S., Buchanan, G. R., Gow, J., Makri, S., Rimmer, J., & Warwick, C. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.01.021 · Cited by 44
Measuring the informativeness of a retrieval process
Tague‐Sutcliffe, J. (1992) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/133160.133171 · Cited by 40
The Vocabulary Problem in Human-System Communication: an Analysis and a Solution
Furnas, G. W., Landauer, T. K., Gomez, L. M., & Dumais, S. (1987) · Cited by 38
Assessing learning outcomes in two information retrieval learning environments
Halttunen, K., & Järvelin, K. (2004) · Information Processing & Management · doi:10.1016/j.ipm.2004.02.004 · Cited by 37
WiIRE: the Web interactive information retrieval experimentation system prototype
Toms, E. G., Freund, L., & Li, C. (2003) · Information Processing & Management · doi:10.1016/j.ipm.2003.08.006 · Cited by 37
How do users find things with PubMed?
Lin, J., & Smucker, M. D. (2008) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1390334.1390340 · Cited by 36
Evaluation problems in interactive information retrieval
Salton, G. (1970) · Information Processing & Management · doi:10.1016/0020-0271(70)90011-2 · Cited by 35
The relative effects of knowledge, interest and confidence in assessing relevance
Ruthven, I. G., Baillie, M., & Elsweiler, D. (2007) · Journal of Documentation · doi:10.1108/00220410710758986 · Cited by 35
Experimental and quasi-experimental designs for research in information science
Haas, D. F., & Kraft, D. H. (1984) · Information Processing & Management · doi:10.1016/0306-4573(84)90053-0 · Cited by 34
Mind the Gap: In Praise of Informal Sources of Formal Theory
Fiske, S. T. (2004) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0802_6 · Cited by 34
Mihaly Csikszentmihalyi, Finding Flow. The Psychology of Engagement with Everyday Life
Vittersø, J. (2000) · Journal of Happiness Studies · doi:10.1023/a:1010032312178 · Cited by 32
Task-based interaction with an integrated multilingual, multimedia information system
Zhang, P., Plettenberg, L., Klavans, J. L., Oard, D. W., & Soergel, D. (2007) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/1255175.1255199 · Cited by 32
Revisiting search task difficulty: Behavioral and individual difference measures
Jacek Gwizdka, A. (2008) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.2008.1450450249 · Cited by 31
Bulmer, M. (2004) · SAGE Publications Ltd eBooks · doi:10.4135/9781446263204 · Cited by 31
Developing and evaluating a reliable measure of user engagement
O'Brien, H. L., Toms, E. G., Kelloway, E. K., & Kelley, E. (2008) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.2008.1450450258 · Cited by 30
Towards Privacy-Preserving Query Log Publishing
Xiong, L., & Agichtein, E. (2007) · Cited by 28
On test collections for adaptive information retrieval
Voorhees, E. M. (2008) · Information Processing & Management · doi:10.1016/j.ipm.2007.12.011 · Cited by 27
Task as a predictable indicator for information seeking behavior on the Web
Kim, J. (2007) · Medical Entomology and Zoology · Cited by 25
Effectiveness of additional representations for the search result presentation on the web
Joho, H., & Jose, J. M. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.02.004 · Cited by 25
Explanation in Information Seeking and Retrieval
Vakkari, P., & Järvelin, K. (2006) · Springer eBooks · doi:10.1007/1-4020-4014-8_7 · Cited by 24
House, E. R. (1983) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 23
Julien, C., Leide, J. E., & Bouthillier, F. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20786 · Cited by 22
Integration Approaches to Relevance
Ruthven, I. G. (2006) · Springer eBooks · doi:10.1007/1-4020-4014-8_4 · Cited by 21
An analysis of two approaches in information retrieval: From frameworks to study designs
Järvelin, K. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20589 · Cited by 19
Exploring the layers of information retrieval evaluation
Dunlop, M. D., Johnson, C. W., & Reid, J. (1998) · Interacting with Computers · doi:10.1016/s0953-5438(98)00014-9 · Cited by 14
Effects of performance feedback on users' evaluations of an interactive IR system
Kelly, D., Shah, C. S., Sugimoto, C. R., Bailey, E. W., Clemens, R. A., Irvine, A., Johnson, N., Ke, W., Oh, S., Poljakova, A., et al. (2008) · Information Interaction in Context Symposium (IIiX) · doi:10.1145/1414694.1414712 · Cited by 12
Self-citationAccess via Features versus Access via Transcripts: User Performance and Satisfaction 1
Wildemuth, B. M., Yang, M., Hughes, A., et al. (2003) · Cited by 9
Automated monitoring to support the analysis and evaluation of information systems
Dominick, W. D., & Penniman, W. D. (1979) · ACM SIGIR Forum · doi:10.1145/1013232.511708 · Cited by 6
(1999) · doi:10.1002/(sici)1097-4571(1999)50:12
Research works printed but not linked: 133
- Adar, 2007, Proceedings of the Workshop on Query Log Analysis: Social and Technological Challenges, at the 16th International World Wide Web Conference
- AgichteinBrillDumais, 2006, Proceedings of the 29th Annual ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’06), 3
- Allan, 2004, TREC2003, Proceedings of the 12th Text Retrieval Conference
- Allan, 2006, TREC2005, Proceedings of the l4th Text Retrieval Conference
- Allen, 1997, Information Seeking in Context: Proceedings of an International Conference on Research in Information Needs, Seeking and Use in Different Contexts, 111
- Amdur, 2003, Institutional Review Board Member Handbook
- American Psychological Association, 2001, Publication Manual of the American Psychological Association
- AmitayMurrayTeevan, 2007, Proceedings of the 16th International World Wide Web Conference
- Anick, 2003, Proceedings of the 26th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ‘03), 88
- ArapakisJose, 2008, Proceedings of the 31st Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ‘08), 395
- BaileyCraswellSoboroffThomasde VriesYilmaz, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’08), 667
- BeaulieuGatfordHuangRobertsonWalkerWilliams, 1997, Proceedings of the 5th Text Retrieval Conference (TREC-5), 143
- Belkin, 1980, “Anomalous states of knowledge as a basis for information retrieval”, Canadian Journal of Information Science, 5, 133
- BelkinCabezasCoolKimNgParkPressmanRiehSavageXie, 1997, Proceedings of the 5th Text Retrieval Conference (TREC-5), 257
- BelkinKellyKimKimLeeMuresanTangYuanCool, 2003, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 205
- BellRuthven, 2004, Proceedings of the 26th Annual International European Conference on Information Retrieval (ECIR 200fi), 57
- Bennett, 1972, “The user interface in interactive systems”, Annual Review of Information Science and Technology, 7, 159
- Bernal, 1948, “Preliminary analysis of pilot questionnaires on the use of scientific literature”, The Royal Society Scientific Information Conference, 589
- BilenkoWhite, 2008, Proceedings of the 17th International Conference on the World Wide Web (WWW ’08), 51
- Borgman, 1983, Proceedings of the 6th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’83), 162
- BorlundIngwersen, 1998, Proceedings of the 21st ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR ’98), 324
- BorlundRuthven, 2008, “Introduction to the special issue on evaluating interactive information retrieval systems”, Information Processing and Management, 44, 1
- BorlundSchneiderLaimasTombrosFeatherKellyde VriesAzzopardi, 2008, Proceedings of the 2nd International Symposium on Information Interaction in Context
- BoyceMeadowKraft, 1994, Measurement in Information Science
- BudzikHammond, 2000, Proceedings of the 5th International Conference on Intelligent User Interfaces (IUI TO), 44
- Capra, 2008, Personal Information Management, CHI 2008 Workshop
- Charmaz, 2002, Handbook of Interview Research: Context and Method
- Cleverdon, 1997, Readings in Information Retrieval, 173
- CowleyHaackLittlefieldHampson, 2006, Proceedings of the 2nd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE ’05), 13
- CzerwinskiHorvitzCutrell, 2001, Proceedings of IHM-HCI 2001 Conference, 167
- CzerwinskiHorvitzWilhite, 2004, Proceedings of ACM Human Factors in Computing Systems Conference (CHI ’Of), 175
- DahlbäckJonssonAhrenberg, 1993, Proceedings for the 1st International Conference on Intelligent User Interfaces (IUI ’93), 193
- DangKellyLin, 2008, TREC2007, Proceedings of the 16th Text Retrieval Conference
- De Mey, 1977, CC77: International Workshop on the Cognitive Viewpoint, xvi
- Dervin, 1992, Qualitative Research in Information Management, 61
- Dervin, 1996, Information Seeking in Context: Proceedings of an International Conference on Research in Information Needs, Seeking and Use in Different Contexts, 13
- DominickPenniman, 1979, Proceedings of the 2nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’79), 2
- DumaisCutrellCadizJanckeSarinRobbins, 2003, Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR ’03), 72
- Dunlop, 1997, Proceedings of the 20th ACM Conference on Research and Development in Information Retrieval (SIGIR ’97), 206
- ElsweilerRuthven, 2007, Proceedings of the 30th ACM Conference on Research and Development in Information Retrieval (SIGIR ’03), 22
- FenstermacherGinsburg, 2003, “Client-side monitoring for Web mining”, Journal of the American Society for Information Science and Technoloqy, 54, 625, 10.1002/asi.10255 · doi:10.1002/asi.10255
- Fiske, 1982, Forms of Validity in Research, 77
- FittsJonesMilton, 1950, “Eye movements of aircraft pilots during instrument-landing approaches”, Aeronautical Engineering Review, 9, 24
- GhaniSupnickRooney, 1991, Proceedings of International Conference on Information Systems (JCIS 1991), 229
- GrimesTangRussell, 2007, Proceedings of the Workshop on Query Log Analysis: Social and Technology Challenges at the 16th International World Wide Web Conference
- GuanLeeCuddihyRamey, 2006, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 1253
- HaradaNaamanSongWangPaepcke, 2004, Proceedings of the 4th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL T4), 325
- Harman, 2005, TREC: Experiment and Evaluation in Information Retrieval, 21
- HartStaveland, 1988, Human Mental Workload, 139
- HawkingBaileyCraswell, 2000, “Efficient and flexible search using text and metadata”, CSIRO Mathematical and Information Sciences Tech Report, 2000-83
- Hersh, 2006, Advances in Open Domain Question Answering, 431
- HershOver, 2001, “Introduction to a special issue on interactivity at the Text Retrieval Conference (TREC)”, Information Processing and Management, 37, 365, 10.1016/S0306-4573(00)00052-2 · doi:10.1016/s0306-4573(00)00052-2
- HershTurpinPriceChanKramerSacherekOlson, 2000, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR TO), 17
- HershElliotHickamWolfMolnarLeichtenstein, 1995, Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’95), 164
- HirshDinkelacker, 2004, “Seeking information in order to produce information: An empirical study at Hewlett Packard Labs”, Journal of the American Society for Information Science and Technology, 55, 807, 10.1002/asi.20024 · doi:10.1002/asi.20024
- HuangWang, 2004, “The influence of document presentation order and number of documents judged on users’ judgments of relevance”, Journal of the American Society for Information Science, 55, 970, 10.1002/asi.20047 · doi:10.1002/asi.20047
- IngwersenWillett, 1995, “An introduction to algorithmic and cognitive approaches for information retrieval”, Libri, 45, 160
- International Standards Office, 1998, Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs): Part II, Guidance on Usability
- JacobKarn, 2003, The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, 573
- JansenRamadossZhangZang, 2006, Workshop on Evaluating Exploratory Search Systems at the 29th Annual International ACM SIGIR Conference on Research and Development on Information Retrieval (SIGIR ‘06)
- JärvelinKekäläinen, 2000, Proceedings of the 23rd ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR ’00), 41
- JärvelinPriceDelcambreNielsen, 2008, Proceedings of the 30th European Conference on Information Retrieval
- Jex, 1988, Human Mental Workload, 5
- JoachimsGrankaPanHembrookeGay, 2005, Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’05), 154
- JohoHannahJose, 2008, Proceedings of the 2nd IHX Symposium on Information Interaction in Context, 89
- JonesTeevan, 2007, Personal Information Management
- KäkiAula, 2008, “Controlling the complexity in comparing search user interfaces via user studies”, Information Processing and Management, 44, 82, 10.1016/j.ipm.2007.02.006 · doi:10.1016/j.ipm.2007.02.006
- KantorKazaiMilic-FraylingWilkinson, 2008, Proceedings of the Conference on Information and Knowledge Management (CIKM ’08)
- Kantor, 1982, “Evaluation of and feedback in information storage and retrieval systems”, Annual Review of Information Science and Technology, 17, 99
- KellyFu, 2006, Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’06), 453
- KellyKantorMorseScholtzSun, 2006, Proceedings of the Workshop on Interactive Question Answering at the Human Language Technology Conference (HLT-NAACL ’06)
- KellyLin, 2007, “Overview of the TREC 2006 ciQA task”, SIGIR Forum, 41, 107, 10.1145/1273221.1273231 · doi:10.1145/1273221.1273231
- KellyShahSugimotoBaileyClemensIrvineJohnsonKeOhPoljakovaRodriguezvan NoordZhang, 2008, Proceedings of the 2nd Symposium on Information Interaction in Context (IIiX), 75
- KellyWacholderRittmanSunKantorSmallStrzalkowski, 2007, “Using interview data to identify evaluation criteria for interactive, analytical question answering systems”, Journal of the American Society for Information Science and Technology, 58, 1032, 10.1002/asi.20575 · doi:10.1002/asi.20575
- King, 1968, “Design and evaluation of information systems”, Annual Review of Information Science and Technology, 3, 61
- LarsenMalikTombros, 2006, INEX 2005, 398
- Likert, 1932, “A technique for the measurement of attitudes”, Archives of Psychology, 140, 1
- LinQuanSinhaBakshiHuynhKatzKarger, 2003, Proceedings of the 9th IFIP TCI3 International Conference on Human-Computer Interaction (INTERACT 2003)
- LinSmucker, 2008, Proceedings of the 31st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 08), 19
- LinBelkin, 2000, Proceedings of the Annual Conference of the American Society for Information Science (ASIS TO), 133
- Losee, 1996, “Evaluating retrieval performance given database and query characteristics: Analytical determination of performance surfaces”, Journal of the American Society for Information Science, 47, 95, 10.1002/(SICI)1097-4571(199601)47:1<95::AID-ASI9>3.0.CO;2-Y · doi:10.1002/(sici)1097-4571(199601)47:1<95::aid-asi9>3.0.co;2-y
- Luhn, 1958, H. P. Luhn: Pioneer of Infor-mation Science. Selected Works, 132
- Lund, 2001, “Measuring usability with the USE Questionnaire”, Usability and User Experience, 8
- Mackay, 1995, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 138
- Marchionini, 2008, “Toward human-Computer information retrieval”, Bulletin of the American Society for Information Science
- Michel, 1994, “What is used during cognitive processing in information retrieval and library searching? Eleven sources of search information”, Journal of the American Society for Information Science, 45, 498, 10.1002/(SICI)1097-4571(199408)45:7<498::AID-ASI5>3.0.CO;2-T · doi:10.1002/(sici)1097-4571(199408)45:7<498::aid-asi5>3.0.co;2-t
- MillettFriedmanFelten, 2001, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 46
- Morahan-Martin, 1998, Psychology and the Internet: Intrapersonal, Interpersonal, and Transpersonal Implications, 169
- MorrisHorvitz, 2007, Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST ’07), 3
- MyersWell, 2003, Research Design and Statistical Analysis
- Nielsen, 2008, “Usability 101: Introduction to usability”, Jakob Nielsen’s Alertbox
- Oard, 2000, Proceedings of the 1st Cross-Language Evaluation Forum
- O’BrienTomsKellowayKelley, 2008, Proceedings of the American Society for Information Science and Technology
- PetersKurthFlahertySandoreKaske, 1993, “An introduction to the special section on transaction log analysis”, Library Hi Tech, 11, 38, 10.1108/eb047883 · doi:10.1108/eb047883
- Petrelli, 2008, “On the role of user-centred evaluation in the advancement of interactive information retrieval”, Information Processing and Management, 44, 22, 10.1016/j.ipm.2007.01.024 · doi:10.1016/j.ipm.2007.01.024
- Robertson, 2006, Proceedings of the 15th ACM international Conference on information and Knowledge Management (CIKM’06), 78
- RoddenFu, 2007, Proceedings of the Web Information-Seeking and Interaction ( WIST) Workshop at the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’Of), 29
- Rorvig, 1988, “Psychometric measurement and information retrieval”, Annual Review of Information Science and Technology, 23, 157
- Ruthven, 2006, Proceedings of the 1st International Conference on Information Interaction in Context, 10.1145/1164820 · doi:10.1145/1164820
- Saracevic, 1967, Proceedings of the Annual Meeting of the American Documentation Institute, 4, 100
- Saracevic, 1995, Proceedings of the 18th Annual ACM SIGIR Conference on Research and Development of Information Retrieval, 138
- SaracevicKantor, 1988, “A study of information seeking and retrieving. II. Users, questions and effectiveness”, Journal of the American Society for Information Science, 39, 177, 10.1002/(SICI)1097-4571(198805)39:3<177::AID-ASI3>3.0.CO;2-F · doi:10.1002/(sici)1097-4571(198805)39:3<177::aid-asi3>3.0.co;2-f
- Savage-KnepshieldBelkin, 1999, “Interaction in information retrieval: Trends over time”, Journal of the American Society for Information Science, 50, 1067, 10.1002/(SICI)1097-4571(1999)50:12<1067::AID-ASI6>3.0.CO;2-4 · doi:10.1002/(sici)1097-4571(1999)50:12<1067::aid-asi6>3.0.co;2-4
- Siatri, 1999, “The evolution of user studies”, Libri, 49, 132, 10.1515/libr.1999.49.3.132 · doi:10.1515/libr.1999.49.3.132
- SmithKantor, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’08), 147
- Sormunen, 2002, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’02), 324
- Spärck-Jones, 1981, Information Retrieval Experiment
- Spink, 1996, “Multiple search session model of end-user behavior: An exploratory study”, Journal of the American Society for Information Science, 47, 603, 10.1002/(SICI)1097-4571(199608)47:8<603::AID-ASI4>3.0.CO;2-X · doi:10.1002/(sici)1097-4571(199608)47:8<603::aid-asi4>3.0.co;2-x
- Spink, 1997, “Study of interactive feedback during mediated information retrieval”, Journal of the American Society for Information Science, 48, 382, 10.1002/(SICI)1097-4571(199705)48:5<382::AID-ASI2>3.0.CO;2-R · doi:10.1002/(sici)1097-4571(199705)48:5<382::aid-asi2>3.0.co;2-r
- Stern, 2003, “Encountering distressing information in online research: A consideration of legal and ethical responsibilities”, New Media and Society, 5, 249, 10.1177/1461444803005002006 · doi:10.1177/1461444803005002006
- Sugar, 1995, “User-centered perspectives of information retrieval research and analysis methods”, Annual Review of Information Science and Technology, 30, 77
- SunKantor, 2006, “Cross-evaluation: A new model for information system evaluation”, Journal of American Society for Information Science and Technology, 56, 614
- Tague, 1987, “Informativeness as an ordinal utility function for information retrieval”, SIGIR Forum, 21, 10, 10.1145/30075.30077 · doi:10.1145/30075.30077
- Tague, 1981, Information Retrieval Experiment, 59
- Tague-Sutcliffe, 1992, Proceedings of the 15th Annual AGM International Conference on Research and Development in Information Retrieval (SIGIR ’92), 23
- TeevanAlvaradoAckermanKarger, 2004, Proceedings of the AGM SIGCHI Conference on Human Factors in Computing Systems (SIGCHI ’04), 415
- TeevanDumaisHorvitz, 2005, Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’05), 449
- ThomasHawking, 2006, Proceedings of the 15th Annual Conference on Information and Knowledge Management (CIKM ’06), 94
- TurpinHersh, 2001, Proceedings of the 24th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’01), 225
- TurpinScholer, 2006, Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’06), 11
- TurtlePennimanHickey, 1981, “Data entry/display devices for interactive information retrieval”, Annual Review of Information Science and Technology, 16, 55
- Urquhart, 1948, “The distribution and use of scientific and technical information”, The Royal Society Scientific Information Conference, 408
- VakkariJärvelin, 2005, New Directions in Cognitive Information Retrieval, 113
- VeerasamyBelkin, 1996, Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 85
- VeerasamyHeikes, 1997, “Effectiveness of a graphical display of retrieval results”, SIGIR Forum, 31, 236, 10.1145/278459.258580 · doi:10.1145/278459.258580
- VoorheesHarman, 2005, TREC: Experiment and Evaluation in Information Retrieval
- Wang, 1999, “Methodologies and methods for user behavior research”, Annual Review of Information Science and Technology, 34, 53
- WenRuthvenBorlund, 2006, Proceedings of the 28th European Conference in Information Retrieval (ECIR 2006’)
- WhiteBilenkoCucerzan, 2007, Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’07), 159
- XiongAgichtein, 2007, Proceedings of the Workshop on Query Log Analysis: Social and Technology Challenges at the 16th International World Wide Web Conference
- YuanMeadow, 1999, “A study of the use of variables in information retrieval user studies”, Journal of the American Society for Information Science, 50, 140, 10.1002/(SICI)1097-4571(1999)50:2<140::AID-ASI5>3.0.CO;2-P · doi:10.1002/(sici)1097-4571(1999)50:2<140::aid-asi5>3.0.co;2-p
- ZhangPlettenbergKlavansOardSoergel, 2007, Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries (JCDL ’07), 117
- Zhang, 2002, “Collaborative relevance judgments: A group consensus method for evaluating user search performance”, Journal of the American Society for Information Science and Technology, 53, 220, 10.1002/asi.10036 · doi:10.1002/asi.10036
Cited by
Crowdsourcing for relevance evaluation
Alonso, O., Rose, D. E., & Stewart, B. (2008) · ACM SIGIR Forum · doi:10.1145/1480506.1480508 · Cited by 329
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317
Mehrotra, R., McInerney, J., Bouchard, H., Lalmas, M., & Diaz, F. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3272027 · Cited by 279
Lalmas, M., O’Brien, H., & Yom-Tov, E. (2015) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02289-0 · Cited by 192
Kelly, D., Arguello, J., Edwards, A., & Wu, W.-C. (2015) · Proceedings of the 2015 International Conference on The Theory of Information Retrieval · doi:10.1145/2808194.2809465 · Cited by 155
Self-citation
Metadata
- Authors
- Kelly, D.
- Venue
- Foundations and Trends® in Information Retrieval
- Publisher
- Emerald
- Published
- 2009-05-27
- Volume
- 3
- Issue
- 1-2
- Pages
- 1-224
- ISSN
- 1554-0669 (ISSN-L)
- 1554-0677
- OpenAlex
- W2035569891
- Cited by
- 633
- References
- 241
- FWCI
- 60.91
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.