Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 633

The DOI matches its Crossref record; 241 of the 241 references it cites were found in a registry or an index.

Methods for Evaluating Interactive Information Retrieval Systems with Users

Kelly, Diane·2009·Foundations and Trends® in Information Retrieval

Abstract

This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users. The primary goal of this article is to catalog and compile material related to this topic into a single source. This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies. This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR). Finally, this article concludes with a discussion of outstanding challenges and future research directions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1561/1500000012

Cites

241 of 241 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 133

  1. Adar, 2007, Proceedings of the Workshop on Query Log Analysis: Social and Technological Challenges, at the 16th International World Wide Web Conference
  2. AgichteinBrillDumais, 2006, Proceedings of the 29th Annual ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’06), 3
  3. Allan, 2004, TREC2003, Proceedings of the 12th Text Retrieval Conference
  4. Allan, 2006, TREC2005, Proceedings of the l4th Text Retrieval Conference
  5. Allen, 1997, Information Seeking in Context: Proceedings of an International Conference on Research in Information Needs, Seeking and Use in Different Contexts, 111
  6. Amdur, 2003, Institutional Review Board Member Handbook
  7. American Psychological Association, 2001, Publication Manual of the American Psychological Association
  8. AmitayMurrayTeevan, 2007, Proceedings of the 16th International World Wide Web Conference
  9. Anick, 2003, Proceedings of the 26th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ‘03), 88
  10. ArapakisJose, 2008, Proceedings of the 31st Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ‘08), 395
  11. BaileyCraswellSoboroffThomasde VriesYilmaz, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’08), 667
  12. BeaulieuGatfordHuangRobertsonWalkerWilliams, 1997, Proceedings of the 5th Text Retrieval Conference (TREC-5), 143
  13. Belkin, 1980, “Anomalous states of knowledge as a basis for information retrieval”, Canadian Journal of Information Science, 5, 133
  14. BelkinCabezasCoolKimNgParkPressmanRiehSavageXie, 1997, Proceedings of the 5th Text Retrieval Conference (TREC-5), 257
  15. BelkinKellyKimKimLeeMuresanTangYuanCool, 2003, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 205
  16. BellRuthven, 2004, Proceedings of the 26th Annual International European Conference on Information Retrieval (ECIR 200fi), 57
  17. Bennett, 1972, “The user interface in interactive systems”, Annual Review of Information Science and Technology, 7, 159
  18. Bernal, 1948, “Preliminary analysis of pilot questionnaires on the use of scientific literature”, The Royal Society Scientific Information Conference, 589
  19. BilenkoWhite, 2008, Proceedings of the 17th International Conference on the World Wide Web (WWW ’08), 51
  20. Borgman, 1983, Proceedings of the 6th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’83), 162
  21. BorlundIngwersen, 1998, Proceedings of the 21st ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR ’98), 324
  22. BorlundRuthven, 2008, “Introduction to the special issue on evaluating interactive information retrieval systems”, Information Processing and Management, 44, 1
  23. BorlundSchneiderLaimasTombrosFeatherKellyde VriesAzzopardi, 2008, Proceedings of the 2nd International Symposium on Information Interaction in Context
  24. BoyceMeadowKraft, 1994, Measurement in Information Science
  25. BudzikHammond, 2000, Proceedings of the 5th International Conference on Intelligent User Interfaces (IUI TO), 44
  26. Capra, 2008, Personal Information Management, CHI 2008 Workshop
  27. Charmaz, 2002, Handbook of Interview Research: Context and Method
  28. Cleverdon, 1997, Readings in Information Retrieval, 173
  29. CowleyHaackLittlefieldHampson, 2006, Proceedings of the 2nd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE ’05), 13
  30. CzerwinskiHorvitzCutrell, 2001, Proceedings of IHM-HCI 2001 Conference, 167
  31. CzerwinskiHorvitzWilhite, 2004, Proceedings of ACM Human Factors in Computing Systems Conference (CHI ’Of), 175
  32. DahlbäckJonssonAhrenberg, 1993, Proceedings for the 1st International Conference on Intelligent User Interfaces (IUI ’93), 193
  33. DangKellyLin, 2008, TREC2007, Proceedings of the 16th Text Retrieval Conference
  34. De Mey, 1977, CC77: International Workshop on the Cognitive Viewpoint, xvi
  35. Dervin, 1992, Qualitative Research in Information Management, 61
  36. Dervin, 1996, Information Seeking in Context: Proceedings of an International Conference on Research in Information Needs, Seeking and Use in Different Contexts, 13
  37. DominickPenniman, 1979, Proceedings of the 2nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’79), 2
  38. DumaisCutrellCadizJanckeSarinRobbins, 2003, Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR ’03), 72
  39. Dunlop, 1997, Proceedings of the 20th ACM Conference on Research and Development in Information Retrieval (SIGIR ’97), 206
  40. ElsweilerRuthven, 2007, Proceedings of the 30th ACM Conference on Research and Development in Information Retrieval (SIGIR ’03), 22
  41. FenstermacherGinsburg, 2003, “Client-side monitoring for Web mining”, Journal of the American Society for Information Science and Technoloqy, 54, 625, 10.1002/asi.10255 · doi:10.1002/asi.10255
  42. Fiske, 1982, Forms of Validity in Research, 77
  43. FittsJonesMilton, 1950, “Eye movements of aircraft pilots during instrument-landing approaches”, Aeronautical Engineering Review, 9, 24
  44. GhaniSupnickRooney, 1991, Proceedings of International Conference on Information Systems (JCIS 1991), 229
  45. GrimesTangRussell, 2007, Proceedings of the Workshop on Query Log Analysis: Social and Technology Challenges at the 16th International World Wide Web Conference
  46. GuanLeeCuddihyRamey, 2006, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 1253
  47. HaradaNaamanSongWangPaepcke, 2004, Proceedings of the 4th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL T4), 325
  48. Harman, 2005, TREC: Experiment and Evaluation in Information Retrieval, 21
  49. HartStaveland, 1988, Human Mental Workload, 139
  50. HawkingBaileyCraswell, 2000, “Efficient and flexible search using text and metadata”, CSIRO Mathematical and Information Sciences Tech Report, 2000-83
  51. Hersh, 2006, Advances in Open Domain Question Answering, 431
  52. HershOver, 2001, “Introduction to a special issue on interactivity at the Text Retrieval Conference (TREC)”, Information Processing and Management, 37, 365, 10.1016/S0306-4573(00)00052-2 · doi:10.1016/s0306-4573(00)00052-2
  53. HershTurpinPriceChanKramerSacherekOlson, 2000, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR TO), 17
  54. HershElliotHickamWolfMolnarLeichtenstein, 1995, Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’95), 164
  55. HirshDinkelacker, 2004, “Seeking information in order to produce information: An empirical study at Hewlett Packard Labs”, Journal of the American Society for Information Science and Technology, 55, 807, 10.1002/asi.20024 · doi:10.1002/asi.20024
  56. HuangWang, 2004, “The influence of document presentation order and number of documents judged on users’ judgments of relevance”, Journal of the American Society for Information Science, 55, 970, 10.1002/asi.20047 · doi:10.1002/asi.20047
  57. IngwersenWillett, 1995, “An introduction to algorithmic and cognitive approaches for information retrieval”, Libri, 45, 160
  58. International Standards Office, 1998, Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs): Part II, Guidance on Usability
  59. JacobKarn, 2003, The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, 573
  60. JansenRamadossZhangZang, 2006, Workshop on Evaluating Exploratory Search Systems at the 29th Annual International ACM SIGIR Conference on Research and Development on Information Retrieval (SIGIR ‘06)
  61. JärvelinKekäläinen, 2000, Proceedings of the 23rd ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR ’00), 41
  62. JärvelinPriceDelcambreNielsen, 2008, Proceedings of the 30th European Conference on Information Retrieval
  63. Jex, 1988, Human Mental Workload, 5
  64. JoachimsGrankaPanHembrookeGay, 2005, Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’05), 154
  65. JohoHannahJose, 2008, Proceedings of the 2nd IHX Symposium on Information Interaction in Context, 89
  66. JonesTeevan, 2007, Personal Information Management
  67. KäkiAula, 2008, “Controlling the complexity in comparing search user interfaces via user studies”, Information Processing and Management, 44, 82, 10.1016/j.ipm.2007.02.006 · doi:10.1016/j.ipm.2007.02.006
  68. KantorKazaiMilic-FraylingWilkinson, 2008, Proceedings of the Conference on Information and Knowledge Management (CIKM ’08)
  69. Kantor, 1982, “Evaluation of and feedback in information storage and retrieval systems”, Annual Review of Information Science and Technology, 17, 99
  70. KellyFu, 2006, Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’06), 453
  71. KellyKantorMorseScholtzSun, 2006, Proceedings of the Workshop on Interactive Question Answering at the Human Language Technology Conference (HLT-NAACL ’06)
  72. KellyLin, 2007, “Overview of the TREC 2006 ciQA task”, SIGIR Forum, 41, 107, 10.1145/1273221.1273231 · doi:10.1145/1273221.1273231
  73. KellyShahSugimotoBaileyClemensIrvineJohnsonKeOhPoljakovaRodriguezvan NoordZhang, 2008, Proceedings of the 2nd Symposium on Information Interaction in Context (IIiX), 75
  74. KellyWacholderRittmanSunKantorSmallStrzalkowski, 2007, “Using interview data to identify evaluation criteria for interactive, analytical question answering systems”, Journal of the American Society for Information Science and Technology, 58, 1032, 10.1002/asi.20575 · doi:10.1002/asi.20575
  75. King, 1968, “Design and evaluation of information systems”, Annual Review of Information Science and Technology, 3, 61
  76. LarsenMalikTombros, 2006, INEX 2005, 398
  77. Likert, 1932, “A technique for the measurement of attitudes”, Archives of Psychology, 140, 1
  78. LinQuanSinhaBakshiHuynhKatzKarger, 2003, Proceedings of the 9th IFIP TCI3 International Conference on Human-Computer Interaction (INTERACT 2003)
  79. LinSmucker, 2008, Proceedings of the 31st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 08), 19
  80. LinBelkin, 2000, Proceedings of the Annual Conference of the American Society for Information Science (ASIS TO), 133
  81. Losee, 1996, “Evaluating retrieval performance given database and query characteristics: Analytical determination of performance surfaces”, Journal of the American Society for Information Science, 47, 95, 10.1002/(SICI)1097-4571(199601)47:1<95::AID-ASI9>3.0.CO;2-Y · doi:10.1002/(sici)1097-4571(199601)47:1<95::aid-asi9>3.0.co;2-y
  82. Luhn, 1958, H. P. Luhn: Pioneer of Infor-mation Science. Selected Works, 132
  83. Lund, 2001, “Measuring usability with the USE Questionnaire”, Usability and User Experience, 8
  84. Mackay, 1995, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 138
  85. Marchionini, 2008, “Toward human-Computer information retrieval”, Bulletin of the American Society for Information Science
  86. Michel, 1994, “What is used during cognitive processing in information retrieval and library searching? Eleven sources of search information”, Journal of the American Society for Information Science, 45, 498, 10.1002/(SICI)1097-4571(199408)45:7<498::AID-ASI5>3.0.CO;2-T · doi:10.1002/(sici)1097-4571(199408)45:7<498::aid-asi5>3.0.co;2-t
  87. MillettFriedmanFelten, 2001, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 46
  88. Morahan-Martin, 1998, Psychology and the Internet: Intrapersonal, Interpersonal, and Transpersonal Implications, 169
  89. MorrisHorvitz, 2007, Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST ’07), 3
  90. MyersWell, 2003, Research Design and Statistical Analysis
  91. Nielsen, 2008, “Usability 101: Introduction to usability”, Jakob Nielsen’s Alertbox
  92. Oard, 2000, Proceedings of the 1st Cross-Language Evaluation Forum
  93. O’BrienTomsKellowayKelley, 2008, Proceedings of the American Society for Information Science and Technology
  94. PetersKurthFlahertySandoreKaske, 1993, “An introduction to the special section on transaction log analysis”, Library Hi Tech, 11, 38, 10.1108/eb047883 · doi:10.1108/eb047883
  95. Petrelli, 2008, “On the role of user-centred evaluation in the advancement of interactive information retrieval”, Information Processing and Management, 44, 22, 10.1016/j.ipm.2007.01.024 · doi:10.1016/j.ipm.2007.01.024
  96. Robertson, 2006, Proceedings of the 15th ACM international Conference on information and Knowledge Management (CIKM’06), 78
  97. RoddenFu, 2007, Proceedings of the Web Information-Seeking and Interaction ( WIST) Workshop at the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’Of), 29
  98. Rorvig, 1988, “Psychometric measurement and information retrieval”, Annual Review of Information Science and Technology, 23, 157
  99. Ruthven, 2006, Proceedings of the 1st International Conference on Information Interaction in Context, 10.1145/1164820 · doi:10.1145/1164820
  100. Saracevic, 1967, Proceedings of the Annual Meeting of the American Documentation Institute, 4, 100
  101. Saracevic, 1995, Proceedings of the 18th Annual ACM SIGIR Conference on Research and Development of Information Retrieval, 138
  102. SaracevicKantor, 1988, “A study of information seeking and retrieving. II. Users, questions and effectiveness”, Journal of the American Society for Information Science, 39, 177, 10.1002/(SICI)1097-4571(198805)39:3<177::AID-ASI3>3.0.CO;2-F · doi:10.1002/(sici)1097-4571(198805)39:3<177::aid-asi3>3.0.co;2-f
  103. Savage-KnepshieldBelkin, 1999, “Interaction in information retrieval: Trends over time”, Journal of the American Society for Information Science, 50, 1067, 10.1002/(SICI)1097-4571(1999)50:12<1067::AID-ASI6>3.0.CO;2-4 · doi:10.1002/(sici)1097-4571(1999)50:12<1067::aid-asi6>3.0.co;2-4
  104. Siatri, 1999, “The evolution of user studies”, Libri, 49, 132, 10.1515/libr.1999.49.3.132 · doi:10.1515/libr.1999.49.3.132
  105. SmithKantor, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’08), 147
  106. Sormunen, 2002, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’02), 324
  107. Spärck-Jones, 1981, Information Retrieval Experiment
  108. Spink, 1996, “Multiple search session model of end-user behavior: An exploratory study”, Journal of the American Society for Information Science, 47, 603, 10.1002/(SICI)1097-4571(199608)47:8<603::AID-ASI4>3.0.CO;2-X · doi:10.1002/(sici)1097-4571(199608)47:8<603::aid-asi4>3.0.co;2-x
  109. Spink, 1997, “Study of interactive feedback during mediated information retrieval”, Journal of the American Society for Information Science, 48, 382, 10.1002/(SICI)1097-4571(199705)48:5<382::AID-ASI2>3.0.CO;2-R · doi:10.1002/(sici)1097-4571(199705)48:5<382::aid-asi2>3.0.co;2-r
  110. Stern, 2003, “Encountering distressing information in online research: A consideration of legal and ethical responsibilities”, New Media and Society, 5, 249, 10.1177/1461444803005002006 · doi:10.1177/1461444803005002006
  111. Sugar, 1995, “User-centered perspectives of information retrieval research and analysis methods”, Annual Review of Information Science and Technology, 30, 77
  112. SunKantor, 2006, “Cross-evaluation: A new model for information system evaluation”, Journal of American Society for Information Science and Technology, 56, 614
  113. Tague, 1987, “Informativeness as an ordinal utility function for information retrieval”, SIGIR Forum, 21, 10, 10.1145/30075.30077 · doi:10.1145/30075.30077
  114. Tague, 1981, Information Retrieval Experiment, 59
  115. Tague-Sutcliffe, 1992, Proceedings of the 15th Annual AGM International Conference on Research and Development in Information Retrieval (SIGIR ’92), 23
  116. TeevanAlvaradoAckermanKarger, 2004, Proceedings of the AGM SIGCHI Conference on Human Factors in Computing Systems (SIGCHI ’04), 415
  117. TeevanDumaisHorvitz, 2005, Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’05), 449
  118. ThomasHawking, 2006, Proceedings of the 15th Annual Conference on Information and Knowledge Management (CIKM ’06), 94
  119. TurpinHersh, 2001, Proceedings of the 24th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’01), 225
  120. TurpinScholer, 2006, Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’06), 11
  121. TurtlePennimanHickey, 1981, “Data entry/display devices for interactive information retrieval”, Annual Review of Information Science and Technology, 16, 55
  122. Urquhart, 1948, “The distribution and use of scientific and technical information”, The Royal Society Scientific Information Conference, 408
  123. VakkariJärvelin, 2005, New Directions in Cognitive Information Retrieval, 113
  124. VeerasamyBelkin, 1996, Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 85
  125. VeerasamyHeikes, 1997, “Effectiveness of a graphical display of retrieval results”, SIGIR Forum, 31, 236, 10.1145/278459.258580 · doi:10.1145/278459.258580
  126. VoorheesHarman, 2005, TREC: Experiment and Evaluation in Information Retrieval
  127. Wang, 1999, “Methodologies and methods for user behavior research”, Annual Review of Information Science and Technology, 34, 53
  128. WenRuthvenBorlund, 2006, Proceedings of the 28th European Conference in Information Retrieval (ECIR 2006’)
  129. WhiteBilenkoCucerzan, 2007, Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR ’07), 159
  130. XiongAgichtein, 2007, Proceedings of the Workshop on Query Log Analysis: Social and Technology Challenges at the 16th International World Wide Web Conference
  131. YuanMeadow, 1999, “A study of the use of variables in information retrieval user studies”, Journal of the American Society for Information Science, 50, 140, 10.1002/(SICI)1097-4571(1999)50:2<140::AID-ASI5>3.0.CO;2-P · doi:10.1002/(sici)1097-4571(1999)50:2<140::aid-asi5>3.0.co;2-p
  132. ZhangPlettenbergKlavansOardSoergel, 2007, Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries (JCDL ’07), 117
  133. Zhang, 2002, “Collaborative relevance judgments: A group consensus method for evaluating user search performance”, Journal of the American Society for Information Science and Technology, 53, 220, 10.1002/asi.10036 · doi:10.1002/asi.10036

Cited by

See all 633

Metadata

Authors
Kelly, D.
Venue
Foundations and Trends® in Information Retrieval
Publisher
Emerald
Published
2009-05-27
Volume
3
Issue
1-2
Pages
1-224
ISSN
  • 1554-0669 (ISSN-L)
  • 1554-0677
OpenAlex
W2035569891
Cited by
633
References
241
FWCI
60.91
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.