CoChat
Sign Up
Cited by 485Information Retrieval and Search BehaviorRecommender Systems and TechniquesAlgorithms and Data Compression

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Test Collection Based Evaluation of Information Retrieval Systems

Sanderson, Mark·2010·Foundations and Trends® in Information Retrieval
DOI 10.1561/1500000009
Open this paper in CoChat — free account

Abstract

Use of test collections and evaluation measures to assess the effectiveness of information retrieval systems has its origins in work dating back to the early 1950s. Across the nearly 60 years since that work started, use of test collections is a de facto standard of evaluation. This monograph surveys the research conducted and explains the methods and measures devised for evaluation of retrieval systems, including a detailed look at the use of statistical significance testing in retrieval experimentation. This monograph reviews more recent examinations of the validity of the test collection approach and evaluation measures as well as outlining trends in current research exploiting query logs and live labs. At its core, the modern-day test collection is little different from the structures that the pioneering researchers in the 1950s and 1960s conceived of. This tutorial and review shows that despite its age, this long-standing evaluation method is still a highly valued tool for retrieval research.

Overview

Integrity signals

Registry

Crossref · 10.1561/1500000009

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 281

  1. Agichtein, 2006, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 3, 10.1145/1148170.1148175 · DOI
  2. Agrawal, 2009, Proceedings of the Second ACM International Conference on Web Search and Data Mining, 5, 10.1145/1498759.1498766 · DOI
  3. Al-Maskari, 2007, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 773, 10.1145/1277741.1277902 · DOI
  4. Al-Maskari, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 59, 10.1145/1390334.1390347 · DOI
  5. Allan, 2002, Topic Detection and Tracking: Event-based Information Organization, The Kluwer International Series on Information Retrieval, 12, 10.1007/978-1-4615-0933-2 · DOI
  6. Allan, 2005, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 433, 10.1145/1076034.1076109 · DOI
  7. Alonso, 2008, “Crowdsourcing for relevance evaluation”, ACM SIGIR Forum, 42, 9, 10.1145/1480506.1480508 · DOI
  8. Amitay, 2004, Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 10, 10.1145/1008992.1008997 · DOI
  9. Arni, 2009, Evaluating Systems for Multilingual and Multimodal Information Access, 5706/2009, 500, 10.1007/978-3-642-04447-2-62 · DOI
  10. Artiles, 2008, Proceedings of the 17th International Conference on World Wide Web, 1071, 10.1145/1367497.1367661 · DOI
  11. Aslam, 2003, Proceedings of the Twelfth International Conference on Information and Knowledge Management, 484, 10.1145/956863.956953 · DOI
  12. Aslam, 2006, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 541, 10.1145/1148170.1148263 · DOI
  13. Aslam, 2005, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 573, 10.1145/1076034.1076134 · DOI
  14. Aslam, 2005, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 27, 10.1145/1076034.1076042 · DOI
  15. Azzopardi, 2007, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 455, 10.1145/1277741.1277820 · DOI · Paper page
  16. Azzopardi, 2008, Proceedings of the 17th ACM Conference on Information and Knowledge Management, 561, 10.1145/1458082.1458157 · DOI
  17. Baeza-Yates, 1999, Modern Information Retrieval
  18. Bailey, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 355, 667
  19. Baillie, 2007, Advances in Information Retrieval, 4425, 271
  20. Baron, 2006, The Fifteenth Text Retrieval Conference (TREC 2006) Proceedings, 500, 79
  21. Bjprner, 2003, “Online before internet, Part 1: Early pioneers tell their stories”, Searcher: The Magazine for Database Professionals, 11
  22. Blair, 1996, “STAIRS redux: Thoughts on STAIRS evaluation, ten years after”, Journal of the American Society for Information Science, 47, 4, 10.1002/(SICI)1097-4571(199601)47:1<4::AID-ASI2>3.0.CO;2-3 · DOI
  23. Blair, 1985, “An evaluation of retrieval effectiveness for a full-text document-retrieval system”, Communications of the ACM, 28, 289, 10.1145/3166.3197 · DOI
  24. Bodoff, 2007, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 367, 10.1145/1277741.1277805 · DOI
  25. Bompada, 2007, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, 359, 10.1145/1277741.1277804 · DOI
  26. Bookstein, 1977, “When most “pertinent” document should not be retrieved — An analysis of Swets model”, Information Processing & Management, 13, 377, 10.1016/0306-4573(77)90057-7 · DOI
  27. Borko, 1962, Evaluating The: Effectiveness of Information Retrieval Systems
  28. Borlund, 2003, “The concept of relevance in IR”, Journal of the American Society for Information Science and Technology, 54, 913, 10.1002/asi.10286 · DOI
  29. Borlund, 1997, “The development of a method for evaluation of interactive information retrieval systems”, Journal of Documentation, 53, 225, 10.1108/EUM0000000007198 · DOI
  30. Bornstein, 1961, “A paradigm for a retrieval effectiveness experiment”, American Documentation, 12, 254, 10.1002/asi.5090120403 · DOI
  31. Braschler, 2004, Information Retrieval, 7, 7, 10.1023/B:INRT.0000009438.69013.fa · DOI
  32. Broder, 2002, “A taxonomy of web search”, SIGIR Forum, 36, 3, 10.1145/792550.792552 · DOI · Paper page
  33. Bryant, 1966, Information Retrieval Among Examining Patent Offices: Fifth Annual Meeting of the Committee for International Cooperation in Information Retrieval among Examining Patent Offices (ICIREPAT), 362
  34. Buckley, 2007, “Bias and limits of pooling for large collections”, Information Retrieval, 10, 491, 10.1007/s10791-007-9032-x · DOI
  35. Buckley, 2000, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 33, 10.1145/345508.345543 · DOI
  36. Buckley, 2004, Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 25, 10.1145/1008992.1009000 · DOI
  37. Buckley, 2005, TREC: Experiment and Evaluation in Information Retrieval, 53
  38. Burgin, 1992, “Variations in relevance judgments and evaluation of retrieval performance”, Information Processing & Management, 28, 619, 10.1016/0306-4573(92)90031-T · DOI
  39. Büttcher, 2006, Proceedings of the Fifteenth Text Retrieval Conference (TREC 2006), 500, 128
  40. Büttcher, 2007, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 63, 10.1145/1277741.1277755 · DOI
  41. Can, 2004, “Automatic performance evaluation of Web search engines”, Information Processing and Management, 40, 495, 10.1016/S0306-4573(03)00040-2 · DOI
  42. Carterette, 2007, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 55, 10.1145/1277741.1277754 · DOI
  43. Carterette, 2009, Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 436, 10.1145/1571941.1572017 · DOI
  44. Carterette, 2006, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 268, 10.1145/1148170.1148219 · DOI
  45. Carterette, 2008, Advances in Information Retrieval, 16, 10.1007/978-3-540-78646-7_5 · DOI
  46. Carterette, 2007, Proceedings of the 21st Annual Conference on Neural Information Processing Systems (NIPS), 217
  47. Carterette, 2008, Advances in Neural Information Processing Systems, 20, 217
  48. Carterette, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 651, 10.1145/1390334.1390445 · DOI
  49. Chapelle, 2009, Proceedings of the 18th ACM conference on Information and knowledge management, 621, 10.1145/1645953.1646033 · DOI
  50. Chen, 2006, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 429, 10.1145/1148170.1148245 · DOI
  51. Clarke, 2009, Working notes of the proceedings of TREC 2009
  52. Clarke, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 659, 10.1145/1390334.1390446 · DOI
  53. Clarke, 2009, Advances in Information Retrieval Theory: Second International Conference on the Theory of Information Retrieval, ICTIR 2009 Cambridge, UK, September 10-12, 2009 Proceedings, 188, 10.1007/978-3-642-04417-5_17 · DOI
  54. Cleverdon, 1959, Proceedings of the International Conference on Scientific Information — Two Volumes, 687
  55. Cleverdon, 1962, Report on the Testing and Analysis of an Investigation Into the Comparative Efficiency of Indexing Systems
  56. Cleverdon, 1970, The Effect of Variations in Relevance Assessments in Comparative Experimental Tests of Index Languages
  57. Cleverdon, 1991, Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 3, 10.1145/122860.122861 · DOI
  58. Cleverdon, 1966, Factors Affecting Performance of Indexing Systems, Vol 2, 37
  59. Clough, 2008, Proceedings of Workshop on Novel Methodologies for Evaluation in Information Retrieval, 33
  60. Clough, 2006, Accessing Multilingual Information Repositories, 4022, 535, 10.1007/11878773_60 · DOI
  61. Clough, 2004, Lecture notes in Computer Science, 243
  62. Cooper, 1968, “Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems”, American Documentation, 19, 30, 10.1002/asi.5090190108 · DOI
  63. Cooper, 1971, “A definition of relevance for information retrieval”, Information storage and retrieval, 7, 19, 10.1016/0020-0271(71)90024-6 · DOI
  64. Cooper, 1973, “On selecting a measure of retrieval effectiveness”, Journal of the American Society for Information Science, 24, 10.1002/asi.4630240204 · DOI
  65. Cormack, 2006, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 533, 10.1145/1148170.1148262 · DOI
  66. Cormack, 1998, Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 282, 10.1145/290941.291009 · DOI
  67. Craswell, 2005, Proceedings of the Fourteenth Text Retrieval Conference (TREC 2005)
  68. Craswell, 2003, The Eleventh Text Retrieval Conference (TREC-2002), 86
  69. Craswell, 2008, Proceedings of the international conference on Web search and web data mining, 87, 10.1145/1341531.1341545 · DOI
  70. Croft, 1978, Proceedings of the 1st Annual International ACM SIGIR Conference on Information Storage and Retrieval, 65
  71. Croft, 2009, Search Engines: Information Retrieval in Practice
  72. Cutrell, 2006, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 261, 10.1145/1124772.1124812 · DOI
  73. Davies, 1983, A Document Test Collection for Use in Information Retrieval Research
  74. Demartini, 2006, “A classification of IR effectiveness metrics”, Advances in Information Retrieval, 3936, 488
  75. Dennis, 1962, “Index manipulation and abstract retrieval by computer”, Journal of Chemical Documentation, 2, 234, 10.1021/c160007a014 · DOI
  76. Dumais, 2003, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 72
  77. Dupret, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 331, 10.1145/1390334.1390392 · DOI
  78. Efron, 1993, An introduction to the bootstrap, 57, 1
  79. Fairthorne, 1963, “Implications of test procedures”, Information Retrieval in Action, 109
  80. Fels, 1963, “Evaluation of the performance of an information-retrieval system by modified Mooers plan”, American Documentation, 14, 28, 10.1002/asi.5090140105 · DOI
  81. Fox, 1990, Virginia Disc One
  82. Fox, 2005, “Evaluating implicit measures to improve web search”, ACM Transactions on Information Systems (TOIS), 23, 147, 10.1145/1059981.1059982 · DOI · Paper page
  83. Frei, 1991, “Determining the effectiveness of retrieval algorithms”, Information Processing and Management: An International Journal, 27, 153, 10.1016/0306-4573(91)90046-O · DOI
  84. Fuhr, “Optimum polynomial retrieval functions based on the probability ranking principle”, ACM Transactions on Information Systems, 7, 10.1145/65943.65944 · DOI
  85. Fuhr, 2002, Proceedings of the SIGIR 2002 Workshop on XML and Information Retrieval
  86. Fuhr, 1984, Proceedings of the 7th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 391
  87. Garofolo, 2000, “The TREC spoken document retrieval track: A success story”, Proceedings of the Eighth Text Retrieval Conference (TREC-8), 107
  88. Gey, 2006, Accessing Multilingual Information Repositories, 4022, 908, 10.1007/11878773_101 · DOI
  89. Gigerenzer, 2004, “Mindless statistics”, Journal of Socio-Economics, 33, 587
  90. Goffman, 1964, “On relevance as a measure”, Information Storage and Retrieval, 2, 201, 10.1016/0020-0271(64)90004-X · DOI
  91. Gull, 1956, “Seven years of work on the organization of materials in the special library”, American Documentation, 7, 320, 10.1002/asi.5090070408 · DOI
  92. Harman, 1992, “Evaluation issues in information retrieval”, Information Processing & Management, 28, 439, 10.1016/0306-4573(92)90001-G · DOI
  93. Harman, 1993, NIST Special Publication. Presented at the Second Text Retrieval Conference (TREC 2), 10.6028/NIST.SP.500-215 · DOI
  94. Harman, 1994, The Third Text Retrieval Conference (TREC-3), Gaithersburg, MD, USA
  95. Harman, 1995, The Forth Text Retrieval Conference (TREC-f), Gaithersburg, MD, USA,
  96. Harman, 1995, “Overview of the second text retrieval conference (TREC-2)”, Information Processing and Management, 31, 271, 10.1016/0306-4573(94)00047-7 · DOI
  97. Harman, 2002, Proceedings of the Eleventh Text Retrieval Conference (TREC 2002), 46
  98. Harman, 2002, Presented at the Center for Language and Speech Processing
  99. Harman, 2005, TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing), 79
  100. Harman, 2004, Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 528, 10.1145/1008992.1009104 · DOI
  101. Harman, 1990, “Retrieving records from a gigabyte of text on a minicomputer using statistical ranking”, Journal of the American Society for Information Science, 41, 581, 10.1002/(SICI)1097-4571(199012)41:8<581::AID-ASI4>3.0.CO;2-U · DOI
  102. Harmandas, 1997, Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 296
  103. Haveliwala, 2002, Proceedings of the 11th International Conference on World Wide Web, 432, 10.1145/511446.511502 · DOI
  104. Hawking, 2001, NIST Special Publication, 87
  105. Hawking, 2000, NIST Special Publication, 307
  106. Hawking, 1998, The Seventh Text Retrieval Conference (TREC-7)
  107. Hawking, 2004, Proceedings of the 15th Australasian database conference, 27, 77
  108. Hawking, 2003, “On collection size and retrieval effectiveness”, Information Retrieval, 6, 99, 10.1023/A:1022904715765 · DOI
  109. Hawking, 1997, The Sixth Text Retrieval Conference (TREC-6)
  110. Hearst, 2009, Search User Interfaces, 10.1017/CBO9781139644082 · DOI · Paper page
  111. Hearst, 1993, Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 59, 10.1145/160688.160695 · DOI
  112. Hersh, 1994, Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 192
  113. Hersh, 2006, The Fifteenth Text Retrieval Conference, 52
  114. Hersh, 2002, Journal of American Medical Informatics Association, 9
  115. Hersh, 2000, proceedings of the Ninth Text REtrieval Conference (TREC-9), 41
  116. Hersh, 2000, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 17, 10.1145/345508.345539 · DOI
  117. Holmstrom, 1948, The Royal Society Scientific Information Conference, 21 June-2 July 19f8: Report and papers submitted
  118. Huffman, 2007, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 567, 10.1145/1277741.1277839 · DOI
  119. Hull, 1993, Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 329, 10.1145/160688.160758 · DOI
  120. Huuskonen, 2008, “Students’ search process and outcome in Medline in writing an essay for a class on evidence-based medicine”, Journal of Documentation, 64, 287, 10.1108/00220410810858065 · DOI
  121. Ide, 1968, Report ISR-I4 to the National Science Foundation
  122. Ingwersen, 2005, The Turn: Integration of Information Seeking and Retrieval in Context
  123. Iwayama, 2003, Proceedings of the third NTCIR Workshop on Research in Information Retrieval, Automatic Text Summarization and Question Answering
  124. J. Jansen, 2006, Information Processing and Management, 42, 248, 10.1016/j.ipm.2004.10.007 · DOI · Paper page
  125. J. Jansen, 2007, “Web searcher interaction with Dogpile.com metasearch engine”, Journal of the American Society for Information Science and Technology, 58, 744, 10.1002/asi.20555 · DOI
  126. Järvelin, 2000, “IR evaluation methods for retrieving highly relevant documents”, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 41
  127. Jensen, 2006, Repeatable Evaluation of Information Retrieval Effectiveness In Dynamic Environments
  128. Jensen, 2007, “Repeatable evaluation of search services in dynamic environments”, ACM Transactions on Information Systems (TOIS), 26, 1, 10.1145/1292591.1292592 · DOI
  129. Joachims, 2002, Proceedings of the SIGIR Workshop on Mathematical/Formal Methods in Information Retrieval, 12
  130. Joachims, 2002, Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 133, 10.1145/775047.775067 · DOI
  131. Joachims, 2007, Computer, 34
  132. Kando, 2003, Comparative Evaluation of Multilingual Information Access Systems. 4th Workshop of the Cross-Language Evaluation Forum, CLEF, 29
  133. Kando, 1999, Proceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, 11
  134. Kantor, 1996, The Fifth Text Retrieval Conference (TREC-5)
  135. Kantor, 2000, 2, 165
  136. Katter, 1968, “The influence of scale form on relevance judgments”, Information Storage and Retrieval, 4, 1, 10.1016/0020-0271(68)90002-8 · DOI
  137. Katzer, 1982, Information Technology: Research and Development, 1, 261
  138. Katzer, 1981, A Study of the Impact of Representations in Information Retrieval Systems
  139. Kazai, 2006, Advances in XML Information Retrieval and Evaluation, 3977, 16
  140. Kazai, 2004, Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 72, 10.1145/1008992.1009008 · DOI
  141. Kazai, 2009, Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 452, 10.1145/1571941.1572019 · DOI
  142. Keen, 1967, Evaluation parameters
  143. Keen, 1992, “Presenting results of experimental retrieval comparisons”, Information Processing and Management: An International Journal, 28, 491, 10.1016/0306-4573(92)90006-L · DOI
  144. Kendall, 1938, “A new measure of rank correlation”, Biometrika, 30, 81, 10.1093/biomet/30.1-2.81 · DOI
  145. Kent, 2002, Encyclopedia of Library and Information Science
  146. Kent, 1955, American Documentation, 6, 93, 10.1002/asi.5090060209 · DOI
  147. Kuriyama, 2002, Information Retrieval, 5, 41, 10.1023/A:1012778807438 · DOI
  148. Laimas, 2007, SIGIR Forum, 41, 40, 10.1145/1273221.1273225 · DOI
  149. Lancaster, 1968, Evaluation of the MEDLARS Demand Search Service, 278
  150. Lancaster, 1968, Information Retrieval Systems Characteristics, Testing, and Evaluation
  151. Ledwith, 1992, “On the difficulties of applying the results of information retrieval research to aid in the searching of large scientific databases”, Information Processing & Management, 28, 451, 10.1016/0306-4573(92)90003-1 · DOI
  152. Léger, 1992, “Bootstrap technology and applications”, Technometrics, 34, 378, 10.1080/00401706.1992.10484950 · DOI
  153. Lesk, 1966, “SIG — The significance programs for testing the evaluation output”, Report ISR-12 to the National Science Foundation
  154. Lesk, 1997, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, 333
  155. Lesk, 1968, “Relevance assessments and retrieval system evaluation*”, Information Storage and Retrieval, 4, 343, 10.1016/0020-0271(68)90029-6 · DOI
  156. Lewis, 1996, The Fifth Text Retrieval Conference (TREC-5), 75
  157. Lin, 2006, “Building a reusable test collection for question answering”, Journal of the American Society for Information Science and Technology, 57, 851, 10.1002/asi.20348 · DOI
  158. Liu, 2009, Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, 31
  159. Liu, 2007, Proceedings of SIGIR 2007 Workshop on Learning to Rank for Information Retrieval, 3
  160. Magnini, 2005, Lecture notes in Computer Science, 3491, 371, 10.1007/11519645_38 · DOI
  161. Majumder, 2008, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 699, 10.1145/1390334.1390458 · DOI
  162. Manmatha, 2001, Proceedings of the 2fth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 267
  163. Maron, 1959, Data Systems Project Office
  164. Meister, 1967, Prepared under Contract No. NASw-1369 by Bunker-Ramo Corporation, Canoga Park, CA. (No. NASA CR-918)
  165. Melucci, 2007, “On rank correlation in information retrieval evaluation”, ACM SIGIR Forum, 41, 18, 10.1145/1273221.1273223 · DOI
  166. Minka, 2008, SIGIR Workshop on Learning to Rank for Information Retrieval, 48
  167. Mizzaro, 2007, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 479, 10.1145/1277741.1277824 · DOI
  168. Moffat, 2007, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 375, 10.1145/1277741.1277806 · DOI
  169. Moffat, 2008, “Rank-biased precision for measurement of retrieval effectiveness”, ACM Transactions on Information Systems, 27, Article No. 2, 10.1145/1416950.1416952 · DOI
  170. Mooers, 1959, Papers Presented at the Western Joint Computer Conference, 81
  171. Morita, 1994, Proceedings of the 17th Annual international ACM SIGIR Conference on Research and Development in Information Retrieval, 272
  172. Nuray, 2006, “Automatic ranking of information retrieval systems using data fusion”, Information Processing and Management, 42, 595, 10.1016/j.ipm.2005.03.023 · DOI
  173. Oard, 2004, Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 41, 10.1145/1008992.1009002 · DOI
  174. Ounis, 2006, Proceedings of the Fifteenth Text Retrieval Conference (TREC 2006), 17
  175. Over, 1997, Proceedings of the sixth Text Retrieval Conference (TREC-6), 500, 73
  176. Over, 2001, “The TREC interactive track: An annotated bibliography”, Information Processing and Management, 37, 369, 10.1016/S0306-4573(00)00053-4 · DOI
  177. Over, 2006, Proceedings of the TREC Video Retrieval Evaluation Notebook Papers
  178. Pearson, 1995, “Comparison of methods for searching protein sequence databases”, Protein Science: A Publication of the Protein Society, 4, 1145, 10.1002/pro.5560040613 · DOI
  179. Piwowarski, 2009, Proceedings of the Second ACM International Conference on Web Search and Data Mining, 162, 10.1145/1498759.1498823 · DOI
  180. Pollock, 1968, “Measures for the comparison of information retrieval systems”, American Documentation, 19, 387, 10.1002/asi.5090190406 · DOI
  181. Radlinski, 2008, Proceeding of the 17th ACM Conference on Information and Knowledge Management, 43, 10.1145/1458082.1458092 · DOI
  182. Rees, 1967, “A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching”
  183. Ritchie, 2006, Proceedings of the Main Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, 391
  184. Robertson, 1969, “The parametric description of retrieval tests: Part II: Overall measures”, Journal of Documentation, 25, 93, 10.1108/eb026466 · DOI
  185. Robertson, 1990, “On sample sizes for non-matched-pair IR experiments”, Information Processing and Management: An International Journal, 26, 739, 10.1016/0306-4573(90)90049-8 · DOI
  186. Robertson, 2000, “Salton award lecture on theoretical argument in information retrieval”, SIGIR Forum, 34, 1, 10.1145/373593.373597 · DOI
  187. Robertson, 2006, Proceedings of the 15th ACM International Conference on Information and Knowledge Management, 78, 10.1145/1183614.1183630 · DOI
  188. Robertson, 2008, “On the history of evaluation in IR”, Journal of Information Science, 34, 439, 10.1177/0165551507086989 · DOI
  189. Robertson, 2001, Proceedings of the Ninth Text REtrieval Conference (TREC-2001), 25
  190. Robertson, 2007, “On rank-based effectiveness measures and optimization”, Information Retrieval, 10, 321, 10.1007/s10791-007-9025-9 · DOI
  191. Roda, 2009, Working Notes for the CLEF 2009 Workshop
  192. Rorvig, 1990, “The simple scalability of documents”, Journal of the American Society for Information Science, 41, 590, 10.1002/(SICI)1097-4571(199012)41:8<590::AID-ASI5>3.0.CO;2-T · DOI
  193. Rose, 1996, Proceedings of the Fifth Text Retrieval Conference (TREC-5), 279
  194. Sakai, 2004, NTCIR-f Proceedings
  195. Sakai, 2006, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 525, 10.1145/1148170.1148261 · DOI
  196. Sakai, 2007, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 71, 10.1145/1277741.1277756 · DOI
  197. Sakai, 2007, “Evaluating information retrieval metrics based on bootstrap hypothesis tests”, Information and Media Technologies, 2, 1062
  198. Sakai, 2008, “On information retrieval metrics designed for evaluation with incomplete relevance assessments”, Information Retrieval, 11, 447, 10.1007/sl0791-008-9059-7 · DOI
  199. Salton, 1968, Automatic Information Organization and Retrieval
  200. Salton, 1971, The Smart Retrieval System. Experiments in Automatic Document Processing

200 of 281 shown

Cited by

Sorted by citations

See all 485

Metadata

Authors
Sanderson, M.
Affiliations
University of Sheffield (GB)
Corresponding author
Sanderson, Mark
ORCID
Sanderson, Mark
Venue
Foundations and Trends® in Information Retrieval
Publisher
Emerald
Published
2010-06-22
Volume
4
Issue
4
Pages
247-375
ISSN
  • 1554-0669 (ISSN-L)
  • 1554-0677
DOI
10.1561/1500000009
OpenAlex
W2022995284
Cited by
485
References
8
Citations by year
2026: 21 · 2025: 20 · 2024: 54 · 2023: 26 · 2022: 28 · 2021: 29 · 2020: 26 · 2019: 28 · 2018: 28 · 2017: 38 · 2016: 59 · 2015: 34 · 2014: 23 · 2013: 26 · 2012: 24
FWCI
38.24
Type
journal article
Language
en
Open access
closed
Locations
  • Foundations and Trends® in Information Retrieval (publishedVersion)
  • RMIT Research Repository (RMIT University Library) (submittedVersion)
  • Figshare (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Mathematics › Theoretical Computer Science › Algorithms and Data Compression
Keywords
test collections, information retrieval evaluation, evaluation metrics, statistical significance testing, query logs, assessment validity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review