The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Repeatable evaluation of search services in dynamic environments
Abstract
In dynamic environments, such as the World Wide Web, a changing document collection, query population, and set of search services demands frequent repetition of search effectiveness (relevance) evaluations. Reconstructing static test collections, such as in TREC, requires considerable human effort, as large collection sizes demand judgments deep into retrieved pools. In practice it is common to perform shallow evaluations over small numbers of live engines (often pairwise, engine A vs. engine B) without system pooling. Although these evaluations are not intended to construct reusable test collections, their utility depends on conclusions generalizing to the query population as a whole. We leverage the bootstrap estimate of the reproducibility probability of hypothesis tests in determining the query sample sizes required to ensure this, finding they are much larger than those required for static collections. We propose a semiautomatic evaluation framework to reduce this effort. We validate this framework against a manual evaluation of the top ten results of ten Web search engines across 896 queries in navigational and informational tasks. Augmenting manual judgments with pseudo-relevance judgments mined from Web taxonomies reduces both the chances of missing a correct pairwise conclusion, and those of finding an errant conclusion, by approximately 50%.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Nonparametric Statistical Methods
(1999) · Technometrics · doi:10.1080/00401706.1999.10485954 · Cited by 6,090
Simultaneous Statistical Inference
Miller, R. G. (1981) · Springer Series in Statistics · doi:10.1007/978-1-4613-8122-8 · Cited by 5,604
Testing Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
Testing Statistical Hypotheses.
Biggins, J. D., & Lehmann, E. L. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982206 · Cited by 966
Efficient crawling through URL ordering
Cho, J., Garcia-Molina, H., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00108-1 · Cited by 837
How are we searching the World Wide Web? A comparison of nine search engine transaction logs
Jansen, B. J., & Spink, A. (2006) · Information Processing & Management · doi:10.1016/j.ipm.2004.10.007 · Cited by 801
Pass, G., Chowdhury, A., & Torgeson, C. (2006) · Proceedings of the 1st international conference on Scalable information systems - InfoScale '06 · doi:10.1145/1146847.1146848 · Cited by 666
Self-citationHow reliable are the results of large-scale information retrieval experiments?
Zobel, J. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291014 · Cited by 579 · cited 1×
“…judgment depth is large enough, these collections are reusable, in that the relative effectiveness of runs from new engines over the same documents and queries can be evaluated simply by applying the existing judgments and assuming documents that are not judged are not relevant [Zobel 1998].”
BackgroundEvaluating Evaluation Measure Stability
Buckley, C., & Voorhees, E. M. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130373 · Cited by 559
Variations in relevance judgments and the measurement of retrieval effectiveness
Voorhees, E. M. (2000) · Information Processing & Management · doi:10.1016/s0306-4573(00)00010-8 · Cited by 504
The concept of relevance in IR
Borlund, P. (2003) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10286 · Cited by 482
Bootstrap tests: how many bootstraps?
Davidson, R., & MacKinnon, J. G. (2000) · Econometric Reviews · doi:10.1080/07474930008800459 · Cited by 460
What's new on the web?: the evolution of the web from a search engine perspective
Ntoulas, A., Cho, J., & Olston, C. (2004) · Proceedings of the 13th international conference on World Wide Web · doi:10.1145/988672.988674 · Cited by 452
Testing Statistical Hypotheses.
Mukerjee, H., & Lehmann, E. L. (1987) · Journal of the American Statistical Association · doi:10.2307/2289421 · Cited by 421
Information retrieval system evaluation: effort, sensitivity, and reliability
Sanderson, M., & Zobel, J. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076064 · Cited by 369
A comment on replication,P‐values and evidence
Goodman, S. N. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110705 · Cited by 309
Variations in relevance judgments and the measurement of retrieval effectiveness
Voorhees, E. M. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291017 · Cited by 303
Hourly analysis of a very large topically categorized web query log
Beitzel, S. M., Jensen, E. C., Chowdhury, A., Grossman, D., & Frieder, O. (2004) · Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1008992.1009048 · Cited by 303
Self-citationSpiegelhalter, D. J., & Freedman, L. S. (1986) · Statistics in Medicine · doi:10.1002/sim.4780050103 · Cited by 276
The effect of topic set size on retrieval experiment error
Voorhees, E. M., & Buckley, C. (2002) · Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/564376.564432 · Cited by 251
Minimal test collections for retrieval evaluation
Carterette, B., Allan, J., & Sitaraman, R. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148219 · Cited by 236
Efficient construction of large test collections
Cormack, G. V., Palmer, C. R., & Clarke, C. L. A. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291009 · Cited by 234
Ranking retrieval systems without relevance judgments
Soboroff, I., Nicholas, C., & Cahan, P. (2001) · Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/383952.383961 · Cited by 233
Evaluating evaluation metrics based on the bootstrap
Sakai, T. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148261 · Cited by 209
On bootstrap resampling and iteration
HALL, P., & MARTIN, M. A. (1988) · Biometrika · doi:10.1093/biomet/75.4.661 · Cited by 209
Evaluating strategies for similarity search on the web
Haveliwala, T. H., Gionis, A., Klein, D., & Indyk, P. (2002) · Proceedings of the 11th international conference on World Wide Web · doi:10.1145/511446.511502 · Cited by 187
A temporal comparison of AltaVista Web searching
Jansen, B. J., Spink, A., & Pedersen, J. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20145 · Cited by 176
Nonparametric Statistical Methods
Sijtsma, K., & Emons, W. H. M. (2010) · International Encyclopedia of Education · doi:10.1016/b978-0-08-044894-7.01353-1 · Cited by 172
A statistical method for system evaluation using incomplete judgments
Aslam, J. A., Pavlu, V., & Yilmaz, E. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148263 · Cited by 171
Proceedings of the 2007 ACM symposium on Applied computing
(2007) · doi:10.1145/1244002 · Cited by 163
Automatic ranking of information retrieval systems using data fusion
Nuray, R., & Can, F. (2006) · Information Processing & Management · doi:10.1016/j.ipm.2005.03.023 · Cited by 162
Statistical inference in retrieval effectiveness evaluation
Savoy, J. (1997) · Information Processing & Management · doi:10.1016/s0306-4573(97)00027-7 · Cited by 161
Results and challenges in Web search evaluation
Hawking, D., Craswell, N., Thistlewaite, P. B., & Harman, D. K. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00024-9 · Cited by 158
Peer review of statistics in medical research: the other problem
Bacchetti, P. (2002) · BMJ · doi:10.1136/bmj.324.7348.1271 · Cited by 157
A Machine Learning Architecture for Optimizing Web Search Engines
Boyan, J. A., & Freitag, D. (1999) · Cited by 130
A Comparative Study of Web Search Service Performance
Ding, W., & Marchionini, G. (1996) · Proceedings of the American Society for Information Science and Technology · Cited by 118
Statistical precision of information retrieval evaluation
Cormack, G. V., & Lynam, T. R. (2006) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1148170.1148262 · Cited by 115
A Unified Approach to Simultaneous Rank Test Procedures in the Unbalanced One-way Layout
Munzel, U., & Hothorn, L. A. (2001) · Biometrical Journal · doi:10.1002/1521-4036(200109)43:5<553::aid-bimj553>3.0.co;2-n · Cited by 111
Reproducibility probability in clinical trials
Shao, J., & Chow, S. (2002) · Statistics in Medicine · doi:10.1002/sim.1177 · Cited by 110
Forming test collections with no system pooling
Sanderson, M., & Joho, H. (2004) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1008992.1009001 · Cited by 97
Methods for ranking information retrieval systems without relevance judgments
Wu, S., & Crestani, F. (2003) · ACM Symposium on Applied Computing · doi:10.1145/952532.952693 · Cited by 83
Büttcher, S., Clarke, C. L. A., & Soboroff, I. M. (2006) · Cited by 74
Some perspectives on the evaluation of information retrieval systems
Tague‐Sutcliffe, J. (1996) · Journal of the American Society for Information Science · Cited by 61
A unified model for metasearch, pooling, and system evaluation
Aslam, J. A., Pavlu, V., & Savell, R. (2003) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/956863.956953 · Cited by 58
Estimating the Power of the Two-Sample Wilcoxon Test for Location Shift
Collings, B. J., & Hamilton, M. A. (1988) · Biometrics · doi:10.2307/2531596 · Cited by 57
Precision Evaluation of Search Engines
Shang, Y., & Li, L. (2002) · World Wide Web · doi:10.1023/a:1019679624079 · Cited by 49
Using titles and category names from editor-driven taxonomies for automatic evaluation
Beitzel, S. M., Jensen, E. C., Chowdhury, A., & Grossman, D. A. (2003) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/956863.956868 · Cited by 36
Self-citationDynamic test collections: measuring search effectiveness on the live web
Soboroff, I. M. (2006) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1148170.1148220 · Cited by 32
Approximating the power of Wilcoxon's rank-sum test against shift alternatives
Troendle, J. F. (1999) · Statistics in Medicine · Cited by 12
Repeatable evaluation of information retrieval effectiveness in dynamic environments
Frieder, O., & Jensen, E. C. (2006) · Cited by 7
Self-citationAutomatic Evaluation of Web Search Services
Chowdhury, A. (2005) · Advances in computers · doi:10.1016/s0065-2458(04)64001-0 · Cited by 4
Self-citation
Cited by
Bobadilla, J., Ortega, F., Hernando, A., & Gutiérrez, A. (2013) · Knowledge-Based Systems · doi:10.1016/j.knosys.2013.03.012 · Cited by 3,158
A literature review and classification of recommender systems research
Park, D. H., Kim, H. K., Choi, I. Y., & Kim, J. K. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2012.02.038 · Cited by 662
More than words: Social networks’ text mining for consumer brand sentiments
Mostafa, M. M. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.01.019 · Cited by 623
Ghiassi, M., Skinner, J., & Zimbra, D. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.05.057 · Cited by 485
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485
“[135], in the context of web search, tested the combining of manual relevance judgments with judgments mined from a website taxonomy, such as DMOZ.”
Method
Metadata
- Authors
- Jensen, E. C., Beitzel, S. M., Chowdhury, A., & Frieder, O.
- Venue
- ACM Transactions on Information Systems
- Published
- 2007-11
- Volume
- 26
- Issue
- 1
- Pages
- 1
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W2077620588
- Cited by
- 269
- References
- 56
- FWCI
- 31.06
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.