The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Repeatable evaluation of search services in dynamic environments
Abstract
In dynamic environments, such as the World Wide Web, a changing document collection, query population, and set of search services demands frequent repetition of search effectiveness (relevance) evaluations. Reconstructing static test collections, such as in TREC, requires considerable human effort, as large collection sizes demand judgments deep into retrieved pools. In practice it is common to perform shallow evaluations over small numbers of live engines (often pairwise, engine A vs. engine B) without system pooling. Although these evaluations are not intended to construct reusable test collections, their utility depends on conclusions generalizing to the query population as a whole. We leverage the bootstrap estimate of the reproducibility probability of hypothesis tests in determining the query sample sizes required to ensure this, finding they are much larger than those required for static collections. We propose a semiautomatic evaluation framework to reduce this effort. We validate this framework against a manual evaluation of the top ten results of ten Web search engines across 896 queries in navigational and informational tasks. Augmenting manual judgments with pseudo-relevance judgments mined from Web taxonomies reduces both the chances of missing a correct pairwise conclusion, and those of finding an errant conclusion, by approximately 50%.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Nonparametric Statistical Methods
(1999) · Technometrics · doi:10.1080/00401706.1999.10485954 · Cited by 6,090
Simultaneous Statistical Inference
Miller, R. G. (1981) · Springer Series in Statistics · doi:10.1007/978-1-4613-8122-8 · Cited by 5,604
Testing Statistical Hypotheses
(2021) · Wiley Series in Probability and Statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,106
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
Testing Statistical Hypotheses.
Biggins, J. D., & Lehmann, E. L. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982206 · Cited by 966
Efficient crawling through URL ordering
Cho, J., Garcia-Molina, H., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00108-1 · Cited by 837
How are we searching the World Wide Web? A comparison of nine search engine transaction logs
Jansen, B. J., & Spink, A. (2006) · Information Processing & Management · doi:10.1016/j.ipm.2004.10.007 · Cited by 801
Cited by
Bobadilla, J., Ortega, F., Hernando, A., & Gutiérrez, A. (2013) · Knowledge-Based Systems · doi:10.1016/j.knosys.2013.03.012 · Cited by 3,158
A literature review and classification of recommender systems research
Park, D. H., Kim, H. K., Choi, I. Y., & Kim, J. K. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2012.02.038 · Cited by 662
More than words: Social networks’ text mining for consumer brand sentiments
Mostafa, M. M. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.01.019 · Cited by 623
Ghiassi, M., Skinner, J., & Zimbra, D. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.05.057 · Cited by 485
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485
“[135], in the context of web search, tested the combining of manual relevance judgments with judgments mined from a website taxonomy, such as DMOZ.”
MethodDetecting Hoaxes, Frauds, and Deception in Writing Style Online
Afroz, S., Brennan, M., & Greenstadt, R. (2012) · 2012 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2012.34 · Cited by 331
SAMAR: Subjectivity and sentiment analysis for Arabic social media
Abdul-Mageed, M., Diab, M., & Kübler, S. (2014) · Computer Speech & Language · doi:10.1016/j.csl.2013.03.001 · Cited by 260
Sentiment analysis of Chinese micro-blog text based on extended sentiment dictionary
Zhang, S., Wei, Z., Wang, Y., & Liao, T. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2017.09.048 · Cited by 255
Cross-generational analysis of predictive factors of addictive behavior in smartphone usage
Zhitomirsky-Geffet, M., & Blau, M. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.07.061 · Cited by 186
Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171
Breaking the Closed-World Assumption in Stylometric Authorship Attribution
Stolerman, A., Overdorf, R., Afroz, S., & Greenstadt, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-662-44952-3_13 · Cited by 28
Metadata
- Authors
- Jensen, E. C., Beitzel, S. M., Chowdhury, A., & Frieder, O.
- Venue
- ACM Transactions on Information Systems
- Published
- 2007-11
- Volume
- 26
- Issue
- 1
- Pages
- 1
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W2077620588
- Cited by
- 269
- References
- 56
- FWCI
- 31.06
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.