Building simulated queries for known-item topics: an analysis using six european languages
The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
There has been increased interest in the use of simulated queries for evaluation and estimation purposes in Information Retrieval. However, there are still many unaddressed issues regarding their usage and impact on evaluation because their quality, in terms of retrieval performance, is unlike real queries. In this paper, wefocus on methods for building simulated known-item topics and explore their quality against real known-item topics. Using existing generation models as our starting point, we explore factors which may influence the generation of the known-item topic. Informed by this detailed analysis (on six European languages) we propose a model with improved document and term selection properties, showing that simulated known-item topics can be generated that are comparable to real known-item topics. This is a significant step towards validating the potential usefulness of simulated queries: for evaluation purposes, and becausebuilding models of querying behavior provides a deeper insight into the querying process so that better retrieval mechanisms can be developed to support the user.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Theory of Modelling and Simulation.
Tocher, K. D., & Zeigler, B. P. (1977) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2345293 · Cited by 761
Cronen-Townsend, S., Zhou, Y., & Croft, W. B. (2002) · Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/564376.564429 · Cited by 663
Information Retrieval as Statistical Translation
Berger, A., & Lafferty, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130371 · Cited by 621
Theory of Modelling and Simulation
Zeigler, B. P., & Oren, T. I. (1979) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1979.4310082 · Cited by 431
Information retrieval as statistical translation
Berger, A., & Lafferty, J. (1999) · Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/312624.312681 · Cited by 396
The Philosophy of Information Retrieval Evaluation
Voorhees, E. M. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45691-0_34 · Cited by 389
Query-based sampling of text databases
Callan, J., & Connell, M. (2001) · ACM Transactions on Information Systems · doi:10.1145/382979.383040 · Cited by 368
Type less, find more: fast autocompletion search with a succinct index
Bast, H., & Weber, I. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148234 · Cited by 173
Type Less, Find More: Fast Autocompletion Search with a Succinct Index
Bast, H., Weber, I. G., Efthimiadis, E. N., Dumais, S., Hawking, D., & Järvellin, K. (2006) · Max Planck Institute for Plasma Physics · Cited by 123
Monolingual Document Retrieval for European Languages
Hollink, V., Kamps, J., Monz, C., & de Rijke, M. (2003) · Information Retrieval · doi:10.1023/b:inrt.0000009439.19151.4c · Cited by 123
Research works printed but not linked: 4
- Balog K., 2006, Working Notes CLEF 2006
- Inoue M., 2006, Proc. SIGIR 2006 Workshop on New Directions in Multilingual Information Access
- Sigurbjörnsson B., 2006, Springer Verlag
- Tague J., 1981, Proc. 3rd annual ACM conference on Research and development in information retrieval, 236
Cited by
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485
“[16] who examined means of creating known item topics for a range of collections and languages; they were inspired by earlier work from Tague and Nelson [252].”
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing
Zamani, H., Dehghani, M., Croft, W. B., Learned-Miller, E., & Kamps, J. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3271800 · Cited by 185
Online Evaluation for Information Retrieval
Hofmann, K., Li, L. C., & Radlinski, F. (2016) · now publishers, Inc. eBooks · doi:10.1561/1500000051 · Cited by 163
Neural information retrieval: at the end of the early years
Onal, K. D., Zhang, Y., Altingovde, I. S., Rahman, M. M., Karagoz, P., Braylan, A., Dang, B., Chang, H.-L., Kim, H., McNamara, Q., et al. (2018) · Information Retrieval Journal · doi:10.1007/s10791-017-9321-y · Cited by 136
The economics in interactive information retrieval
Azzopardi, L. A. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2009923 · Cited by 131
Self-citation
Metadata
- Authors
- Azzopardi, L., de Rijke, M., & Balog, K.
- Affiliations
- University of Glasgow (GB)
- University of Amsterdam, Amsterdam, Netherlands
- University of Amsterdam (NL)
- Venue
- Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
- Publisher
- ACM
- Published
- 2007-07-23
- Pages
- 455-462
- OpenAlex
- W2159665776
- Cited by
- 135
- References
- 20
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 7 · 2023: 8 · 2022: 7 · 2021: 8 · 2020: 4 · 2019: 6 · 2018: 9 · 2017: 10 · 2016: 10 · 2015: 10 · 2014: 8 · 2013: 6 · 2012: 8
- FWCI
- 4.36
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Mathematics › Theoretical Computer Science › Algorithms and Data Compression
- Keywords
- query generation, information retrieval evaluation, document selection, term selection, multilingual information retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Building simulated queries for known-item topics: an analysis using six european languages. https://app.cochat.ai/papers/10.1145/1277741.1277820
- The paper
- Azzopardi, L., de Rijke, M., & Balog, K. (2007). Building simulated queries for known-item topics: an analysis using six european languages. Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. https://doi.org/10.1145/1277741.1277820
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.