Building simulated queries for known-item topics: an analysis using six european languages
The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
There has been increased interest in the use of simulated queries for evaluation and estimation purposes in Information Retrieval. However, there are still many unaddressed issues regarding their usage and impact on evaluation because their quality, in terms of retrieval performance, is unlike real queries. In this paper, wefocus on methods for building simulated known-item topics and explore their quality against real known-item topics. Using existing generation models as our starting point, we explore factors which may influence the generation of the known-item topic. Informed by this detailed analysis (on six European languages) we propose a model with improved document and term selection properties, showing that simulated known-item topics can be generated that are comparable to real known-item topics. This is a significant step towards validating the potential usefulness of simulated queries: for evaluation purposes, and becausebuilding models of querying behavior provides a deeper insight into the querying process so that better retrieval mechanisms can be developed to support the user.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Theory of Modelling and Simulation.
Tocher, K. D., & Zeigler, B. P. (1977) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2345293 · Cited by 761
Cronen-Townsend, S., Zhou, Y., & Croft, W. B. (2002) · Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/564376.564429 · Cited by 663 · cited 1×
“Such query models are also useful for identifying structure and context within the query [6], estimating the difficultly of the query for performance prediction [8], automatically completing queries by predicting subsequent terms [3], and automatic query expansion[5].”
MethodInformation Retrieval as Statistical Translation
Berger, A., & Lafferty, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130371 · Cited by 621
Theory of Modelling and Simulation
Zeigler, B. P., & Oren, T. I. (1979) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1979.4310082 · Cited by 431 · cited 3×
“According to Zeigler [18], there are three kinds of validation that can be performed on a simulation; predictive, structural and replicative.”
BackgroundInfluentialInformation retrieval as statistical translation
Berger, A., & Lafferty, J. (1999) · Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/312624.312681 · Cited by 396 · cited 1×
“Berger and Lafferty [4] proposed a sophisticated language model (the translation model) which required significant”
BackgroundThe Philosophy of Information Retrieval Evaluation
Voorhees, E. M. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45691-0_34 · Cited by 389 · cited 1×
“Evaluation plays a central role in Information Retrieval because it enables the benchmarking and comparison of different retrieval techniques [17].”
BackgroundQuery-based sampling of text databases
Callan, J., & Connell, M. (2001) · ACM Transactions on Information Systems · doi:10.1145/382979.383040 · Cited by 368
Type less, find more: fast autocompletion search with a succinct index
Bast, H., & Weber, I. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148234 · Cited by 173 · cited 1×
“Such query models are also useful for identifying structure and context within the query [6], estimating the difficultly of the query for performance prediction [8], automatically completing queries by predicting subsequent terms [3], and automatic query expansion[5].”
MethodType Less, Find More: Fast Autocompletion Search with a Succinct Index
Bast, H., Weber, I. G., Efthimiadis, E. N., Dumais, S., Hawking, D., & Järvellin, K. (2006) · Max Planck Institute for Plasma Physics · Cited by 123
Monolingual Document Retrieval for European Languages
Hollink, V., Kamps, J., Monz, C., & de Rijke, M. (2003) · Information Retrieval · doi:10.1023/b:inrt.0000009439.19151.4c · Cited by 123 · cited 1×
“In the former case it could increase the size of the vocabulary, and it could do so in a way that drastically changes the statistics of terms [9].”
BackgroundUsing controlled query generation to evaluate blind relevance feedback algorithms
Jordan, C. A., Watters, C. R., & Gao, Q. (2006) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/1141753.1141818 · Cited by 54 · cited 5×
“Recently, there has been number of studies which have constructed a simulated test collection for evaluation and training purposes [1, 11, 12].”
BackgroundMethodInfluentialAutomatic construction of known-item finding test beds
Azzopardi, L. A., & de Rijke, M. (2006) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1148170.1148276 · Cited by 48 · cited 9×
“In [1], a probabilistic framework is presented for the generation of simulated queries for the task of known-item search.”
BackgroundMethodInfluentialSelf-citationProblems in the simulation of bibliographic retrieval systems
Tague, J. M., Nelson, M. J., & Wu, H. (1980) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.5555/636669.636684 · Cited by 47
Searching web databases by structuring keyword-based queries
Calado, P. P., da Silva, A. S., Vieira, R. C., Laender, A. H. F., & Ribeiro-Neto, B. A. (2002) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/584792.584801 · Cited by 26 · cited 1×
“Such query models are also useful for identifying structure and context within the query [6], estimating the difficultly of the query for performance prediction [8], automatically completing queries by predicting subsequent terms [3], and automatic query expansion[5].”
MethodEuroGOV: Engineering a Multilingual Web Corpus
Sigurbjörnsson, B., Kamps, J., & de Rijke, M. (2006) · Lecture notes in computer science · doi:10.1007/11878773_90 · Cited by 26 · cited 1×
“The document collections we used are from the EuroGOV corpus [13], a multilingual web corpus built from a crawl of European government-related sites, which contains over 3.”
MethodAutomatic query expansion based on divergence
Cai, D., van Rijsbergen, C. J., & Jose, J. M. (2001) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/502585.502656 · Cited by 23 · cited 2×
“Such query models are also useful for identifying structure and context within the query [6], estimating the difficultly of the query for performance prediction [8], automatically completing queries by predicting subsequent terms [3], and automatic query expansion[5].”
MethodRetrieving lightly annotated images using image similarities
Inoue, M., & Ueda, N. (2005) · ACM Symposium on Applied Computing · doi:10.1145/1066677.1066914 · Cited by 18 · cited 4×
“The focus of this study is on one particular type of simulated test collections built from the automatic generation of known-item topics [1, 11].”
BackgroundInfluentialSigurbjörnsson, B., Kamps, J., & de Rijke, M. (2006) · Lecture notes in computer science · doi:10.1007/11878773_89 · Cited by 18
Simulation of user judgments in bibliographic retrieval systems
Tague, J. M., & Nelson, M. J. (1981) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/511754.511764 · Cited by 17 · cited 6×
“[15] J. Tague, M. Nelson, and H. Wu.”
BackgroundMethodInfluentialSimulation of user judgments in bibliographic retrieval systems
Tague, J. M., & Nelson, M. J. (1981) · ACM SIGIR Forum · doi:10.1145/1013228.511764 · Cited by 15
Research works printed but not linked: 4
- Balog K., 2006, Working Notes CLEF 2006
- Inoue M., 2006, Proc. SIGIR 2006 Workshop on New Directions in Multilingual Information Access
- Sigurbjörnsson B., 2006, Springer Verlag
- Tague J., 1981, Proc. 3rd annual ACM conference on Research and development in information retrieval, 236
Cited by
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485
“[16] who examined means of creating known item topics for a range of collections and languages; they were inspired by earlier work from Tague and Nelson [252].”
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing
Zamani, H., Dehghani, M., Croft, W. B., Learned-Miller, E., & Kamps, J. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3271800 · Cited by 185
Online Evaluation for Information Retrieval
Hofmann, K., Li, L. C., & Radlinski, F. (2016) · now publishers, Inc. eBooks · doi:10.1561/1500000051 · Cited by 163
Neural information retrieval: at the end of the early years
Onal, K. D., Zhang, Y., Altingovde, I. S., Rahman, M. M., Karagoz, P., Braylan, A., Dang, B., Chang, H.-L., Kim, H., McNamara, Q., et al. (2018) · Information Retrieval Journal · doi:10.1007/s10791-017-9321-y · Cited by 136
“Or should we frame this more positively as an opportunity for research on generating training material or even simulation, as has previously been pursued for, e.g., learning to rank (Liu 2009), see, e.g., Azzopardi et al. (2007), Berendsen et al. (2013b)?”
BackgroundThe economics in interactive information retrieval
Azzopardi, L. A. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2009923 · Cited by 131
“In this paper, we adopt the approach taken in [12], where controlled queries are created, as opposed to probabilistically generating random queries as suggested in [3].”
BackgroundMethodSelf-citation
Metadata
- Authors
- Azzopardi, L., de Rijke, M., & Balog, K.
- Affiliations
- University of Glasgow (GB)
- University of Amsterdam, Amsterdam, Netherlands
- University of Amsterdam (NL)
- Venue
- Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
- Publisher
- ACM
- Published
- 2007-07-23
- Pages
- 455-462
- OpenAlex
- W2159665776
- Cited by
- 135
- References
- 20
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 7 · 2023: 8 · 2022: 7 · 2021: 8 · 2020: 4 · 2019: 6 · 2018: 9 · 2017: 10 · 2016: 10 · 2015: 10 · 2014: 8 · 2013: 6 · 2012: 8
- FWCI
- 4.36
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Mathematics › Theoretical Computer Science › Algorithms and Data Compression
- Keywords
- query generation, information retrieval evaluation, document selection, term selection, multilingual information retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.