A systematic review of interactive information retrieval evaluation studies, 1967–2006
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
With the increasing number and diversity of search tools available, interest in the evaluation of search systems, particularly from a user perspective, has grown among researchers. More researchers are designing and evaluating interactive information retrieval (IIR) systems and beginning to innovate in evaluation methods. Maturation of a research specialty relies on the ability to replicate research, provide standards for measurement and analysis, and understand past endeavors. This article presents a historical overview of 40 years of IIR evaluation studies using the method of systematic review. A total of 2,791 journal and conference units were manually examined and 127 articles were selected for analysis in this study, based on predefined inclusion and exclusion criteria. These articles were systematically coded using features such as author, publication date, sources and references, and properties of the research method used in the articles, such as number of subjects, tasks, corpora, and measures. Results include data describing the growth of IIR studies over time, the most frequently occurring and cited authors and sources, and the most common types of corpora and measures used. An additional product of this research is a bibliography of IIR evaluation research that can be used by students, teachers, and those new to the area. To the authors' knowledge, this is the first historical, systematic characterization of the IIR evaluation literature, including the documentation of methods and measures used by researchers in this specialty.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Structure of Scientific Revolutions
Kuhn, T. S. (1996) · doi:10.7208/chicago/9780226458106.001.0001 · Cited by 7,281
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
The vocabulary problem in human-system communication
Furnas, G. W., Landauer, T. K., Gomez, L. M., & Dumais, S. T. (1987) · Communications of the ACM · doi:10.1145/32206.32212 · Cited by 1,504
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
The Intellectual and Social Organization Of the Sciences
Whitley, R. (2000) · doi:10.1093/oso/9780199240531.001.0001 · Cited by 1,076
Wolf, F. (1986) · doi:10.4135/9781412984980 · Cited by 784
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485
Do Sociological Theories Grow?
Wagner, D. G., & Berger, J. (1985) · American Journal of Sociology · doi:10.1086/228142 · Cited by 242
Cited by
O’Brien, H. L., Cairns, P., & Hall, M. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.01.004 · Cited by 1,014
O’Connor, A. M., Tsafnat, G., Thomas, J., Glasziou, P., Gilbert, S. B., & Hutton, B. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1062-0 · Cited by 167
Interactive IR User Study Design, Evaluation, and Reporting
Liu, J., & Shah, C. (2019) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02319-4 · Cited by 26
Metadata
- Authors
- Kelly, D., & Sugimoto, C. R.
- Venue
- Journal of the American Society for Information Science and Technology
- Publisher
- Wiley
- Published
- 2013-03
- Volume
- 64
- Issue
- 4
- Pages
- 745-770
- ISSN
- 1532-2882 (ISSN-L)
- 1532-2890
- OpenAlex
- W1999764269
- Cited by
- 109
- References
- 8
- FWCI
- 5.81
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A systematic review of interactive information retrieval evaluation studies, 1967–2006. https://app.cochat.ai/papers/10.1002/asi.22799
- The paper
- Kelly, D., & Sugimoto, C. R. (2013). A systematic review of interactive information retrieval evaluation studies, 1967–2006. Journal of the American Society for Information Science and Technology. https://doi.org/10.1002/asi.22799
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.