The DOI matches its Crossref record; 59 of the 62 references it cites were found in a registry or an index; none of the 63 sources its claims cite is checked for support yet.
Interactive Information Retrieval: An Introduction
Open accessAbstract
The paper introduces the research area of interactive information retrieval (IIR) from a historical point of view. Further, the focus here is on evaluation, because much research in IR deals with IR evaluation methodology due to the core research interest in IR performance, system interaction and satisfaction with retrieved information. In order to position IIR evaluation, the Cranfield model and the series of tests that led to the Cranfield model are outlined. Three iconic user-oriented studies and projects that all have contributed to how IIR is perceived and understood today are presented: The MEDLARS test, the Book House fiction retrieval system, and the OKAPI project. On this basis the call for alternative IIR evaluation approaches motivated by the three revolutions (the cognitive, the relevance, and the interactive revolutions) put forward by Robertson & Hancock-Beaulieu (1992) is presented. As a response to this call the ‘IIR evaluation model’ by Borlund (e.g., 2003a) is introduced. The objective of the IIR evaluation model is to facilitate IIR evaluation as close as possible to actual information searching and IR processes, though still in a relatively controlled evaluation environment, in which the test instrument of a simulated work task situation plays a central part.
Overview
CoChat Summary
TL;DR
This paper introduces interactive information retrieval (IIR) from a historical perspective, with a focus on evaluation. It traces the Cranfield model and three user-oriented projects, then presents the three revolutions and Borlund’s IIR evaluation model using simulated work task situations.
Key findings
- The paper positions IIR as the user-oriented counterpart to system-oriented Cranfield/TREC-style evaluation. It outlines the Cranfield tests: ASTIA-Uniterm used 15,000 documents and 98 requests, with disagreement over relevance; Cranfield-Uniterm used 200 documents and 40 artificial requests; Cranfield I used 18,000 documents and 1,200 requests; Cranfield II used 1,400 documents, 211 generated requests, and 33 indexing languages.
- Three iconic user-oriented studies are presented: the MEDLARS test evaluated an operational system with approximately 700,000 items and 302 real information requests, with relevance assessed as major, minor, or no value; the Book House fiction retrieval system developed fiction classification dimensions, a picture association thesaurus, and five search strategies (four implemented), and was evaluated over six months in 1988 with 2,850 search logs and 75 complete user observations; the OKAPI project built end-user test retrieval systems and participated in TREC.
- Robertson and Hancock-Beaulieu’s three revolutions—cognitive, relevance, and interactive—call for alternative IIR evaluation because the Cranfield model treats information needs as static and relevance as binary and topical.
- Borlund’s IIR evaluation model responds by involving potential users, applying dynamic and individual information needs, using multidimensional and dynamic relevance judgements, and employing simulated work task situations; the model has three parts and six requirements for simulated work task situations.
Methods
- Design: historical and conceptual introduction/review, not an empirical study; no new experiment is reported.
- Data or sample: draws on published IR evaluation literature and classic tests/projects, including ASTIA/Cranfield, MEDLARS, Book House, and OKAPI, plus theoretical work on the three revolutions and the IIR evaluation model.
- Main analysis: narrative synthesis and historical positioning; no statistical analysis is presented.
Limitations
- The paper states that more research is needed on IIR evaluation, especially the generalization of IIR evaluation results; it also notes Belkin’s point that the contradiction between realism and comparability/generalization has not yet been solved.
- The paper states that the major challenge is the design of realistic and applicable simulated work task situations.
- Evident from the text: as an introduction and historical overview, it does not present new empirical data or a systematic review, and its account is based on selected iconic studies and prior literature.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 4 Not valid · 51 Not checked · 68 Own results and interpretations
None of the sources cited for this claim have been checked yet.
In the paper
“The dynamic nature of relevance is demonstrated by, e.g., Bruce (1994), and Spink and colleagues (Spink, Greisdorf & Bateman, 1998) who found that users’ relevance criteria may change over the course of session time.”
What the sources say
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The two groups were unable to come to an agreement over which documents were relevant to which requests.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The Cranfield II test (Cleverdon & Keen, 1966; Cleverdon, Mills & Keen, 1966), was a test of 33 different types of indexing languages, which varied in terminologies and structures.”
What the sources say
- Not matchedcites Cleverdon & Keen, 1966
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites Cleverdon, Mills & Keen, 1966
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“Further, the main characteristic of the system is its metaphoric use of the familiar (Danish) public library of which the user is invited into a virtual space version.”
§ 3. USER-ORIENTED IR > 3.2. The Book House Fiction Retrieval System
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“much research in IR deals with IR evaluation methodology due to the core research interest in IR performance, system interaction, and satisfaction with retrieved information (Järvelin, 2011, p. 113).”
What the sources say
- Not matchedcites Järvelin, 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
23 more claims
Create a free account to see all 123 claims and to verify the 39 references still open.
- Probably not a substantial claim. 79
- It may state two separate claims as one. 39
Citations
Cites
Handbook of Human-Computer Interaction
(1988) · Elsevier eBooks · doi:10.1016/c2009-0-12113-x · Cited by 2,058
IR evaluation methods for retrieving highly relevant documents
Järvelin, K., & Kekäläinen, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130374 · Cited by 1,444
Cognitive Systems Engineering: An Integrative Living Laboratory Framework
(2017) · CRC Press eBooks · doi:10.4324/9781315155401 · Cited by 1,193
A BEHAVIOURAL APPROACH TO INFORMATION RETRIEVAL SYSTEM DESIGN
ELLIS, D. (1989) · Journal of Documentation · doi:10.1108/eb026843 · Cited by 962
ASK FOR INFORMATION RETRIEVAL: PART I. BACKGROUND AND THEORY
BELKIN, N. J., ODDY, R. N., & BROOKS, H. M. (1982) · Journal of Documentation · doi:10.1108/eb026722 · Cited by 944
TREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
Methods for Evaluating Interactive Information Retrieval Systems with Users
Kelly, D. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000012 · Cited by 633
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485 · cited 1×
“…model and the related series of tests have over the years been described and discussed by numerous scholars (e.g., Ellis, 1996; Robertson, 1981; Sanderson, 2010; Sharp, 1964; Sparck Jones, 1981b; Swanson, 1965) in addition to the original publications (Cleverdon, 1960; Cleverdon, 1962;…”
The concept of relevance in IR
Borlund, P. (2003) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10286 · Cited by 482
Self-citationA re-examination of relevance: toward a dynamic, situational definition∗
Schamber, L., Eisenberg, M. B., & Nilan, M. S. (1990) · Information Processing & Management · doi:10.1016/0306-4573(90)90050-c · Cited by 426
Research works printed but not linked: 5
- 1997, Annual Review of Information Science and Technology, 32, 3
- 2009, Foundations and Trends in Information Retrieval, 3, 1
- 2008, Annual Review of Information Science and Technology, 24, 43
- 1964, Journal of Documentation, 20, 170
- 2001, Annual Review of Information Science and Technology, 34, 53
Cited by
Information Retrieval meets Large Language Models: A strategic report from Chinese IR community
Ai, Q., Bai, T., Cao, Z., Chang, Y., Chen, J., Chen, Z., Cheng, Z., Dong, S., Dou, Z., Feng, F., et al. (2023) · AI Open · doi:10.1016/j.aiopen.2023.08.001 · Cited by 141
Information Retrieval: Advanced Topics and Techniques
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 88
Making Sense of Conflicting Science Information: Exploring Bias in the Search Engine Result Page
Novin, A., & Meyers, E. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020185 · Cited by 87
A Survey of Quantum Theory Inspired Approaches to Information Retrieval
Uprety, S., Gkoumas, D., & Song, D. (2020) · ACM Computing Surveys · doi:10.1145/3402179 · Cited by 52
Search Interface Design and Evaluation
Liu, C., Liu, Y., Liu, J., & Bierig, R. (2021) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000073 · Cited by 39
Memory-Augmented Dialogue Management for Task-Oriented Dialogue Systems
Zhang, Z., Huang, M., Zhao, Z., Ji, F., Chen, H., & Zhu, X. (2019) · ACM Transactions on Information Systems · doi:10.1145/3317612 · Cited by 37
Memory-augmented Dialogue Management for Task-oriented Dialogue Systems
Zhang, Z., Huang, M., Zhao, Z., Ji, F., Chen, H., & Zhu, X. (2018) · arXiv · doi:10.48550/arxiv.1805.00150 · Cited by 36
Harvey, M., & Brazier, D. H. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.102985 · Cited by 24
An Overview of Semantic Search Evaluation Initiatives
Elbedweihy, K., Wrigley, S. N., Clough, P. D., & Ciravegna, F. (2014) · Journal of Web Semantics · doi:10.1016/j.websem.2014.10.001 · Cited by 22
DocFlow: A Visual Analytics System for Question-Based Document Retrieval and Categorization
Qiu, R., Tu, Y., Wang, Y.-S., Yen, P., & Shen, H. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3219762 · Cited by 16
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 11
Engerer, V. (2019) · Journal of Information Science Theory and Practice · doi:10.1633/jistap.2019.7.2.1 · Cited by 10
Jiang, T., Li, W., & Liu, J. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.25020 · Cited by 9
Čudanov, M., Parlić, D., & Sofronijevic, A. M. (2014) · Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'21) · doi:10.1145/2669711.2669898 · Cited by 8
GenIR: Generative Visual Feedback for Mental Image Retrieval
Yang, D., Liu, M., Lo, C.-H., Zhang, Y., & Davis, J. (2025) · neural information processing systems · doi:10.52202/085713-4579 · Cited by 7
Engerer, V. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23783 · Cited by 7
Trippas, J. R. (2024) · ACM eBooks · doi:10.1145/3674127.3674135 · Cited by 6
Ferro, N., & Maistro, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674132 · Cited by 5
Gómez, E., Cuesta, H., Gkiokas, A., Gómez-Cañón, J. S., Porcaro, L., & Yesiler, F. (2024) · ACM eBooks · doi:10.1145/3674127.3702959 · Cited by 5
Verberne, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674141 · Cited by 5
Tonellotto, N. (2024) · ACM eBooks · doi:10.1145/3674127.3674130 · Cited by 5
Architecture Students' Search Behavior in Parametric Design
Dissaux, T., & Jancart, S. (2022) · eCAADe proceedings · doi:10.52842/conf.ecaade.2022.1.171 · Cited by 4
Dissaux, T., & Jancart, S. (2023) · Springer eBooks · doi:10.1007/978-3-031-20418-0_40 · Cited by 4
Engerer, V. P. (2020) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqaa008 · Cited by 4
Gomes, G. M. R., & Cendón, B. V. (2015) · Transinformação · doi:10.1590/0103-37862015000300009 · Cited by 4
lafmejani, S. N., Alipour‐Hafezi, M., Babalhavaeji, F., Hariri, N., & Fahimnia, F. (2021) · پژوهشنامه پردازش و مدیریت اطلاعات · doi:10.52547/jipm.37.1.135 · Cited by 4
Galuščáková, P., Oard, D. W., & Nair, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674136 · Cited by 3
Youth Research under the Microscope: A Conceptual Analysis of Youth Information Interaction Studies
Figueiredo, V., & Meyers, E. M. (2021) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.443 · Cited by 3
Search Interface Design and Evaluation
Liu, C., Liu, Y., Liu, J., & Bierig, R. (2021) · doi:10.1561/9781680839234 · Cited by 3
Information Professionals Going Beyond the Needful User in Digital Humanities Project Collaboration
Engerer, V. (2020) · Journal of Information Science Theory and Practice · doi:10.1633/jistap.2020.8.1.1 · Cited by 3
Temporal Extraction and Retrieval
Alonso, O., & Spaniol, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674137 · Cited by 2
Contextualizing the Cognition Crisis.
Novin, A. (2017) · Cited by 2
Privacy in Information Retrieval
Soboroff, I. M. (2024) · ACM eBooks · doi:10.1145/3674127.3674139 · Cited by 2
Performance Analysis of Data Leakage Detection System
Srivastava, V., Majumdar, A., & Jeyasekar, A. (2022) · International Conference for Emerging Technology (INCET) · doi:10.1109/incet54531.2022.9824111 · Cited by 2
Обогащённая сеть проблем как ядро метаданных электронной библиотеки
Gagarin, A. P., & Filimonov, I. A. (2021) · Sovremennye informacionnye tehnologii i IT-obrazovanie. · doi:10.25559/sitito.17.202104.860-870 · Cited by 2
Behnert, C. (2022) · Springer eBooks · doi:10.1007/978-3-658-37512-6 · Cited by 1
Unsupervised Automated Keyphrase Extraction Approaches: A Literature Review
Kahdum, A. A., & Al-Hameed, W. (2023) · International Conference on Information Technology, Applied Mathematics and Statistics (ICITAMS) · doi:10.1109/icitams57610.2023.10525639 · Cited by 1
eCommerce Search and Discovery
Kallumadi, S. T., & King, T. H. (2024) · ACM eBooks · doi:10.1145/3674127.3674140 · Cited by 1
Bast, H., Kalmbach, J., Klumpp, T., & Korzen, C. (2024) · ACM eBooks · doi:10.1145/3674127.3674134 · Cited by 1
DJUMANAZAROVA, Z., Djumabaeva, J. S., Yuldasheva, K., Adilmuratovna, J. R., Amanova, M., Atadjanova, M. I., Aripova, S., & Kilicheva, K. (2025) · Indian Journal of Information Sources and Services · doi:10.51983/ijiss-2025.ijiss.15.2.40 · Cited by 1
Streamlining NASA Research Retrieval with RAG-Inspired Semantic Search
Brooks, S., Ortega, I., & Sandidge, H. (2025) · International Conference on Environmental Systems (ICES) · doi:10.32865/2346/102677 · Cited by 1
Novin, A. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3022174 · Cited by 1
INTERACTIVE INFORMATION RETRIEVAL: AN OVERVIEW
Sonawane, C. S., Kulkarni, A. M., & Muthumari, P. (2020) · Lincoln (University of Nebraska) · Cited by 1
Hersh, W. (2020) · Health informatics · doi:10.1007/978-3-030-47686-1_8 · Cited by 1
Cruz-Vázquez, A., & Rojas, D. N. R. (2023) · Universitas · doi:10.17163/uni.n38.2023.07 · Cited by 1
Baeza-Yates, R. A., Murgai, L., & Dudy, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674138 · Cited by 1
The Prior eArchive as Virtual Research Environment:Towards Serendipity and Explorability
Sabir, F., & Engerer, V. P. (2019) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 1
Fremfinning og bruk av digitalt skapt materiale i arkivdepotene
Andresen, H. (2019) · NORA - Norwegian Open Research Archives · Cited by 1
Werner, K., Mandl, T., & Womser‐Hacker, C. (2016) · DOAJ (DOAJ: Directory of Open Access Journals)
A Realistic Approach Towards Users' Simulation.
Barifah, M. H., Landoni, M., & Crestani, F. (2017) · CLEF (Working Notes)
van Dijk, H. (2021) · Research Repository (Delft University of Technology)
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674128
Searching Personal Collections
Bendersky, M., Metzler, D., Najork, M., & Wang, X. (2024) · ACM eBooks · doi:10.1145/3674127.3674142
Query-driven Search Functionality
Li, L. (2024) · ACM eBooks · doi:10.1145/3674127.3674133
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702962
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674129
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702961
Hersh, W., & Roberts, K. E. (2024) · ACM eBooks · doi:10.1145/3674127.3702960
A Tool for Researching how AI Affects Information Seeking
Novin, A., & Towne, G. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716477
Østbye, S. E. (2018) · NORA - Norwegian Open Research Archives
Procjena utjecaja aktivnog učenja korištenjem proširene stvarnosti na znanje kadeta
Tuta, J. (2023) · University North Digital Repository (University North)
Interactive information retrieval on the web by elementary school children: a Delphi study
Ejei, F., Ghaebi, A., & Baradar, R. (2026) · Journal of Documentation · doi:10.1108/jd-09-2025-0260
Human Preference aligned Tabular SimilarityPreprint
Hoppe, F., Franz, A., Michaelis, M., Kleinemeier, L., & Göbel, U. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.24880
Metadata
- Authors
- Borlund, P.
- Affiliations
- University of Copenhagen (DK)
- Corresponding author
- Borlund, Pia
- ORCID
- Borlund, Pia
- Venue
- Journal of Information Science Theory and Practice
- Published
- 2013-09-30
- Volume
- 1
- Issue
- 3
- Pages
- 12-32
- ISSN
- 2287-4577 (ISSN-L)
- 2287-9099
- OpenAlex
- W2093162167
- Semantic Scholar
- 54dc822a6c7d2d45283cc77baab0e0dd2e4a8b7c
- Cited by
- 63
- References
- 62
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 20 · 2023: 6 · 2022: 5 · 2021: 6 · 2020: 5 · 2019: 4 · 2018: 1 · 2017: 5 · 2016: 1 · 2015: 1 · 2014: 2
- FWCI
- 2.18
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Information Science Theory and Practice (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- interactive information retrieval, user-centered evaluation, simulated work tasks, information seeking, cognitive revolution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.