The DOI matches its Crossref record; 59 of the 62 references it cites were found in a registry or an index; none of the 63 sources its claims cite is checked for support yet.
Interactive Information Retrieval: An Introduction
Open accessAbstract
The paper introduces the research area of interactive information retrieval (IIR) from a historical point of view. Further, the focus here is on evaluation, because much research in IR deals with IR evaluation methodology due to the core research interest in IR performance, system interaction and satisfaction with retrieved information. In order to position IIR evaluation, the Cranfield model and the series of tests that led to the Cranfield model are outlined. Three iconic user-oriented studies and projects that all have contributed to how IIR is perceived and understood today are presented: The MEDLARS test, the Book House fiction retrieval system, and the OKAPI project. On this basis the call for alternative IIR evaluation approaches motivated by the three revolutions (the cognitive, the relevance, and the interactive revolutions) put forward by Robertson & Hancock-Beaulieu (1992) is presented. As a response to this call the ‘IIR evaluation model’ by Borlund (e.g., 2003a) is introduced. The objective of the IIR evaluation model is to facilitate IIR evaluation as close as possible to actual information searching and IR processes, though still in a relatively controlled evaluation environment, in which the test instrument of a simulated work task situation plays a central part.
Overview
CoChat Summary
TL;DR
This paper introduces interactive information retrieval (IIR) from a historical perspective, with a focus on evaluation. It traces the Cranfield model and three user-oriented projects, then presents the three revolutions and Borlund’s IIR evaluation model using simulated work task situations.
Key findings
- The paper positions IIR as the user-oriented counterpart to system-oriented Cranfield/TREC-style evaluation. It outlines the Cranfield tests: ASTIA-Uniterm used 15,000 documents and 98 requests, with disagreement over relevance; Cranfield-Uniterm used 200 documents and 40 artificial requests; Cranfield I used 18,000 documents and 1,200 requests; Cranfield II used 1,400 documents, 211 generated requests, and 33 indexing languages.
- Three iconic user-oriented studies are presented: the MEDLARS test evaluated an operational system with approximately 700,000 items and 302 real information requests, with relevance assessed as major, minor, or no value; the Book House fiction retrieval system developed fiction classification dimensions, a picture association thesaurus, and five search strategies (four implemented), and was evaluated over six months in 1988 with 2,850 search logs and 75 complete user observations; the OKAPI project built end-user test retrieval systems and participated in TREC.
- Robertson and Hancock-Beaulieu’s three revolutions—cognitive, relevance, and interactive—call for alternative IIR evaluation because the Cranfield model treats information needs as static and relevance as binary and topical.
- Borlund’s IIR evaluation model responds by involving potential users, applying dynamic and individual information needs, using multidimensional and dynamic relevance judgements, and employing simulated work task situations; the model has three parts and six requirements for simulated work task situations.
Methods
- Design: historical and conceptual introduction/review, not an empirical study; no new experiment is reported.
- Data or sample: draws on published IR evaluation literature and classic tests/projects, including ASTIA/Cranfield, MEDLARS, Book House, and OKAPI, plus theoretical work on the three revolutions and the IIR evaluation model.
- Main analysis: narrative synthesis and historical positioning; no statistical analysis is presented.
Limitations
- The paper states that more research is needed on IIR evaluation, especially the generalization of IIR evaluation results; it also notes Belkin’s point that the contradiction between realism and comparability/generalization has not yet been solved.
- The paper states that the major challenge is the design of realistic and applicable simulated work task situations.
- Evident from the text: as an introduction and historical overview, it does not present new empirical data or a systematic review, and its account is based on selected iconic studies and prior literature.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 4 Not valid · 51 Not checked · 68 Own results and interpretations
None of the sources cited for this claim have been checked yet.
In the paper
“The dynamic nature of relevance is demonstrated by, e.g., Bruce (1994), and Spink and colleagues (Spink, Greisdorf & Bateman, 1998) who found that users’ relevance criteria may change over the course of session time.”
What the sources say
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The two groups were unable to come to an agreement over which documents were relevant to which requests.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The Cranfield II test (Cleverdon & Keen, 1966; Cleverdon, Mills & Keen, 1966), was a test of 33 different types of indexing languages, which varied in terminologies and structures.”
What the sources say
- Not matchedcites Cleverdon & Keen, 1966
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites Cleverdon, Mills & Keen, 1966
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“Further, the main characteristic of the system is its metaphoric use of the familiar (Danish) public library of which the user is invited into a virtual space version.”
§ 3. USER-ORIENTED IR > 3.2. The Book House Fiction Retrieval System
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“much research in IR deals with IR evaluation methodology due to the core research interest in IR performance, system interaction, and satisfaction with retrieved information (Järvelin, 2011, p. 113).”
What the sources say
- Not matchedcites Järvelin, 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
23 more claims
Create a free account to see all 123 claims and to verify the 39 references still open.
- Probably not a substantial claim. 79
- It may state two separate claims as one. 39
Citations
Cites
Handbook of Human-Computer Interaction
(1988) · Elsevier eBooks · doi:10.1016/c2009-0-12113-x · Cited by 2,058
IR evaluation methods for retrieving highly relevant documents
Järvelin, K., & Kekäläinen, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130374 · Cited by 1,444
Cognitive Systems Engineering: An Integrative Living Laboratory Framework
(2017) · CRC Press eBooks · doi:10.4324/9781315155401 · Cited by 1,193
A BEHAVIOURAL APPROACH TO INFORMATION RETRIEVAL SYSTEM DESIGN
ELLIS, D. (1989) · Journal of Documentation · doi:10.1108/eb026843 · Cited by 962
ASK FOR INFORMATION RETRIEVAL: PART I. BACKGROUND AND THEORY
BELKIN, N. J., ODDY, R. N., & BROOKS, H. M. (1982) · Journal of Documentation · doi:10.1108/eb026722 · Cited by 944
TREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
Methods for Evaluating Interactive Information Retrieval Systems with Users
Kelly, D. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000012 · Cited by 633
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485 · cited 1×
“…model and the related series of tests have over the years been described and discussed by numerous scholars (e.g., Ellis, 1996; Robertson, 1981; Sanderson, 2010; Sharp, 1964; Sparck Jones, 1981b; Swanson, 1965) in addition to the original publications (Cleverdon, 1960; Cleverdon, 1962;…”
The concept of relevance in IR
Borlund, P. (2003) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10286 · Cited by 482
Self-citationA re-examination of relevance: toward a dynamic, situational definition∗
Schamber, L., Eisenberg, M. B., & Nilan, M. S. (1990) · Information Processing & Management · doi:10.1016/0306-4573(90)90050-c · Cited by 426
Research works printed but not linked: 5
- 1997, Annual Review of Information Science and Technology, 32, 3
- 2009, Foundations and Trends in Information Retrieval, 3, 1
- 2008, Annual Review of Information Science and Technology, 24, 43
- 1964, Journal of Documentation, 20, 170
- 2001, Annual Review of Information Science and Technology, 34, 53
Cited by
Information Retrieval meets Large Language Models: A strategic report from Chinese IR community
Ai, Q., Bai, T., Cao, Z., Chang, Y., Chen, J., Chen, Z., Cheng, Z., Dong, S., Dou, Z., Feng, F., et al. (2023) · AI Open · doi:10.1016/j.aiopen.2023.08.001 · Cited by 141
Information Retrieval: Advanced Topics and Techniques
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 88
Making Sense of Conflicting Science Information: Exploring Bias in the Search Engine Result Page
Novin, A., & Meyers, E. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020185 · Cited by 87
A Survey of Quantum Theory Inspired Approaches to Information Retrieval
Uprety, S., Gkoumas, D., & Song, D. (2020) · ACM Computing Surveys · doi:10.1145/3402179 · Cited by 52
Search Interface Design and Evaluation
Liu, C., Liu, Y., Liu, J., & Bierig, R. (2021) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000073 · Cited by 39
Metadata
- Authors
- Borlund, P.
- Affiliations
- University of Copenhagen (DK)
- Corresponding author
- Borlund, Pia
- ORCID
- Borlund, Pia
- Venue
- Journal of Information Science Theory and Practice
- Published
- 2013-09-30
- Volume
- 1
- Issue
- 3
- Pages
- 12-32
- ISSN
- 2287-4577 (ISSN-L)
- 2287-9099
- OpenAlex
- W2093162167
- Semantic Scholar
- 54dc822a6c7d2d45283cc77baab0e0dd2e4a8b7c
- Cited by
- 63
- References
- 62
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 20 · 2023: 6 · 2022: 5 · 2021: 6 · 2020: 5 · 2019: 4 · 2018: 1 · 2017: 5 · 2016: 1 · 2015: 1 · 2014: 2
- FWCI
- 2.18
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Information Science Theory and Practice (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- interactive information retrieval, user-centered evaluation, simulated work tasks, information seeking, cognitive revolution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interactive Information Retrieval: An Introduction. https://app.cochat.ai/papers/10.1633/jistap.2013.1.3.2
- The paper
- Borlund, P. (2013). Interactive Information Retrieval: An Introduction. Journal of Information Science Theory and Practice. https://doi.org/10.1633/jistap.2013.1.3.2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.