The DOI matches its Crossref record; 59 of the 62 references it cites were found in a registry or an index; none of the 63 sources its claims cite is checked for support yet.
Interactive Information Retrieval: An Introduction
Open accessAbstract
The paper introduces the research area of interactive information retrieval (IIR) from a historical point of view. Further, the focus here is on evaluation, because much research in IR deals with IR evaluation methodology due to the core research interest in IR performance, system interaction and satisfaction with retrieved information. In order to position IIR evaluation, the Cranfield model and the series of tests that led to the Cranfield model are outlined. Three iconic user-oriented studies and projects that all have contributed to how IIR is perceived and understood today are presented: The MEDLARS test, the Book House fiction retrieval system, and the OKAPI project. On this basis the call for alternative IIR evaluation approaches motivated by the three revolutions (the cognitive, the relevance, and the interactive revolutions) put forward by Robertson & Hancock-Beaulieu (1992) is presented. As a response to this call the ‘IIR evaluation model’ by Borlund (e.g., 2003a) is introduced. The objective of the IIR evaluation model is to facilitate IIR evaluation as close as possible to actual information searching and IR processes, though still in a relatively controlled evaluation environment, in which the test instrument of a simulated work task situation plays a central part.
Overview
CoChat Summary
TL;DR
This paper introduces interactive information retrieval (IIR) from a historical perspective, with a focus on evaluation. It traces the Cranfield model and three user-oriented projects, then presents the three revolutions and Borlund’s IIR evaluation model using simulated work task situations.
Key findings
- The paper positions IIR as the user-oriented counterpart to system-oriented Cranfield/TREC-style evaluation. It outlines the Cranfield tests: ASTIA-Uniterm used 15,000 documents and 98 requests, with disagreement over relevance; Cranfield-Uniterm used 200 documents and 40 artificial requests; Cranfield I used 18,000 documents and 1,200 requests; Cranfield II used 1,400 documents, 211 generated requests, and 33 indexing languages.
- Three iconic user-oriented studies are presented: the MEDLARS test evaluated an operational system with approximately 700,000 items and 302 real information requests, with relevance assessed as major, minor, or no value; the Book House fiction retrieval system developed fiction classification dimensions, a picture association thesaurus, and five search strategies (four implemented), and was evaluated over six months in 1988 with 2,850 search logs and 75 complete user observations; the OKAPI project built end-user test retrieval systems and participated in TREC.
- Robertson and Hancock-Beaulieu’s three revolutions—cognitive, relevance, and interactive—call for alternative IIR evaluation because the Cranfield model treats information needs as static and relevance as binary and topical.
- Borlund’s IIR evaluation model responds by involving potential users, applying dynamic and individual information needs, using multidimensional and dynamic relevance judgements, and employing simulated work task situations; the model has three parts and six requirements for simulated work task situations.
Methods
- Design: historical and conceptual introduction/review, not an empirical study; no new experiment is reported.
- Data or sample: draws on published IR evaluation literature and classic tests/projects, including ASTIA/Cranfield, MEDLARS, Book House, and OKAPI, plus theoretical work on the three revolutions and the IIR evaluation model.
- Main analysis: narrative synthesis and historical positioning; no statistical analysis is presented.
Limitations
- The paper states that more research is needed on IIR evaluation, especially the generalization of IIR evaluation results; it also notes Belkin’s point that the contradiction between realism and comparability/generalization has not yet been solved.
- The paper states that the major challenge is the design of realistic and applicable simulated work task situations.
- Evident from the text: as an introduction and historical overview, it does not present new empirical data or a systematic review, and its account is based on selected iconic studies and prior literature.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 4 Not valid · 51 Not checked · 68 Own results and interpretations
None of the sources cited for this claim have been checked yet.
In the paper
“The dynamic nature of relevance is demonstrated by, e.g., Bruce (1994), and Spink and colleagues (Spink, Greisdorf & Bateman, 1998) who found that users’ relevance criteria may change over the course of session time.”
What the sources say
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The two groups were unable to come to an agreement over which documents were relevant to which requests.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The Cranfield II test (Cleverdon & Keen, 1966; Cleverdon, Mills & Keen, 1966), was a test of 33 different types of indexing languages, which varied in terminologies and structures.”
What the sources say
- Not matchedcites Cleverdon & Keen, 1966
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites Cleverdon, Mills & Keen, 1966
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“Further, the main characteristic of the system is its metaphoric use of the familiar (Danish) public library of which the user is invited into a virtual space version.”
§ 3. USER-ORIENTED IR > 3.2. The Book House Fiction Retrieval System
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“much research in IR deals with IR evaluation methodology due to the core research interest in IR performance, system interaction, and satisfaction with retrieved information (Järvelin, 2011, p. 113).”
What the sources say
- Not matchedcites Järvelin, 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
23 more claims
Create a free account to see all 123 claims and to verify the 39 references still open.
- Probably not a substantial claim. 79
- It may state two separate claims as one. 39
Citations
Cites
Handbook of Human-Computer Interaction
(1988) · Elsevier eBooks · doi:10.1016/c2009-0-12113-x · Cited by 2,058
IR evaluation methods for retrieving highly relevant documents
Järvelin, K., & Kekäläinen, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130374 · Cited by 1,444
Cognitive Systems Engineering: An Integrative Living Laboratory Framework
(2017) · CRC Press eBooks · doi:10.4324/9781315155401 · Cited by 1,193
A BEHAVIOURAL APPROACH TO INFORMATION RETRIEVAL SYSTEM DESIGN
ELLIS, D. (1989) · Journal of Documentation · doi:10.1108/eb026843 · Cited by 962
ASK FOR INFORMATION RETRIEVAL: PART I. BACKGROUND AND THEORY
BELKIN, N. J., ODDY, R. N., & BROOKS, H. M. (1982) · Journal of Documentation · doi:10.1108/eb026722 · Cited by 944
TREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
Methods for Evaluating Interactive Information Retrieval Systems with Users
Kelly, D. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000012 · Cited by 633
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485 · cited 1×
“…model and the related series of tests have over the years been described and discussed by numerous scholars (e.g., Ellis, 1996; Robertson, 1981; Sanderson, 2010; Sharp, 1964; Sparck Jones, 1981b; Swanson, 1965) in addition to the original publications (Cleverdon, 1960; Cleverdon, 1962;…”
The concept of relevance in IR
Borlund, P. (2003) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10286 · Cited by 482
Self-citationA re-examination of relevance: toward a dynamic, situational definition∗
Schamber, L., Eisenberg, M. B., & Nilan, M. S. (1990) · Information Processing & Management · doi:10.1016/0306-4573(90)90050-c · Cited by 426
The IIR evaluation model: a framework for evaluation of interactive information retrieval systems
Borlund, P. (2003) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 403
Self-citationExperimental components for the evaluation of interactive information retrieval systems
Borlund, P. (2000) · Journal of Documentation · doi:10.1108/eum0000000007110 · Cited by 307
Self-citationInformation Retrieval Experiment
Jones, K. S. (1971) · Cited by 299
The development of a method for the evaluation of interactive information retrieval systems
Borlund, P., & Ingwersen, P. (1997) · Journal of Documentation · doi:10.1108/eum0000000007198 · Cited by 282
Self-citationSpink, A., Greisdorf, H., & Bateman, J. (1998) · Information Processing & Management · doi:10.1016/s0306-4573(98)00025-9 · Cited by 242
Annual Review of Information Science and Technology (ARIST)
Benito, C. (2003) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 231
The pragmatics of information retrieval experimentation, revisited
Tague-Sutcliffe, J. (1992) · Information Processing & Management · doi:10.1016/0306-4573(92)90005-k · Cited by 224
Overview of the Okapi projects
Robertson, S. E. (1997) · Journal of Documentation · doi:10.1108/eum0000000007186 · Cited by 182
Cleverdon, C. W., & Keen, M. (1966) · CERES (Cranfield University) · Cited by 172
The effect multiple query representations on information retrieval system performance
Belkin, N. J., Cool, C., Croft, W. B., & Callan, J. P. (1993) · Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '93 · doi:10.1145/160688.160760 · Cited by 170
Variations in relevance assessments and the measurement of retrieval effectiveness
Harter, S. P. (1996) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199601)47:1<37::aid-asi4>3.0.co;2-3 · Cited by 159
Human Information Interaction: An Ecological Approach to Information Behavior
Fidel, R. (2012) · Project Muse (Johns Hopkins University) · Cited by 118
Methodologies and Methods for User Behavioral Research.
Wang, P. (1999) · Cited by 111
Experiments on interfaces to support query expansion
Beaulieu, M. (1997) · Journal of Documentation · doi:10.1108/eum0000000007187 · Cited by 111
A new comparison between conventional indexing (MEDLARS) and automatic text processing (SMART)
Salton, G. (1972) · Journal of the American Society for Information Science · doi:10.1002/asi.4630230202 · Cited by 98
On the evaluation of IR systems
Robertson, S. E., & Hancock-Beaulieu, M. M. (1992) · Information Processing & Management · doi:10.1016/0306-4573(92)90004-j · Cited by 97
Some(what) grand challenges for information retrieval
Belkin, N. J. (2008) · ACM SIGIR Forum · doi:10.1145/1394251.1394261 · Cited by 91
Field-Weighted XML Retrieval Based on BM25
Lu, W. P., Robertson, S. E., & MacFarlane, A. (2006) · Lecture notes in computer science · doi:10.1007/978-3-540-34963-1_12 · Cited by 71
Interactive Information Retrieval in Digital Environments
Xie, I. (2008) · IGI Global eBooks · doi:10.4018/978-1-59904-240-4 · Cited by 68
Pejtersen, A. M., & AUSTIN, J. (1984) · Journal of Documentation · doi:10.1108/eb026755 · Cited by 63
Pejtersen, A. M., & AUSTIN, J. (1983) · Journal of Documentation · doi:10.1108/eb026750 · Cited by 62
Subjective versus Objective Relevance in Bibliographic Retrieval Systems
Swanson, D. R. (1986) · The Library Quarterly · doi:10.1086/601800 · Cited by 54
MEDLARS: report on the evaluation of its operating efficiency
Lancaster, F. W. (1969) · American Documentation · doi:10.1002/asi.4630200204 · Cited by 54
Interactive Information Seeking, Behaviour and Retrieval
Saračević, T., Kelly, D., Ruthven, I. G., Cool, C., Belkin, N. J., Wang, P., Toms, E. G., Fidel, R., Smucker, M. D., Rasmussen, E., et al. (2013) · Facet eBooks · doi:10.29085/9781856049740 · Cited by 53
A cognitive view of the situational dynamism of user-centered relevance estimation
Bruce, H. W. (1994) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199404)45:3<142::aid-asi4>3.0.co;2-6 · Cited by 49
Investigative Methods in Library and Information Science: An Introduction
Martyn, J. R. C., & Lancaster, F. W. (1981) · Virtual Defense Library (Ministerio de Defensa) · Cited by 48
Fuhr, N., Lalmas, M., Malik, S., & Kazai, G. (2006) · Springer eBooks · Cited by 40
Improving subject retrieval in online catalogues
Willett, P. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90014-3 · Cited by 38
Evaluating interactive systems in TREC
Beaulieu, M., Robertson, S., & Rasmussen, E. (1996) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199601)47:1<85::aid-asi8>3.0.co;2-z · Cited by 33
Seven years of work on the organization of materials in the special library
Gull, C. D. (1956) · American Documentation · doi:10.1002/asi.5090070408 · Cited by 32
Interactive searching and interface issues in the Okapi best match probabilistic retrieval system
Beaulieu, M., & Jones, S. (1998) · Interacting with Computers · doi:10.1016/s0953-5438(98)00008-3 · Cited by 30
Relevance feedback and query expansion
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809071.010 · Cited by 28
Aitchison, J., & Cleverdon, C. W. (1963) · CERES (Cranfield University) · Cited by 28
New model for multimedia interfaces to online public access catalogues
Pejtersen, A. M. (1992) · The Electronic Library · doi:10.1108/eb045194 · Cited by 27
The Evidence Underlying the Cranfield Results
Swanson, D. R. (1965) · The Library Quarterly · doi:10.1086/619296 · Cited by 26
Laboratory experiments with Okapi: participation in the TREC programme
Robertson, S. E., Walker, S., & Beaulieu, M. (1997) · Journal of Documentation · doi:10.1108/eum0000000007188 · Cited by 20
Interactive information retrieval: history and background
Cool, C., & Belkin, N. J. (2018) · Facet eBooks · doi:10.29085/9781856049740.003 · Cited by 19
Effectiveness testing of complex systems
Pejtersen, A. M., & Rasmussen, J. (1997) · Cited by 17
Interfaces for information retrieval
Wilson, M. L. (2018) · Facet eBooks · doi:10.29085/9781856049740.010 · Cited by 14
The Okapi online catalogue research projects
Walker, S. (1997) · Morgan Kaufmann Publishers Inc. eBooks · Cited by 13
THE EFFICIENCY OF SUBJECT CATALOGUES AND THE COST OF INFORMATION SEARCHES
THORNE, R. G. (1955) · Journal of Documentation · doi:10.1108/eb026215 · Cited by 11
XML-Structured Documents: Retrievable Units and Inheritance
Robertson, S. E., Lu, W. P., & MacFarlane, A. (2006) · Lecture notes in computer science · doi:10.1007/11766254_11 · Cited by 7
Lu, W. P., Robertson, S. E., & MacFarlane, A. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73888-6_6 · Cited by 5
(1996)
(1992)
(1994)
(1966) · CERES (Cranfield University)
(1962) · CERES (Cranfield University)
(1960) · CERES (Cranfield University)
Research works printed but not linked: 5
- 1997, Annual Review of Information Science and Technology, 32, 3
- 2009, Foundations and Trends in Information Retrieval, 3, 1
- 2008, Annual Review of Information Science and Technology, 24, 43
- 1964, Journal of Documentation, 20, 170
- 2001, Annual Review of Information Science and Technology, 34, 53
Cited by
Information Retrieval meets Large Language Models: A strategic report from Chinese IR community
Ai, Q., Bai, T., Cao, Z., Chang, Y., Chen, J., Chen, Z., Cheng, Z., Dong, S., Dou, Z., Feng, F., et al. (2023) · AI Open · doi:10.1016/j.aiopen.2023.08.001 · Cited by 141
Information Retrieval: Advanced Topics and Techniques
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 88
Making Sense of Conflicting Science Information: Exploring Bias in the Search Engine Result Page
Novin, A., & Meyers, E. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020185 · Cited by 87
A Survey of Quantum Theory Inspired Approaches to Information Retrieval
Uprety, S., Gkoumas, D., & Song, D. (2020) · ACM Computing Surveys · doi:10.1145/3402179 · Cited by 52
Search Interface Design and Evaluation
Liu, C., Liu, Y., Liu, J., & Bierig, R. (2021) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000073 · Cited by 39
Metadata
- Authors
- Borlund, P.
- Affiliations
- University of Copenhagen (DK)
- Corresponding author
- Borlund, Pia
- ORCID
- Borlund, Pia
- Venue
- Journal of Information Science Theory and Practice
- Published
- 2013-09-30
- Volume
- 1
- Issue
- 3
- Pages
- 12-32
- ISSN
- 2287-4577 (ISSN-L)
- 2287-9099
- OpenAlex
- W2093162167
- Semantic Scholar
- 54dc822a6c7d2d45283cc77baab0e0dd2e4a8b7c
- Cited by
- 63
- References
- 62
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 20 · 2023: 6 · 2022: 5 · 2021: 6 · 2020: 5 · 2019: 4 · 2018: 1 · 2017: 5 · 2016: 1 · 2015: 1 · 2014: 2
- FWCI
- 2.18
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Information Science Theory and Practice (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- interactive information retrieval, user-centered evaluation, simulated work tasks, information seeking, cognitive revolution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.