The DOI matches its Crossref record; 125 of the 135 references it cites were found in a registry or an index.
Task-Based Information Interaction Evaluation: The Viewpoint of Program Theory
Abstract
Evaluation is central in research and development of information retrieval (IR). In addition to designing and implementing new retrieval mechanisms, one must also show through rigorous evaluation that they are effective. A major focus in IR is IR mechanisms’ capability of ranking relevant documents optimally for the users, given a query. Searching for information in practice involves searchers, however, and is highly interactive. When human searchers have been incorporated in evaluation studies, the results have often suggested that better ranking does not necessarily lead to better search task, or work task, performance. Therefore, it is not clear which system or interface features should be developed to improve the effectiveness of human task performance. In the present article, we focus on the evaluation of task-based information interaction (TBII). We give special emphasis to learning tasks to discuss TBII in more concrete terms. Information interaction is here understood as behavioral and cognitive activities related to task planning, searching information items, selecting between them, working with them, and synthesizing and reporting. These five generic activities contribute to task performance and outcome and can be supported by information systems. In an attempt toward task-based evaluation, we introduce program theory as the evaluation framework. Such evaluation can investigate whether a program consisting of TBII activities and tools works and how it works and, further, provides a causal description of program (in)effectiveness. Our goal in the present article is to structure TBII on the basis of the five generic activities and consider the evaluation of each activity using the program theory framework. Finally, we combine these activity-based program theories in an overall evaluation framework for TBII. Such an evaluation is complex due to the large number of factors affecting information interaction. Instead of presenting tested program theories, we illustrate how the evaluation of TBII should be accomplished using the program theory framework in the evaluation of systems and behaviors, and their interactions, comprehensively in context.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry
Flavell, J. H. (1979) · American Psychologist · doi:10.1037/0003-066x.34.10.906 · Cited by 11,282 · cited 1×
“Task planning and reflective assessment also intertwines with the actor’s metacognitive process, and the task itself [Flavell 1979].”
BackgroundBloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956) · Cited by 9,342
A revision of bloom's taxonomy: an overview
Krathwohl, D. R. (2002) · Theory Into Practice · doi:10.1207/s15430421tip4104_2 · Cited by 8,132
Plans and Situated Actions: The Problem of Human-Machine Communication
Kantowitz, B., & Suchman, L. A. (1990) · The American Journal of Psychology · doi:10.2307/1423221 · Cited by 5,299
Evaluation: A Systematic Approach
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (1979) · Cited by 2,943
Inside the search process: Information seeking from the user's perspective
Kuhlthau, C. C. (1991) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199106)42:5<361::aid-asi6>3.0.co;2-# · Cited by 2,166
Evaluation: A Systematic Approach
Wallace, D. P. (2005) · Library & Information Science Research · doi:10.1016/j.lisr.2004.09.009 · Cited by 1,922
The design of browsing and berrypicking techniques for the online search interface
Bates, M. J. (1989) · Online Review · doi:10.1108/eb024320 · Cited by 1,757
Plans and situated actions: The problem of human-machine communication
Agre, P. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90078-e · Cited by 1,669
Plans and Situated Actions: The Problem of Human Machine Communication.
Woolgar, S., & Suchman, L. A. (1989) · Contemporary Sociology · doi:10.2307/2073874 · Cited by 1,604
Research works printed but not linked: 11
- P. Arvola and J. Kekäläinen . 2010. Simulating user interaction in result document browsing . In Workshop Proceedings of the 33rd Annual International ACM SIGIR Conference: Simulation of Interaction, Automated Evaluation of Interactive IR. ACM , New York, NY, 27--28. P. Arvola and J. Kekäläinen. 2010. Simulating user interaction in result document browsing. In Workshop Proceedings of the 33rd Annual International ACM SIGIR Conference: Simulation of Interaction, Automated Evaluation of Interactive IR. ACM, New York, NY, 27--28.
- N. J. Belkin . 1980 . Anomalous states of knowledge as a basis for information retrieval . Can. J. Inform. Lib. Sci. 5 (1980), 133 -- 143 . N. J. Belkin. 1980. Anomalous states of knowledge as a basis for information retrieval. Can. J. Inform. Lib. Sci. 5 (1980), 133--143.
- N. J. Belkin . 2010. On the evaluation of interactive information retrieval systems . In The Janus Faced Scholar. A Festschrift in Honor of Peter Ingwersen . Det Informationsvidenskabelige Akademi (Royal School of Library and Information Science , Copenhagen ); ISSI (International Society for Scientometrics and Informetrics), Copenhagen, Denmark, 13--22. N. J. Belkin. 2010. On the evaluation of interactive information retrieval systems. In The Janus Faced Scholar. A Festschrift in Honor of Peter Ingwersen. Det Informationsvidenskabelige Akademi (Royal School of Library and Information Science, Copenhagen); ISSI (International Society for Scientometrics and Informetrics), Copenhagen, Denmark, 13--22.
- 10.1002/asi.v56:10 · doi:10.1002/asi.v56:10
- 10.1037/0012-1649.25.5.765 · doi:10.1037/0012-1649.25.5.765
- C. Cool and N. J. Belkin . 2002. A classification of interactions with information. In Emerging Frameworks and Methods , Proceedings of the 4th International Conference on Conceptions of Library and Information Science (CoLIS4) . Greenwood Village, CO, Libraries Unlimited, 1--15. C. Cool and N. J. Belkin. 2002. A classification of interactions with information. In Emerging Frameworks and Methods, Proceedings of the 4th International Conference on Conceptions of Library and Information Science (CoLIS4). Greenwood Village, CO, Libraries Unlimited, 1--15.
- S. Goldman K. Lawless and F. Manning. 2013. Research and development of multiple source comprehension assessment. In Reading - From Words to Multiple Texts M. A. Britt S. R. Goldman and J.-F. Rouet (Eds.). Routledge New York NY 180--199. S. Goldman K. Lawless and F. Manning. 2013. Research and development of multiple source comprehension assessment. In Reading - From Words to Multiple Texts M. A. Britt S. R. Goldman and J.-F. Rouet (Eds.). Routledge New York NY 180--199.
- 10.1002/(sici)1097-4571(199601)47:1%3c50::aid-asi5%3e3.3.co;2-y · doi:10.1002/(sici)1097-4571(199601)47:1<50::aid-asi5>3.3.co;2-y
- K. Järvelin . 2011. Evaluation . In Interactive Information Seeking, Behaviour and Retrieval, I. Ruthven and D. Kelly (Eds.). Facet Publishing , London, UK , 113--138. K. Järvelin. 2011. Evaluation. In Interactive Information Seeking, Behaviour and Retrieval, I. Ruthven and D. Kelly (Eds.). Facet Publishing, London, UK, 113--138.
- 10.1002/aris.144.v42:1 · doi:10.1002/aris.144.v42:1
- T. Saracevic . 1996 b. Relevance reconsidered ’96. In Information science: Integration in Perspectives . Proceedings of the 2nd Conference on Conceptions of Library and Information Science. The Royal School of Librarianship, Copenhagen, DK, 201--218 . T. Saracevic. 1996b. Relevance reconsidered ’96. In Information science: Integration in Perspectives. Proceedings of the 2nd Conference on Conceptions of Library and Information Science. The Royal School of Librarianship, Copenhagen, DK, 201--218.
Cited by
Li, Y., & Liu, C. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2018.10.012 · Cited by 79
Effects of task complexity on online search behavior of adolescents
Walhout, J., Oomen, P., Jarodzka, H., & Brand‐Gruwel, S. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23782 · Cited by 62
Kruschwitz, U., & Hull, C. (2017) · now publishers, Inc. eBooks · doi:10.1561/9781680833058 · Cited by 58
Late, E., & Kumpulainen, S. (2021) · Journal of Documentation · doi:10.1108/jd-04-2021-0078 · Cited by 52
Self-citationvon Hoyer, J. F., Hoppe, A., Kammerer, Y., Otto, C. A., Pardi, G., Rokicki, M., Yu, R., Dietze, S., Ewerth, R., & Holtz, P. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.827748 · Cited by 45
Metadata
- Authors
- Järvelin, K., Vakkari, P., Arvola, P., Baskaya, F., Järvelin, A., Kekäläinen, J., Keskustalo, H., Kumpulainen, S., Saastamoinen, M., Savolainen, R., & Sormunen, E.
- Affiliations
- University of Tampere (FI)
- University of Tampere, Finland
- ORCID
- Venue
- ACM Transactions on Information Systems
- Published
- 2015-02-17
- Volume
- 33
- Issue
- 1
- Pages
- 1-30
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W2013726998
- Cited by
- 80
- References
- 135
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 6 · 2023: 8 · 2022: 7 · 2021: 14 · 2020: 4 · 2019: 7 · 2018: 13 · 2017: 6 · 2016: 2 · 2015: 1
- FWCI
- 8.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Information Systems (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Academy of Finland (#133021)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Keywords
- information retrieval evaluation, program theory, learning tasks, information seeking, task performance, causality assessment, information system effectiveness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Task-Based Information Interaction Evaluation: The Viewpoint of Program Theory. https://app.cochat.ai/papers/10.1145/2699660
- The paper
- Järvelin, K., Vakkari, P., Arvola, P., Baskaya, F., Järvelin, A., Kekäläinen, J., Keskustalo, H., Kumpulainen, S., Saastamoinen, M., Savolainen, R., & Sormunen, E. (2015). Task-Based Information Interaction Evaluation: The Viewpoint of Program Theory. ACM Transactions on Information Systems. https://doi.org/10.1145/2699660
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.