The DOI matches its Crossref record; 125 of the 135 references it cites were found in a registry or an index.
Task-Based Information Interaction Evaluation: The Viewpoint of Program Theory
Abstract
Evaluation is central in research and development of information retrieval (IR). In addition to designing and implementing new retrieval mechanisms, one must also show through rigorous evaluation that they are effective. A major focus in IR is IR mechanisms’ capability of ranking relevant documents optimally for the users, given a query. Searching for information in practice involves searchers, however, and is highly interactive. When human searchers have been incorporated in evaluation studies, the results have often suggested that better ranking does not necessarily lead to better search task, or work task, performance. Therefore, it is not clear which system or interface features should be developed to improve the effectiveness of human task performance. In the present article, we focus on the evaluation of task-based information interaction (TBII). We give special emphasis to learning tasks to discuss TBII in more concrete terms. Information interaction is here understood as behavioral and cognitive activities related to task planning, searching information items, selecting between them, working with them, and synthesizing and reporting. These five generic activities contribute to task performance and outcome and can be supported by information systems. In an attempt toward task-based evaluation, we introduce program theory as the evaluation framework. Such evaluation can investigate whether a program consisting of TBII activities and tools works and how it works and, further, provides a causal description of program (in)effectiveness. Our goal in the present article is to structure TBII on the basis of the five generic activities and consider the evaluation of each activity using the program theory framework. Finally, we combine these activity-based program theories in an overall evaluation framework for TBII. Such an evaluation is complex due to the large number of factors affecting information interaction. Instead of presenting tested program theories, we illustrate how the evaluation of TBII should be accomplished using the program theory framework in the evaluation of systems and behaviors, and their interactions, comprehensively in context.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry
Flavell, J. H. (1979) · American Psychologist · doi:10.1037/0003-066x.34.10.906 · Cited by 11,282 · cited 1×
“Task planning and reflective assessment also intertwines with the actor’s metacognitive process, and the task itself [Flavell 1979].”
BackgroundBloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956) · Cited by 9,342
A revision of bloom's taxonomy: an overview
Krathwohl, D. R. (2002) · Theory Into Practice · doi:10.1207/s15430421tip4104_2 · Cited by 8,132
Plans and Situated Actions: The Problem of Human-Machine Communication
Kantowitz, B., & Suchman, L. A. (1990) · The American Journal of Psychology · doi:10.2307/1423221 · Cited by 5,299
Evaluation: A Systematic Approach
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (1979) · Cited by 2,943
Inside the search process: Information seeking from the user's perspective
Kuhlthau, C. C. (1991) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199106)42:5<361::aid-asi6>3.0.co;2-# · Cited by 2,166
Evaluation: A Systematic Approach
Wallace, D. P. (2005) · Library & Information Science Research · doi:10.1016/j.lisr.2004.09.009 · Cited by 1,922
The design of browsing and berrypicking techniques for the online search interface
Bates, M. J. (1989) · Online Review · doi:10.1108/eb024320 · Cited by 1,757
Plans and situated actions: The problem of human-machine communication
Agre, P. E. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90078-e · Cited by 1,669
Plans and Situated Actions: The Problem of Human Machine Communication.
Woolgar, S., & Suchman, L. A. (1989) · Contemporary Sociology · doi:10.2307/2073874 · Cited by 1,604
Research works printed but not linked: 11
- P. Arvola and J. Kekäläinen . 2010. Simulating user interaction in result document browsing . In Workshop Proceedings of the 33rd Annual International ACM SIGIR Conference: Simulation of Interaction, Automated Evaluation of Interactive IR. ACM , New York, NY, 27--28. P. Arvola and J. Kekäläinen. 2010. Simulating user interaction in result document browsing. In Workshop Proceedings of the 33rd Annual International ACM SIGIR Conference: Simulation of Interaction, Automated Evaluation of Interactive IR. ACM, New York, NY, 27--28.
- N. J. Belkin . 1980 . Anomalous states of knowledge as a basis for information retrieval . Can. J. Inform. Lib. Sci. 5 (1980), 133 -- 143 . N. J. Belkin. 1980. Anomalous states of knowledge as a basis for information retrieval. Can. J. Inform. Lib. Sci. 5 (1980), 133--143.
- N. J. Belkin . 2010. On the evaluation of interactive information retrieval systems . In The Janus Faced Scholar. A Festschrift in Honor of Peter Ingwersen . Det Informationsvidenskabelige Akademi (Royal School of Library and Information Science , Copenhagen ); ISSI (International Society for Scientometrics and Informetrics), Copenhagen, Denmark, 13--22. N. J. Belkin. 2010. On the evaluation of interactive information retrieval systems. In The Janus Faced Scholar. A Festschrift in Honor of Peter Ingwersen. Det Informationsvidenskabelige Akademi (Royal School of Library and Information Science, Copenhagen); ISSI (International Society for Scientometrics and Informetrics), Copenhagen, Denmark, 13--22.
- 10.1002/asi.v56:10 · doi:10.1002/asi.v56:10
- 10.1037/0012-1649.25.5.765 · doi:10.1037/0012-1649.25.5.765
- C. Cool and N. J. Belkin . 2002. A classification of interactions with information. In Emerging Frameworks and Methods , Proceedings of the 4th International Conference on Conceptions of Library and Information Science (CoLIS4) . Greenwood Village, CO, Libraries Unlimited, 1--15. C. Cool and N. J. Belkin. 2002. A classification of interactions with information. In Emerging Frameworks and Methods, Proceedings of the 4th International Conference on Conceptions of Library and Information Science (CoLIS4). Greenwood Village, CO, Libraries Unlimited, 1--15.
- S. Goldman K. Lawless and F. Manning. 2013. Research and development of multiple source comprehension assessment. In Reading - From Words to Multiple Texts M. A. Britt S. R. Goldman and J.-F. Rouet (Eds.). Routledge New York NY 180--199. S. Goldman K. Lawless and F. Manning. 2013. Research and development of multiple source comprehension assessment. In Reading - From Words to Multiple Texts M. A. Britt S. R. Goldman and J.-F. Rouet (Eds.). Routledge New York NY 180--199.
- 10.1002/(sici)1097-4571(199601)47:1%3c50::aid-asi5%3e3.3.co;2-y · doi:10.1002/(sici)1097-4571(199601)47:1<50::aid-asi5>3.3.co;2-y
- K. Järvelin . 2011. Evaluation . In Interactive Information Seeking, Behaviour and Retrieval, I. Ruthven and D. Kelly (Eds.). Facet Publishing , London, UK , 113--138. K. Järvelin. 2011. Evaluation. In Interactive Information Seeking, Behaviour and Retrieval, I. Ruthven and D. Kelly (Eds.). Facet Publishing, London, UK, 113--138.
- 10.1002/aris.144.v42:1 · doi:10.1002/aris.144.v42:1
- T. Saracevic . 1996 b. Relevance reconsidered ’96. In Information science: Integration in Perspectives . Proceedings of the 2nd Conference on Conceptions of Library and Information Science. The Royal School of Librarianship, Copenhagen, DK, 201--218 . T. Saracevic. 1996b. Relevance reconsidered ’96. In Information science: Integration in Perspectives. Proceedings of the 2nd Conference on Conceptions of Library and Information Science. The Royal School of Librarianship, Copenhagen, DK, 201--218.
Cited by
Li, Y., & Liu, C. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2018.10.012 · Cited by 79
Effects of task complexity on online search behavior of adolescents
Walhout, J., Oomen, P., Jarodzka, H., & Brand‐Gruwel, S. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23782 · Cited by 62
Kruschwitz, U., & Hull, C. (2017) · now publishers, Inc. eBooks · doi:10.1561/9781680833058 · Cited by 58
Late, E., & Kumpulainen, S. (2021) · Journal of Documentation · doi:10.1108/jd-04-2021-0078 · Cited by 52
Self-citationvon Hoyer, J. F., Hoppe, A., Kammerer, Y., Otto, C. A., Pardi, G., Rokicki, M., Yu, R., Dietze, S., Ewerth, R., & Holtz, P. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.827748 · Cited by 45
Vertical and horizontal relationships amongst task-based information needs
Byström, K., & Kumpulainen, S. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.102065 · Cited by 41
Self-citationSearch Interface Design and Evaluation
Liu, C., Liu, Y., Liu, J., & Bierig, R. (2021) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000073 · Cited by 39
A Workflow Analysis Perspective to Scholarly Research Tasks
Koolen, M., Kumpulainen, S., & Estrada, L. M. M. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377969 · Cited by 36
Self-citationShah, C. S., White, R. W., Thomas, P., Mitra, B., Sarkar, S., & Belkin, N. J. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578288 · Cited by 34
Tamine, L., & Daoud, M. (2018) · ACM Computing Surveys · doi:10.1145/3204940 · Cited by 33
Modeling the usefulness of search results as measured by information use
Vakkari, P., Völske, M., Potthast, M., Hagen, M., & Stein, B. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.02.001 · Cited by 33
Self-citationSong, X., Liu, C., & Zhang, Y. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102489 · Cited by 31
Kumpulainen, S., & Late, E. (2021) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24608 · Cited by 31
Self-citationEntity Recommendation for Everyday Digital Tasks
Jacucci, G., Daee, P., Vuong, T., Andolina, S., Klouche, K., Sjöberg, M., Ruotsalo, T., & Samuel Kaski, A. (2021) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3458919 · Cited by 27
Search task features in work tasks of varying types and complexity
Saastamoinen, M., & Järvelin, K. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23766 · Cited by 26
Self-citationKorkeamäki, L., & Kumpulainen, S. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298931 · Cited by 24
Self-citationThe Usefulness of Search Results: A Systematization of Types and Predictors
Vakkari, P. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377955 · Cited by 21
Self-citationRelationships between work task types, complexity and dwell time of information resources
Saastamoinen, M., & Järvelin, K. (2017) · Journal of Information Science · doi:10.1177/0165551516687726 · Cited by 21
Self-citationHistorical reasoning in authentic research tasks: Mapping cognitive and document spaces
Kumpulainen, S., Heikki Keskustalo, B., Zhang, B., & Stefanidis, K. (2019) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24216 · Cited by 19
Self-citationIn a Perfect World: Exploring the Desires and Realities for Digitized Historical Image Archives
Late, E., Ruotsalainen, H., & Kumpulainen, S. (2023) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.785 · Cited by 15
Self-citationA Performance-Based Test for Assessing Students’ Online Inquiry Competences in Schools
Sormunen, E., González‐Ibáñez, R., Kiili, C., Leppänen, P. H. T., Mikkilä‐Erdmann, M., Erdmann, N., & Escobar-Macaya, M. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-74334-9_69 · Cited by 15
Self-citationQueries in authentic work tasks: the effects of task type and complexity
Saastamoinen, M., & Järvelin, K. (2016) · Journal of Documentation · doi:10.1108/jd-09-2015-0119 · Cited by 14
Self-citationSormunen, E., Erdmann, N., Otieno, S., Mikkilä‐Erdmann, M., Laakkonen, E., Mikkonen, T., Hossain, M. A., González‐Ibáñez, R., Gatica, M. Q., Leppänen, P. H. T., et al. (2021) · Journal of Information Science · doi:10.1177/01655515211043708 · Cited by 13
Self-citationFrom To-Do to Ta-Da: Transforming Task-Focused IR with Generative AI
Shah, C. S., & White, R. W. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730352 · Cited by 12
Investigating Users' Time Perception during Web Search
Luo, C., Li, X., Liu, Y., Sakai, T., Zhang, F., Zhang, M., & Ma, S. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020184 · Cited by 12
Constructing and meta-evaluating state-aware evaluation metrics for interactive search systems
Markwald, M., Liu, J., & Yu, R. (2023) · Information Retrieval · doi:10.1007/s10791-023-09426-1 · Cited by 11
Untangling the Concept of Task in Information Seeking and Retrieval
Soufan, A., Ruthven, I. G., & Azzopardi, L. A. (2021) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3471158.3472259 · Cited by 11
Task Intelligence for Search and Recommendation
Shah, C. S., & White, R. W. (2021) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s01103ed1v01y202105icr074 · Cited by 11
Characterising users’ task completion process in learning-related tasks:A search pace model
Liu, C., Song, X., & Hansen, P. (2021) · Journal of Information Science · doi:10.1177/01655515211060527 · Cited by 10
Sendra, A., Late, E., & Kumpulainen, S. (2025) · Aslib Journal of Information Management · doi:10.1108/ajim-12-2024-0959 · Cited by 10
Self-citationSong, X., Liu, C., & Liu, H. (2018) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2018.14505501050 · Cited by 10
Saastamoinen, M. (2017) · Tampere University Institutional Repository (Tampere University) · Cited by 10
Self-citationA Blueprint of IR Evaluation Integrating Task and User Characteristics
Järvelin, K., & Sormunen, E. (2024) · ACM Transactions on Information Systems · doi:10.1145/3675162 · Cited by 9
Self-citationThe effects of topic familiarity on college students' learning search process
Chen, Y., Qiu, Y., Lin, H., & Zhao, Y. (2022) · Aslib Journal of Information Management · doi:10.1108/ajim-09-2021-0252 · Cited by 9
Information searching in cultural heritage archives: a user study
Borlund, P., Pharo, N., & Liu, Y. (2024) · Journal of Documentation · doi:10.1108/jd-06-2023-0120 · Cited by 8
Predicting Retrieval Success Based on Information Use for Writing Tasks
Vakkari, P., Völske, M., Potthast, M., Hagen, M., & Stein, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00066-0_14 · Cited by 8
Self-citationTask information types related to data gathering in media studies
Korkeamäki, L., Heikki Keskustalo, B., & Kumpulainen, S. (2022) · Journal of Documentation · doi:10.1108/jd-04-2022-0082 · Cited by 7
Self-citationPredicting essay quality from search and writing behavior
Vakkari, P., Völske, M., Potthast, M., Hagen, M., & Stein, B. (2021) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24451 · Cited by 7
Self-citationTask Intelligence for Search and Recommendation
Shah, C. S., & White, R. W. (2021) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-02326-2 · Cited by 7
Alamettälä, T., & Sormunen, E. (2020) · Tampere University Institutional Repository (Tampere University) · Cited by 7
Self-citationSearching Wartime Photograph Archive for Serious Leisure Purposes
Kumpulainen, S., & Ruotsalainen, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16802-4_7 · Cited by 7
Self-citationThe perceived outcomes of generative artificial intelligence in users’ everyday life
Han, Z., Zhao, H., Chen, J., & Yin, S. (2025) · Aslib Journal of Information Management · doi:10.1108/ajim-02-2025-0073 · Cited by 6
IRVILAB: Gamified Searching on Multilingual Wikipedia
Arvola, P., & Alamettälä, T. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531662 · Cited by 6
Self-citationGOLF: Goal-Oriented Long-term liFe tasks supported by human-AI collaboration
Wang, B. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657655 · Cited by 5
Salton Award Keynote: Information Interaction in Context
Järvelin, K. (2019) · ACM SIGIR Forum · doi:10.1145/3308774.3308782 · Cited by 4
Self-citationImages as data – modelling data interactions in social science and humanities research
Late, E., Matres, I., Sendra, A., & Kumpulainen, S. (2024) · Journal of Documentation · doi:10.1108/jd-08-2024-0195 · Cited by 4
Self-citationEffects of subject‐oriented visualization tools on search by novices and intermediates
Wu, I., & Vakkari, P. (2018) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24070 · Cited by 4
Self-citationlafmejani, S. N., Alipour‐Hafezi, M., Babalhavaeji, F., Hariri, N., & Fahimnia, F. (2021) · پژوهشنامه پردازش و مدیریت اطلاعات · doi:10.52547/jipm.37.1.135 · Cited by 4
Cooking with context: Leveraging context for procedural question answering
Frummet, A., Elsweiler, D., & Kruschwitz, U. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104212 · Cited by 3
Information Search Processes in Complex Tasks
Vakkari, P. (2018) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3176349.3176570 · Cited by 3
Self-citationSearch Interface Design and Evaluation
Liu, C., Liu, Y., Liu, J., & Bierig, R. (2021) · doi:10.1561/9781680839234 · Cited by 3
Enhancing Information Literacy Skills: A Game Design for Seeking Information and Making Queries
Arvola, P., & Alamettälä, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-22124-8_33 · Cited by 3
Self-citationA Probabilistic Model Toward How People Search to Build Outcomes
Zhao, Y., Bai, Y., Zhang, Y., Zhang, B., & Vakkari, P. (2023) · IEEE Access · doi:10.1109/access.2023.3252369 · Cited by 2
Self-citationTeacher Online Educational Resource Search in Education System Context
Vizgirda, V., McNeill, F., & Robertson, J. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716448 · Cited by 2
Supporting information use and task accomplishment: What system features do users like and expect?
Liu, J., & Li, Y. (2020) · Journal of Information Science · doi:10.1177/0165551520917100 · Cited by 2
Zhu, H., Liu, C., & Song, X. (2018) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2018.14505501194 · Cited by 2
Personalization needs extension towards task stages in collaborative research work tasks
Kumpulainen, S., Huurdeman, H., & Heikki Keskustalo, B. (2018) · UvA-DARE (University of Amsterdam) · doi:10.13140/rg.2.2.34679.55205 · Cited by 2
Self-citationTypes of domain and task‐solving information in media scholars' data interaction
Korkeamäki, L., Heikki Keskustalo, B., & Kumpulainen, S. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24863 · Cited by 2
Self-citationTalikka, M. (2018) · LUTPub (LUT University) · Cited by 2
Historiantutkijoiden tehtäväperusteinen informaatiovuorovaikutus digitaalisissa ympäristöissä
Korkeamäki, L. (2018) · Tampere University Institutional Repository (Tampere University) · Cited by 2
Open image collections – sources of research data?
Late, E., & Kumpulainen, S. (2022) · Informaatiotutkimus · doi:10.23978/inf.122591 · Cited by 1
Self-citationCollaborative image data interaction in social science and humanities research tasks
Late, E., Matres, I., Sendra, A., & Kumpulainen, S. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716451 · Cited by 1
Self-citationInformaatiosta tiedon vuoksi — Informaatiotutkimuksen ilmiöitä ja uomia
Kumpulainen, S., & Sihvonen, M. (2025) · doi:10.58181/vp9789523972834 · Cited by 1
Self-citationWang, A., Zhao, Y., & Ma, F. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70037 · Cited by 1
Analyzing Users’ Search Patterns to Explore Topic Knowledge from Aggregated Search Results
Huang, Y.-C., Ho, Y.-P., & Wu, I. (2017) · Communications in computer and information science · doi:10.1007/978-3-319-58750-9_61 · Cited by 1
Salton Award Keynote: Information Interaction in Context
Järvelin, K. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210230 · Cited by 1
Self-citationHeikki Keskustalo, B., Late, E., Korkeamäki, L., Kumpulainen, S., & Kettunen, K. (2021) · Informaatiotutkimus · doi:10.23978/inf.107890 · Cited by 1
Self-citationAssessing Information Search by Task Outcome
Vakkari, P. (2023) · SRELS Journal of Information Management · doi:10.17821/srels/2023/v60i1/170893 · Cited by 1
Self-citationEpisodic focus formulation in media scholars’ data interaction
Korkeamäki, L., Heikki Keskustalo, B., & Kumpulainen, S. (2024) · Information Research · doi:10.47989/ir293764 · Cited by 1
Self-citationLee, H. S. H. (2018) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/75724 · Cited by 1
Evaluation: Thinking Outside the (Search) Box
Clough, P. D. (2015) · Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE) · doi:10.1145/2824864.2824890
Kenttätutkimus akateemisten tutkijoiden informaatiovuorovaikutuksesta digitaalisissa ympäristöissä
Korkeamäki, L. (2018) · Informaatiotutkimus · doi:10.23978/inf.76079
Informaatiovuorovaikutus kontekstissa
Järvelin, K. (2019) · Informaatiotutkimus · doi:10.23978/inf.87043
Self-citationInformation Search Patterns in Complex Tasks
Vakkari, P. (2023) · SRELS Journal of Information Management · doi:10.17821/srels/2023/v60i1/170892
Self-citationExploring How Muddled Information Needs Influence Learning Outcomes
Wu, I., Yu, H.-K., Chang, F.-C., & Arvola, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92823-9_24
Self-citationKorkeamäki, L. (2024) · Tampere University Institutional Repository (Tampere University)
Metsänomistajien tiedonlähteet puukaupan alkuvaiheessa
Salmi, S. (2026) · Tampere University Institutional Repository (Tampere University)
Uses of generative AI and related information interactions in journalistic story creation process
Korkeamäki, L. (2026) · Information Research · doi:10.47989/ir31263111
Li, Y., Deng, J., Zhao, J., & Yim, J. (2026) · Applied Sciences · doi:10.3390/app16147254
Machine Translation or Human Translation: Which is the Better Servant?
Mikkola, P. (2023) · UEF eRepo (University of Eastern Finland)
Metadata
- Authors
- Järvelin, K., Vakkari, P., Arvola, P., Baskaya, F., Järvelin, A., Kekäläinen, J., Keskustalo, H., Kumpulainen, S., Saastamoinen, M., Savolainen, R., & Sormunen, E.
- Affiliations
- University of Tampere (FI)
- University of Tampere, Finland
- ORCID
- Venue
- ACM Transactions on Information Systems
- Published
- 2015-02-17
- Volume
- 33
- Issue
- 1
- Pages
- 1-30
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W2013726998
- Cited by
- 80
- References
- 135
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 6 · 2023: 8 · 2022: 7 · 2021: 14 · 2020: 4 · 2019: 7 · 2018: 13 · 2017: 6 · 2016: 2 · 2015: 1
- FWCI
- 8.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Information Systems (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Academy of Finland (#133021)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Library and Information Sciences › Library Science and Information Literacy
- Keywords
- information retrieval evaluation, program theory, learning tasks, information seeking, task performance, causality assessment, information system effectiveness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.