Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework
The DOI matches its Crossref record; 44 of the 51 references it cites were found in a registry or an index.
Abstract
One of the most challenging aspects of designing interactive information retrieval (IIR) experiments with users is the development of search tasks. We describe an evaluation of 20 search tasks that were designed for use in IIR experiments and developed using a cognitive complexity framework from educational theory. The search tasks represent five levels of cognitive complexity and four topical domains. The tasks were evaluated in the context of a laboratory IIR experiment with 48 participants. Behavioral and self-report data were used to characterize and understand differences among tasks. Results showed more cognitively complex tasks required significantly more search activity from participants (e.g., more queries, clicks, and time to complete). However, participants did not evaluate more cognitively complex tasks as more difficult and were equally satisfied with their performances across tasks. Our work makes four contributions: (1) it adds to what is known about the relationship among task, search behaviors and user experience; (2) it presents a framework for task creation and evaluation; (3) it provides tasks and questionnaires that can be reused by others and (4) it raises questions about findings and assumptions of many recent studies that only use behavioral signals from search logs as evidence for task difficulty and searcher satisfaction, as many of our results directly contradict these findings.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Search task complexity was shown to affect user behaviour [18].”
From A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity (2020) · cites this paper as [18]
In the citing paper
“Prior research in visual text search has shown that more complex tasks lead to greater levels of search interactivity, for example, through increased clicks, queries, and time on task [18].”
From A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity (2020) · cites this paper as [18]
Citations
Cites
(2001) · Cited by 13,930
Broder, A. (2002) · ACM SIGIR Forum · doi:10.1145/792550.792552 · Cited by 1,944 · cited 1×
“Broder’s [12] task classification was one of the first in the context of Web search and resulted in three categories of tasks: navigational, informational and transactional.”
BackgroundTask Complexity: A Review and Analysis
Campbell, D. J. (1988) · The Academy of Management Review · doi:10.5465/amr.1988.4306775 · Cited by 1,634 · cited 5×
“Using Campbell’s [15] conceptualization of objective task complexity, Remember tasks have the fewest solution paths, the fewest number of solutions or outcomes, the least amount of conflicting interdependencies between paths and the least amount of uncertainty regarding paths, while create tasks have the greatest numb…”
BackgroundMethodResultInfluentialTask complexity affects information seeking and use
Byström, K., & Järvelin, K. (1995) · Information Processing & Management · doi:10.1016/0306-4573(94)00041-z · Cited by 885 · cited 5×
“The tasks can also be related to Byström and Jӓrvelin’s [14] definition of task complexity in that the expected inputs, processes and outcomes become less certain as one moves from Remember to Create tasks.”
BackgroundMethodInfluentialMethods for Evaluating Interactive Information Retrieval Systems with Users
Kelly, D. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000012 · Cited by 633 · cited 2×
“Those studying IIR have used these collections in studies with research participants, although not without some challenges [23].”
BackgroundSelf-citationExploratory Search: Beyond the Query-Response Paradigm
White, R. W., & Roth, R. A. (2009) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00174ed1v01y200901icr003 · Cited by 632 · cited 1×
“White and Roth [40] provide a discussion of these types of tasks and situate them in the context of other task types.”
BackgroundThe IIR evaluation model: a framework for evaluation of interactive information retrieval systems
Borlund, P. (2003) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 403
Task‐based information searching
Vakkari, P. (2003) · Annual Review of Information Science and Technology · doi:10.1002/aris.1440370110 · Cited by 366 · cited 5×
“In one of the first reviews of task-based information seeking research, Vakkari [38] defines a task as an “activity to be performed in order to accomplish a goal” (p.”
BackgroundInfluentialA faceted approach to conceptualizing tasks in information seeking
Li, Y., & Belkin, N. J. (2008) · Information Processing & Management · doi:10.1016/j.ipm.2008.07.005 · Cited by 336 · cited 8×
“For example, Li and Belkin [28] found that objective task complexity was related to number of queries issued, mean query length, number of pages viewed, and number of sources consulted.”
BackgroundMethodResultInfluentialConceptual framework for tasks in information studies
Byström, K., & Hansen, P. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20197 · Cited by 220 · cited 1×
“Byström and Hansen [13] distinguish between a work task and a search task, one or more of which might be conducted to address the work task.”
Background
Research works printed but not linked: 4
- 10.1002/asi.v56:10 · doi:10.1002/asi.v56:10
- 10.5555/2655780.2655902 · doi:10.5555/2655780.2655902
- Dumais , S. T. , & Belkin , N. J. ( 2005 ). The TREC Interactive Tracks: Putting the user into search . In E. M. Voorhees & D . K. Harman (Eds.) TREC: Experiment and Evaluation in Information Retrieval ( pp. 123 -- 153 ), Cambridge, MA : MIT Press . Dumais, S. T., & Belkin, N. J. (2005). The TREC Interactive Tracks: Putting the user into search. In E. M. Voorhees & D. K. Harman (Eds.) TREC: Experiment and Evaluation in Information Retrieval (pp. 123--153), Cambridge, MA: MIT Press.
- 10.5555/2655780.2655906 · doi:10.5555/2655780.2655906
Cited by
Towards searching as a learning process: A review of current perspectives and future directions
Rieh, S. Y., Collins-Thompson, K., Hansen, P., & Lee, H.-J. (2016) · Journal of Information Science · doi:10.1177/0165551515615841 · Cited by 208
“[26] also used Anderson and Krathwhol’s [9] taxonomy when they evaluated a set of search tasks in terms of levels of cognitive complexity across multiple domains.”
MethodCognitive Biases in Search: A Review and Reflection of Cognitive Biases in Information Retrieval
Azzopardi, L. (2021) · Proceedings of the 2021 Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446023 · Cited by 176
“number of items, this may not necessarily be the case, and it may even increase the cost of browsing [6, 38].”
BackgroundIncorporating User Expectations and Behavior into the Measurement of Search Effectiveness
Moffat, A., Bailey, P., Scholer, F., & Thomas, P. (2017) · ACM Transactions on Information Systems · doi:10.1145/3052768 · Cited by 107
“” We also use a third level, Analyze; tasks at this level of the hierarchy involve “breaking material into constituent parts, determining how the parts relate to one another and to an overall structure or purpose through differentiating, organizing, and attributing” [Kelly et al. 2015].”
MethodPredicting User Satisfaction with Intelligent Assistants
Kiseleva, J., Williams, K., Hassan Awadallah, A., Crook, A. C., Zitouni, I., & Anastasakos, T. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911521 · Cited by 108
“Moreover, Kelly et al. [29] have provided evidence that the most complex search tasks were similar to the work [6] characterization of complex tasks with respect to having multiple interdependent parts that needed to be addressed separately.”
BackgroundSearching as Learning: Exploring Search Behavior and Learning Outcomes in Learning-related Tasks
Ghosh, S., Rath, M., & Shah, C. S. (2018) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3176349.3176386 · Cited by 102
“Several studies in LIS literature that have employed the taxonomy of learning to evaluate and classify various search tasks and that required different cognitive processes and demands from users, [9, 13, 14, 28, 32].”
BackgroundResultAn empirical study of interest, task complexity, and search behaviour on user engagement
O'Brien, H. L., Arguello, J., & Capra, R. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102226 · Cited by 81
Self-citationLiu, J. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102522 · Cited by 73
Liu, J., Mitsui, M., Belkin, N. J., & Shah, C. (2019) · Proceedings of the 2019 Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298922 · Cited by 54
“To decompose search task, prior studies mainly focused on a set of widely-discussed search task features (e.g., task complexity, task difficulty, task type) and their connections to the variations in Web search behavior (e.g., [1, 12]).”
MethodEffects of task complexity on online search behavior of adolescents
Walhout, J., Oomen, P., Jarodzka, H., & Brand‐Gruwel, S. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23782 · Cited by 62
“…search queries, more search keywords, and adapt the search query more often (Aula, Khan, & Guan, 2010; Bar-sky & Bar-Ilan, 2012; Borlund & Dreier, 2014; Kelly et al., 2015; Liu, Gwizdka, Liu, & Belkin, 2010; Saastamoinen et al., 2012; Singer, Norbisrath, & Lewandowski, 2012a; Toms et al., 2008).”
BackgroundMethodResultInfluentialIdentifying and Predicting the States of Complex Search Tasks
Liu, J., Sarkar, S., & Shah, C. S. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377976 · Cited by 57
“Many existing task models (e.g., [13, 31, 34]) have jointly supported a large body of IIR studies concerning identifying and responding to users’ tasks as static, overarching goals that motivates search actions.”
BackgroundMethodInfluentialResearchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for Deep Research
Rosset, C., Chung, H.-L., Qin, G., Chau, E. C., Feng, Z., Awadallah, A. H., Neville, J., & Rao, N. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730275 · Cited by 57
“More complicated questions ought to require more effort to answer [28].”
Impact of Response Latency on User Behaviour in Mobile Web Search
Arapakis, I., Park, S., & Pielot, M. (2021) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446038 · Cited by 54
“[19] that analyzes and categorizes the tasks by their complexity.”
BackgroundPrivacy Nudging in Search: Investigating Potential Impacts
Zimmerman, S. M., Thorpe, A., Fox, C., & Kruschwitz, U. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298952 · Cited by 53
“Multiple methods to measure cognitive load exist [15], however we only consider the time it takes (in seconds) to complete a task.”
BackgroundMethodInfluentialUrgo, K., & Arguello, J. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2021.102821 · Cited by 50
Self-citationPersonalization in text information retrieval: A survey
Liu, J., Liu, C., & Belkin, N. J. (2019) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24234 · Cited by 50
“…queries (Kim, 2006; Liu et al., 2010; Kelly, Arguello, Edwards, & Wu, 2015), query length and diversity (Kelly et al., 2015), number of SERP clicks (Kelly et al., 2015), number of visited documents (Kim, 2006; Gwizdka, 2008; Liu et al., 2010; Kelly et al., 2015), number of relevant/saved documents…”
BackgroundInfluentialLaypeople's source selection in online health information‐seeking process
Chi, Y., He, D., & Jeng, W. (2020) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24343 · Cited by 48
Zou, J., Aliannejadi, M., Kanoulas, E., Pera, M. S., & Liu, Y. (2022) · ACM Transactions on Information Systems · doi:10.1145/3524110 · Cited by 48
“Many studies in our community (e.g., [20, 21, 23, 29, 30, 37, 38]) used a limited number of experimental tasks to control multiple factors while maintaining the cost of the study at a reasonable level.”
The Effects of Manipulating Task Determinability on Search Behaviors and Outcomes
Capra, R., Arguello, J., O'Brien, H., Li, Y., & Choi, B. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210047 · Cited by 42
“[13] found that participants reported higher levels of engagement for tasks at higher levels of cognitive complexity, suggesting that the tasks used in their study did not exceed the threshold that leads to frustration and disengagement.”
BackgroundMethodInfluentialSelf-citationThe Role of Domain Knowledge in Search as Learning
O'Brien, H. L., Kampen, A., Cole, A. W., & Brennan, K. P. J. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377989 · Cited by 42
“In many SAL studies, search tasks vary by type but tend to be on different topics [5,7,11].”
BackgroundWhy People Search for Images using Web Search Engines
Xie, X., Liu, Y., de Rijke, M., He, J., Zhang, M., & Ma, S. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3159686 · Cited by 41
“Temporal information is a widely-used feedback feature in the setting of document relevance [4, 28], and it varies with di erent search tasks [22].”
BackgroundAnderson and Krathwohl's Two-Dimensional Taxonomy Applied to Task Creation and Learning Assessment
Urgo, K., Arguello, J., & Capra, R. (2019) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3341981.3344226 · Cited by 41
“To illustrate, most of the studies above [2, 8, 15, 21, 23] have used tasks developed by Kelly et al. [15], which vary across four topical domains and five cognitive process levels from A&K’s taxonomy.”
BackgroundMethodInfluentialSelf-citationExamining the impact of domain and cognitive complexity on query formulation and reformulation
Wildemuth, B. M., Kelly, D., Boettcher, E., Moore, E., & Dimitrova, G. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.01.009 · Cited by 40
Self-citationUnderstanding User Search Behavior Across Varying Cognitive Levels
Kalyani, R., & Gadiraju, U. (2019) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3342220.3343643 · Cited by 37
“The search tasks designed were either for all cognitive learning levels of Bloom’s taxonomy [19] or for a subset of cognitive learning levels [13, 20].”
BackgroundMethodInfluentialA Workflow Analysis Perspective to Scholarly Research Tasks
Koolen, M., Kumpulainen, S., & Estrada, L. M. M. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377969 · Cited by 36
“been used in information retrieval to construct and study search tasks and related searching behavior [22, 26].”
MethodUnderstanding How People use Search to Support their Everyday Creative Tasks
Zhang, Y., & Capra, R. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298936 · Cited by 36
“Based on these perspectives, we see opportunities for future IR research to address users’ needs in di erent creative stages (e.g., see work on “create” tasks by Kelly et al. [31]).”
BackgroundEngaged or Frustrated? Disambiguating Emotional State in Search
Edwards, A., & Kelly, D. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080818 · Cited by 35
“The tasks used in this study were based on tasks created and evaluated by Kelly, et al. [17].”
MethodSelf-citationFrummet, A., Elsweiler, D., & Ludwig, B. (2022) · ACM Transactions on Information Systems · doi:10.1145/3498330 · Cited by 34
“This often requires systems to utilise a memory function in which systems can reference past statements [67] in order to maintain an understanding of context [6, 51].”
Shah, C. S., White, R. W., Thomas, P., Mitra, B., Sarkar, S., & Belkin, N. J. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578288 · Cited by 34
Task complexity and difficulty in music information retrieval
Hu, X., & Kando, N. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23803 · Cited by 33
“It is well recognized in the education field and has been used in interactive IR (Arguello et al., 2012; Kelly et al., 2015).”
BackgroundResultInfluentialSong, X., Liu, C., & Zhang, Y. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102489 · Cited by 31
Understanding Ephemeral State of Relevance
Jiang, J., He, D., Kelly, D., & Allan, J. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020176 · Cited by 30
“We assigned different tasks to each group, which was to increase task diversity and reduce the concern regarding task parity in experiment design [17].”
MethodSelf-citationChoi, B., Ward, A. R., Li, Y., Arguello, J., & Capra, R. (2019) · ACM Transactions on Information Systems · doi:10.1145/3371707 · Cited by 30
“More closely related to our work, task complexity has also been studied from the perspective of cognitive complexity [12, 23, 25, 28, 49], which relates to the types (and variety) of mental processes required by the task.”
BackgroundMethodInfluentialSelf-citationA Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity
Vtyurina, A., Clarke, C. L. A., Law, E., Trippas, J. R., & Bota, H. (2020) · Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval · doi:10.1145/3409256.3409822 · Cited by 29
“Prior research in visual text search has shown that more complex tasks lead to greater levels of search interactivity, for example, through increased clicks, queries, and time on task [18].”
BackgroundInfluentialData sets for spoken conversational search
Trippas, J. R., & Thomas, P. (2019) · RMIT Research Repository (RMIT University Library) · doi:10.1000/611faketemporaryforimport · Cited by 29
Comparing In Situ and Multidimensional Relevance Judgments
Jiang, J., He, D., & Allan, J. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080840 · Cited by 29
“way we balance different types of tasks is similar to previous studies [22, 24, 30, 33, 36].”
ResultThe Effects of Working Memory, Perceptual Speed, and Inhibition in Aggregated Search
Arguello, J., & Choi, B. (2019) · ACM Transactions on Information Systems · doi:10.1145/3322128 · Cited by 29
“18Within the cognitive complexity framework [35], comparative tasks fall under analyze tasks, which have an intermediate”
BackgroundSelf-citationThe Effects of Learning Objectives on Searchers' Perceptions and Behaviors
Urgo, K., Arguello, J., & Capra, R. (2020) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3409256.3409815 · Cited by 29
“Prior work has studied search task complexity through the lens of cognitive complexity (leveraging A&K’s cognitive process dimension) [5, 6, 13, 15, 20, 21].”
BackgroundMethodInfluentialSelf-citationGómez, C., Yin, J., Huang, C., & Unberath, M. (2024) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1427463 · Cited by 28
Antecedents and learning outcomes of online news engagement
O'Brien, H. L. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23854 · Cited by 28
“Online searchers encounter a range of content and media during information retrieval, and although research is exploring search engine results pages (Kelly et al., 2015) and individual documents (Freund et al., 2016), eventually researchers will want to capture the dynamic movements of searchers…”
BackgroundInfluentialToward Cranfield-inspired reusability assessment in interactive information retrieval evaluation
Liu, J. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103007 · Cited by 27
What Affects Word Changes in Query Reformulation During a Task-based Search Session?
Jiang, J., & Ni, C. (2016) · Conference on Human Information Interaction and Retrieval · doi:10.1145/2854946.2854978 · Cited by 26
“Other ways of characterizing search tasks exist [16, 32], but we do not consider them in this study.”
BackgroundAsking Clarifying Questions: To benefit or to disturb users in Web search?
Zou, J., Sun, A., Long, C., Aliannejadi, M., & Kanoulas, E. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103176 · Cited by 26
“We use 𝑞 , 𝑐 , 𝐴 , 𝑆 , and 𝑌 as input factors, as the quality of 𝑞 , 𝑐 , 𝐴 , and 𝑆 contain cues on how necessary to show the CQ pane, while 𝑌 demonstrates the user preference to view the CQ pane.”
Liu, J., & Jung, Y. J. (2021) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446015 · Cited by 26
“Learning & Knowledge • Predefined learning goal or task labels [30, 44, 75];”
BackgroundEveryday Cross-session Search: How and Why Do People Search Across Multiple Sessions?
Li, Y., Capra, R., & Zhang, Y. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377970 · Cited by 25
“Prior work in interactive information retrieval has used Anderson and Krath-wohl’s taxonomy to consider task complexity [12, 18, 22].”
BackgroundMethodUnderstanding User Perceptions of Response Delays in Crowd-Powered Conversational Systems
Abbas, T., Gadiraju, U., Khan, V.-J., & Markopoulos, P. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555765 · Cited by 25
“Kelly et al. [57] has already developed a dataset of reusable search tasks based on the cognitive complexity framework.”
BackgroundMethodInfluentialUser Behaviour and Task Characteristics: A Field Study of Daily Information Behaviour
He, J., & Yılmaz, E. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020188 · Cited by 23
“(For each option, we provided to complete the following task? explanations following [16]).”
BackgroundMethodInfluentialSearch Engines vs. Symptom Checkers: A Comparison of their Effectiveness for Online Health Advice
Cross, S., Mourad, A., Zuccon, G., & Koopman, B. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450140 · Cited by 23
“, [11, 15, 21, 23]) which enables control over the experiment conditions and comparison of results across participants [1, 10].”
ResultUnderstanding the “Pathway” Towards a Searcher’s Learning Objective
Urgo, K., & Arguello, J. (2022) · ACM Transactions on Information Systems · doi:10.1145/3495222 · Cited by 23
“To this end, studies have leveraged the A&K taxonomy [5, 7, 21, 24, 44, 51].”
BackgroundResultSelf-citationThe impact of result diversification on search behaviour and performance
Maxwell, D., Azzopardi, L. A., & Moshfeghi, Y. (2019) · Information Retrieval · doi:10.1007/s10791-019-09353-0 · Cited by 23
“Indeed, Kelly et al. (2015) showed that more complex search tasks require a greater number of queries to provide sufficient coverage of the topic.”
BackgroundThe Effects of Search Task Determinability on Search Behavior
Capra, R., Arguello, J., & Zhang, Y. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-56608-5_9 · Cited by 22
“Studies have found that complex tasks are associated with higher levels of expected difficulty [9, 7, 3], experienced difficulty [2, 7, 9, 3], and search effort as indicated by measures derived from queries, clicks, bookmarks, and the task completion time [2, 7–9].”
BackgroundMethodSelf-citationThe Usefulness of Search Results: A Systematization of Types and Predictors
Vakkari, P. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377955 · Cited by 21
“For instance, [45] designed search tasks which are based on a taxonomy of learning goals with an aim to improve the validity and generalizability of search tasks and consequent search behavior.”
BackgroundTaxonomy of User Needs and ActionsPreprint
Shelby, R. M., Díaz, F., & Prabhakaran, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.06124 · Cited by 21
The Effects of Goal-setting on Learning Outcomes and Self-Regulated Learning Processes
Urgo, K., & Arguello, J. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638348 · Cited by 20
Self-citationA need for information on information needs.
Borlund, P., & Pharo, N. (2019) · Information Research · Cited by 20
Sanchiz, M., Amadieu, F., & Chevalier, A. (2020) · Human-computer interaction series · doi:10.1007/978-3-030-38825-6_5 · Cited by 21
Adapting Generative Information Retrieval Systems to Users, Tasks, and Scenarios
Trippas, J. R., Spina, D., & Scholer, F. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_4 · Cited by 19
The Role of the Task Topic in Web Search of Different Task Types
Hienert, D., Mitsui, M. R., Mayr, P., Shah, C. S., & Belkin, N. J. (2018) · doi:10.1145/3176349.3176382 · Cited by 19
“in their study [12] who used four domains (health, commerce, entertainment, science & technology) and different topics tailored to study participants in the sense of Borlund’s Simulated Work Task [3].”
BackgroundMethodInfluentialAn Eye Tracking Study of Web Search by People With and Without Dyslexia
Palani, S., Fourney, A., Williams, S., Larson, K., Spiridonova, I., & Morris, M. R. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401103 · Cited by 19
“Finally, we note that in contrast to prior work [33, 56], we did not fnd Remember tasks to be systematically easier, or harder, than Understand tasks – there were no signifcant diferences in task completion time, nor were there signifcant diferences in reported task load.”
ResultUser Experience in Dataset Search
Zhao, Y., Meroño-Peñuela, A., & Simperl, E. P. B. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-83845-3_7 · Cited by 18
“User-centric literature retrieval: from literature [11,15,8], they summary 10 user considerations: ’Relevance of Results’ focuses on query interpretation accuracy, context sensitivity, and result ordering; ’Ease of Use’ highlights intu-4 itive interfaces and straightforward navigation; ’Performance…”
BackgroundLiu, J., Wang, Y., Mandal, S., & Shah, C. S. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.02.011 · Cited by 18
Comparing Academic and Everyday-Life Information Seeking Behavior Among Millennial Students
Bartlett, J. C., Frissen, I., & Beheshti, J. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3378006 · Cited by 17
“Variation is seen with respect to task type, task difficulty, and task topic [21-25], as well as the domain of the topic and a person’s domain knowledge [26-30].”
BackgroundTo Blend or Not to Blend? Perceptual Speed, Visual Memory and Aggregated Search
Turpin, L., Kelly, D., & Arguello, J. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2914739 · Cited by 17
“The search tasks come from Kelly et al.’s [11] work; specifically, we used the four “Create” tasks.”
MethodSelf-citationHealth Cards for Consumer Health Search
Jimmy, J., Zuccon, G., Koopman, B., & Demartini, G. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331194 · Cited by 17
“While the arti � cial search scenario may not represent the participants’ information need, yet, we selected this approach as this is a common approach (e.g., [14, 18, 23, 27]) which enables control over the experiment conditions and comparison of results across participants [5, 13].”
BackgroundMethodThe Effects of Working Memory during Search Tasks of Varying Complexity
Choi, B., Capra, R., & Arguello, J. (2019) · Proceedings of the 2019 Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298948 · Cited by 16
Self-citationThe Effects of Domain and Search Expertise on Learning Outcomes in Digital Library Use
O'Brien, H. L., Cole, A. W., Kampen, A., & Brennan, K. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505761 · Cited by 16
“Typically, this is done to examine the effects of task complexity and the cognitive effort required to perform a task [28] on learning and search performance without introducing order effects.”
BackgroundMethodDecoy Effect in Search Interaction: Understanding User Behavior and Measuring System Vulnerability
Chen, N., Liu, J., Fang, H., Luo, Y., Sakai, T., & Wu, X.-M. (2025) · ACM Transactions on Information Systems · doi:10.1145/3708884 · Cited by 16
Making Time Fly: Using Fillers to Improve Perceived Latency in Crowd-Powered Conversational Systems
Abbas, T., Gadiraju, U., Khan, V.-J., & Markopoulos, P. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18935 · Cited by 16
“To control the complexity of user input, we provided custom keyboard options or quick replies that adhered to the cognitive complexity framework’s recommendations (Kelly et al. 2015).”
MethodInfluentialLiu, J., & Shah, C. S. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530926 · Cited by 16
“Built upon Bloom et al. [4]’s taxonomy, Kelly et al. [16] explored the cognitive aspect of task complexity and classified search tasks into different categories according to the level of complexity of associated learning goals.”
BackgroundUsing Principal Component Analysis to Better Understand Behavioral Measures and their Effects
Arguello, J., & Crescenzi, A. (2019) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3341981.3344222 · Cited by 16
“intellectual) [18, 22, 24], and task complexity [20].”
BackgroundSelf-citationWang, B., & Liu, J. (2024) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24935 · Cited by 15
“…IR tasks and are useful for encouraging multi-round search interactions and controlling the potential effects of contextual variables (such as differences in participants' topic familiarity and domain expertise) (Cole et al., 2015; Kelly et al., 2015; Li, 2009; Li & Belkin, 2008; Liu et al., 2010).”
Scientific Table Search Using Keyword QueriesPreprint
Gao, K. Y., & Callan, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03423 · Cited by 15
Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search Tasks
Liu, S., Hu, Y., Tian, Z., Jin, Z., Ruan, S., & Mao, J. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698406 · Cited by 14
“Traditional Information Retrieval (IR) systems consider the query as the representation of the user’s needs [24], but SAL emphasizes the learning behavior that occurs during the search process [63], giving rise to a focus on a specific type of search task called "learning-oriented search task".”
How does Interest in a Work Task Impact Search Behavior and Engagement?
Edwards, A., & Kelly, D. (2016) · Conference on Human Information Interaction and Retrieval · doi:10.1145/2854946.2855000 · Cited by 14
“The tasks used in this study were modeled after evaluate tasks we used in previous work [6].”
MethodSelf-citationBrazier, D. H., & Harvey, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-56608-5_21 · Cited by 13
“Tasks were distributed to participants using a Latin square design to account for task fatigue and potential learning effects [11].”
MethodFrom To-Do to Ta-Da: Transforming Task-Focused IR with Generative AI
Shah, C. S., & White, R. W. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730352 · Cited by 12
Social Cards Probably Provide For Better Understanding Of Web Archive Collections
Jones, S. M., Weigle, M. C., & Nelson, M. L. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3358039 · Cited by 12
“Kelly [25] applies Anderson and Krathwohl’s revision [6] of Bloom’s taxonomy [10] of learning objectives to the development of IR search tasks.”
MethodHow Misinformation Density Affects Health Information Search
Song, Q., & Jiang, J. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512141 · Cited by 12
“Previous research has found that as a task’s cognitive complexity level increases, users spend more time, issue more queries, click on more search results, and visit more URLs [23, 40, 50].”
BackgroundInvestigating Users' Time Perception during Web Search
Luo, C., Li, X., Liu, Y., Sakai, T., Zhang, F., Zhang, M., & Ma, S. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3020184 · Cited by 12
The Paradox of Personalization: Does Task Prediction Require Individualized Models?
Mitsui, M. R., Liu, J., & Shah, C. S. (2018) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3176349.3176887 · Cited by 11
“Similar relationships exist for task complexity rather than type [1, 9].”
BackgroundAn Analysis of Information Types and Cognitive Activities Involved in Cross-session Search
Li, Y., Ward, A. R., & Capra, R. (2021) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446044 · Cited by 11
“Researchers have found that tasks with a higher-level of cognitive complexity could trigger more search activities [8, 10].”
BackgroundMethodDosso, C., Tamine, L., Paubel, P., & Chevalier, A. (2023) · Behaviour and Information Technology · doi:10.1080/0144929x.2023.2242514 · Cited by 10
“Lastly, authors have also sought to pinpoint di ff erences in search behaviours between simple lookup and complex exploratory tasks (Athukorala et al. 2015; Jansen, Booth, and Smith 2009; Kelly et al. 2015) and between complex lookup and complex exploratory tasks (Jiang, He, and Allan 2014).”
InfluentialEngagement and Usability of Conversational Search – A Study of a Medical Resource Center Chatbot
Fergencs, T., & Meier, F. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-71292-1_26 · Cited by 10
“The reason behind the low scores could be that the protocol of the experiment gave little room for substantial immersion: the Understand type tasks we used did not require high-level cognitive processing, only identifying and compiling information [17].”
BackgroundMethodEngaged or Frustrated? Disambiguating engagement and frustration in search
Edwards, A. (2016) · ACM SIGIR Forum · doi:10.1145/2964797.2964813 · Cited by 10
Self-citationHealth Cards to Assist Decision Making in Consumer Health Search.
Jimmy, J., Zuccon, G., Demartini, G., & Koopman, B. (2019) · PubMed · Cited by 10
Tutorial on Task-Based Search and Assistance
Shah, C. S., & White, R. W. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401422 · Cited by 10
“Many existing task models (e.g., [11, 20, 22]) have investigated and identified searchers’ tasks as static and overarching goals that motivate search actions.”
BackgroundTime Limits, Information Search and the Use of Search Assistance
Crescenzi, A., Capra, R., & Arguello, J. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3022150 · Cited by 10
“The tasks were adapted from Kelly et al. [8] and were based on Anderson and Krath-wohl’s Taxonomy of Learning [1].”
MethodSelf-citationStrangers in a Strange Land: A Study of Second Language Speakers Searching for e-Services
Brazier, D. H., & Harvey, M. (2017) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3020165.3022133 · Cited by 10
“At the end of each task the participant was also required to complete a post-task questionnaire [1, 5, 6, 8, 9] with responses on a five point Likert scale where 1 is “Not at all confident” and 5 is “Very confident”.”
MethodOrganizing Found Information in Public Digital Library Search
Orin, K., & Hoeber, O. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716471 · Cited by 9
Proactive identification of query failure
Liu, J., & Shah, C. S. (2019) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.15 · Cited by 9
“Kelly, Arguello, Edwards, and Wu (2015) examined the cognitive complexity of task and adopted Bloom (1956) ’ s taxonomy of learning domains in characterizing different levels of task complexity.”
BackgroundEvaluating the effectiveness of Google, Parsijoo, Rismoon, and Yooz to retrieve Persian documents
Tazehkandi, M. Z., & Nowkarizi, M. (2020) · Library Hi Tech · doi:10.1108/lht-11-2019-0229 · Cited by 9
Modeling User Behavior for Vertical Search: Images, Apps and Products
Xie, X., Mao, J., Liu, Y., & de Rijke, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401423 · Cited by 8
“It has been shown that Bro-der’s taxonomy [3], where the search intent of general web search users is categorized into three classes (informational, transactional and navigational), is not applicable for vertical search [9].”
Exploring Online and Offline Search Behavior Based on the Varying Task Complexity
Rath, M., Ghosh, S., & Shah, C. S. (2018) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3176349.3176890 · Cited by 8
“Studies have emphasized information seeking research through a social and cultural context [7, 15, 22, 23] underlying the biological, physical, and anthropological layers, with respect to information seeking and searching [2], more like everyday information seeking in the context of “Way of Life”…”
BackgroundAn exploration of search-as-learning in digital archives of an online museum
Wu, I., Vakkari, P., & Huang, B. (2023) · Journal of Documentation · doi:10.1108/jd-03-2023-0056 · Cited by 7
A reference dependence approach to enhancing early prediction of session behavior and satisfaction
Brown, T., & Liu, J. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3533294 · Cited by 7
“The results echo the finding [10, 17] that incorporating user expectations can increase our early predictive capabilities as well as help us understand that users expect to complete their searches quickly and become rapidly dissatisfied once their expectation is exceeded.”
BackgroundLiu, Z., Wang, X., & Li, L. (2024) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3677389.3702595 · Cited by 7
“In terms of search task design, this study references the framework for designing interactive retrieval tasks based on cognitive complexity [4].”
Toward Connecting Speech Acts and Search Actions in Conversational Search Tasks
Ghosh, S., Ghosh, S., & Shah, C. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00027 · Cited by 9
“We developed the tasks following prior literature [44, 105], and tasks were designed to simulate naturalistic search behavior among the users.”
MethodInteractive Evaluation of Conversational Agents: Reflections on the Impact of Search Task Design
Dubiel, M., Halvey, M. J., Azzopardi, L. A., & Daronnat, S. (2020) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3409256.3409814 · Cited by 7
“Although insightful, the above studies ([5, 12, 14] focused only on a traditional search where queries are typed in a search box and results examined on the screen.”
BackgroundDEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn DialoguesPreprint
Li, Z., Park, D., Kiseleva, J., Kim, Y., & Lee, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01287 · Cited by 8
“Also, Kelly et al. (2015) provides evidence that the most complex search tasks were similar to the work (Campbell, 1988) characterization of complex tasks with respect to having multiple interdependent parts that needed to be addressed separately.”
BackgroundImproving the Effectiveness and Efficiency of Web-Based Search Tasks for Policy Workers
Schoegje, T., de Vries, A. P., Hardman, L., & Pieters, T. (2023) · Information · doi:10.3390/info14070371 · Cited by 6
“Because simulated tasks should be realistic and engaging [46,47], the tasks were adapted from real tasks that PWs have performed.”
BackgroundUrgo, K. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377947 · Cited by 6
“Kelly et al. [13] used all 20 search tasks from Wu et al. [16] in a subsequent study and confirmed those trends from Wu et al. [16].”
Method
Metadata
- Authors
- Kelly, D., Arguello, J., Edwards, A., & Wu, W.-C.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- ORCID
- Arguello, Jaime
- Venue
- Proceedings of the 2015 International Conference on The Theory of Information Retrieval
- Publisher
- ACM
- Published
- 2015-09-27
- Pages
- 101-110
- OpenAlex
- W2069049590
- Cited by
- 155
- References
- 51
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 14 · 2023: 8 · 2022: 16 · 2021: 17 · 2020: 19 · 2019: 27 · 2018: 12 · 2017: 19 · 2016: 8 · 2015: 1
- FWCI
- 11.59
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- interactive information retrieval, task difficulty, user experience, search behavior, task evaluation, user satisfaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.