Supporting Metacognition during Exploratory Search with the OrgBox
Open accessAbstract
Current search systems provide effective support to users engaged in fact-finding and look-up oriented tasks. However, they provide relatively little support for users engaged in exploratory search tasks that involve cognitive and metacognitive activities such as learning, synthesis, planning, and reflection. We conducted a within-subject user study (N=24) that investigated the effects of a novel knowledge organization tool called the OrgBox, designed to assist users with organizing and synthesizing information, and metacognitive activities. The OrgBox included features to allow users to drag-drop information they found through search into "boxes" that could be created, labelled, and re-arranged. Study participants completed two exploratory search tasks, one with the OrgBox, and one with the OrgDoc, a baseline tool that included features of a rich-text editor (e.g., formatting, bullets) for taking notes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Similar systems have been shown to improve users' planning and other metacognitive processes.
“Similar systems have been shown to improve users' planning and other metacognitive processes [38].”
From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [38]
Section 5.4 reports that OrgBox users perceived significantly greater support for monitoring and evaluation than users of the comparison tool. Planning support was rated higher, but the difference was not statistically significant (p=.059). The study measured perceived support, not demonstrated improvements in users’ metacognitive processes. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry
Flavell, J. H. (1979) · American Psychologist · doi:10.1037/0003-066x.34.10.906 · Cited by 11,282 · cited 1× · see citing works
“, “thinking about thinking” [17]) as searchers engage in the search and learning processes.”
BackgroundSmall sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552 · see citing works
Exploratory search: from finding to understanding
Marchionini, G. (2006) · Communications of the ACM · doi:10.1145/1121949.1121979 · Cited by 1,506 · see citing works
Information seeking in electronic environments
(1996) · Choice Reviews Online · doi:10.5860/choice.33-3366 · Cited by 1,487 · see citing works
Children's Metacognition About Reading: issues in Definition, Measurement, and Instruction
Jacobs, J. E., & Paris, S. (1987) · Educational Psychologist · doi:10.1207/s15326985ep2203 · Cited by 988 · see citing works
Information Seeking in Electronic Environments
Marchionini, G. (1995) · doi:10.1017/cbo9780511626388 · Cited by 833 · see citing works
Sense‐making theory and practice: an overview of user interests in knowledge seeking and use
Dervin, B. L. (1998) · Journal of Knowledge Management · doi:10.1108/13673279810249369 · Cited by 784 · see citing works
The cost structure of sensemaking
Russell, D. M., Stefik, M., Pirolli, P. L. T., & Card, S. K. (1993) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/169059.169209 · Cited by 738 · see citing works
Exploratory Search: Beyond the Query-Response Paradigm
White, R. W., & Roth, R. A. (2009) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00174ed1v01y200901icr003 · Cited by 632 · see citing works
Does adaptive scaffolding facilitate students’ ability to regulate their learning with hypermedia?
Azevedo, R., Cromley, J. G., & Seibert, D. C. (2003) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2003.09.002 · Cited by 474 · see citing works
Ge, X., & Land, S. M. (2004) · Educational Technology Research and Development · doi:10.1007/bf02504836 · Cited by 429 · see citing works
Teaching and learning in digital environments: The resurgence of resource-based learning
Hill, J. R., & Hannafin, M. J. (2001) · Educational Technology Research and Development · doi:10.1007/bf02504914 · Cited by 424 · see citing works
The IIR evaluation model:a framework for evaluation of interactive information retrieval systems
Borlund, P. (2003) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 403 · see citing works
Children's Metacognition About Reading: issues in Definition, Measurement, and Instruction
Jacobs, J. E., & Paris, S. G. (1987) · Educational Psychologist · doi:10.1080/00461520.1987.9653052 · Cited by 396 · see citing works
A descriptive model of information problem solving while using internet
Brand‐Gruwel, S., Wopereis, I. G. J. H., & Walraven, A. (2009) · Computers & Education · doi:10.1016/j.compedu.2009.06.004 · Cited by 340 · see citing works
Research Frontiers in Information Retrieval
Culpepper, J. S., Díaz, F., & Smucker, M. D. (2018) · ACM SIGIR Forum · doi:10.1145/3274784.3274788 · Cited by 182 · see citing works
Morris, D., Morris, M. R., & Venolia, G. D. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357242 · Cited by 143 · see citing works
Effects of the metacognitive computer-tool met.a.ware on the web search of laypersons
Stadtler, M., & Bromme, R. (2007) · Computers in Human Behavior · doi:10.1016/j.chb.2007.01.023 · Cited by 141 · see citing works
Introduction to this Special Issue on Sensemaking
Pirolli, P. L. T., & Russell, D. M. (2011) · Human-Computer Interaction · doi:10.1080/07370024.2011.556557 · Cited by 139 · see citing works
McCardle, L., & Hadwin, A. F. (2015) · Metacognition and Learning · doi:10.1007/s11409-014-9132-0 · Cited by 133 · see citing works
Effects of Metacognitive Training Embedded in Cooperative Settings on Mathematical Problem Solving
Mevarech, Z. R. (1999) · The Journal of Educational Research · doi:10.1080/00220679909597597 · Cited by 133 · see citing works
Scaffolding strategies for supporting middle school students’ online inquiry processes
Zhang, M., & Quintana, C. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.07.016 · Cited by 128 · see citing works
Evaluating metacognitive scaffolding in Guided Invention Activities
Roll, I., Holmes, N. G., Day, J., & Bonn, D. A. (2012) · Instructional Science · doi:10.1007/s11251-012-9208-7 · Cited by 126 · see citing works
Effects of metacognitive instruction on low achievers in mathematics problems
Cardelle-Elawar, M. (1995) · Teaching and Teacher Education · doi:10.1016/0742-051x(94)00019-3 · Cited by 113 · see citing works
Zion, M., Michalsky, T., & Mevarech, Z. R. (2005) · International Journal of Science Education · doi:10.1080/09500690500068626 · Cited by 106 · see citing works
Donato, D., Bonchi, F., Chi, T., & Maarek, Y. (2010) · International World Wide Web Conference (WWW) · doi:10.1145/1772690.1772724 · Cited by 88 · see citing works
Elementary School Children Reading Scientific Texts: Effects of Metacognitive Instruction
Michalsky, T., Mevarech, Z. R., & Haibi, L. (2009) · The Journal of Educational Research · doi:10.3200/joer.102.5.363-376 · Cited by 86 · see citing works
Bates, M. J. (1990) · Cited by 84 · see citing works
Bates, M. J. (1979) · Journal of the American Society for Information Science · doi:10.1002/asi.4630300507 · Cited by 82 · see citing works
Self-Regulated Learning in Technology Enhanced Learning Environments
Persico, D., & Steffens, K. (2017) · Springer eBooks · doi:10.1007/978-3-319-02600-8_11 · Cited by 74 · see citing works
Active reading and its discontents
Tashman, C. S., & Edwards, W. K. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979376 · Cited by 65 · see citing works
An, Y., & Cao, L. (2014) · Cited by 58 · see citing works
Kuo, F.-R., Chen, N., & Hwang, G. (2013) · Computers & Education · doi:10.1016/j.compedu.2013.11.005 · Cited by 54 · see citing works
McKenzie, P. J. (2010) · Scholarship@Western (Western University) · Cited by 42 · see citing works
A taxonomy of adolescent metacognitive knowledge during the information search process
Bowler, L. (2009) · Library & Information Science Research · doi:10.1016/j.lisr.2009.09.005 · Cited by 40 · see citing works
Search as Learning (Dagstuhl Seminar 17092)
Collins‐Thompson, K., Hansen, P., & Hauff, C. (2017) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/dagrep.7.2.135 · Cited by 40 · see citing works
Search as Learning (SAL) Workshop 2016
Jacek Gwizdka, A., Hansen, P., Hauff, C., He, J., & Kando, N. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2917766 · Cited by 32 · see citing works
NSF workshop on task-based information search systems
Kelly, D., Arguello, J., & Capra, R. (2013) · ACM SIGIR Forum · doi:10.1145/2568388.2568407 · Cited by 27 · see citing works
Self-citationIntroduction to the special issue on search as learning
Eickhoff, C., Jacek Gwizdka, A., Hauff, C., & He, J. (2017) · Information Retrieval · doi:10.1007/s10791-017-9315-9 · Cited by 25 · see citing works
Huertas, A., Vargas, O. L., & Rodríguez, L. B. S. (2016) · Journal of Educational Computing Research · doi:10.1177/0735633116656634 · Cited by 24 · see citing works
OrgBox: Supporting Cognitive and Metacognitive Activities During Exploratory Search
Ward, A. R., & Capra, R. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462790 · Cited by 17 · see citing works
Self-citationQvarfordt, P., Tretter, S., Golovchinsky, G., & Dunnigan, T. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2609620 · Cited by 17 · see citing works
Choi, B., Arguello, J., Capra, R., & Ward, A. R. (2021) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446029 · Cited by 17 · see citing works
Self-citationCrescenzi, A., Li, Y., Zhang, Y., & Capra, R. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331309 · Cited by 16 · see citing works
Self-citationTalk as a metacognitive strategy during the information search process of adolescents
Bowler, L. (2010) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 11 · see citing works
Wang, S., Unal, D. S., & Walker, E. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300258 · Cited by 11 · see citing works
KuoFan-Ray, ChenNian-Shing, & HwangGwo-Jen (2014) · Computers in Education · doi:10.5555/2753879.2754105 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Anita Crescenzi. 2016. Metacognitive knowledge and metacognitive regulation in time-constrained information search. In Search as Learning workshop at SIGIR '16, Vol. 1647. 1-5. http://ceur-ws.org/Vol-1647/SAL2016_paper_5.pdf
- Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Human mental workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. Elsevier Science Publishers B. V., North Holland, 138-183.
Cited by
Sorted by citations
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 280 · see citing works
Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
Suh, S., Min, B., Palani, S., & Xia, H. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606756 · Cited by 151 · see citing works
Generative AI in Knowledge Work: Design Implications for Data Navigation and Decision-Making
Yun, B., Feng, D., Chen, A. S., Nikzad, A., & Salehi, N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713337 · Cited by 28 · see citing works
Toward Cranfield-inspired reusability assessment in interactive information retrieval evaluation
Liu, J. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103007 · Cited by 21 · see citing works
Yang, Q., Feng, S., Zhao, T., & Kalantari, S. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103229 · Cited by 17 · see citing works
OrgBox: Supporting Cognitive and Metacognitive Activities During Exploratory Search
Ward, A. R., & Capra, R. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462790 · Cited by 17 · see citing works
Self-citationAnalyzing information resources that support the creative process
Li, Y., Zhang, Y., & Capra, R. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505817 · Cited by 15 · see citing works
Self-citationSearchIdea: An Idea Generation Tool to Support Creativity in Academic Search
Chavula, C., Choi, Y., & Rieh, S. Y. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578294 · Cited by 16 · see citing works
Yun, S., & Lim, Y.-K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713347 · Cited by 15 · see citing works
Design Principles for Exploratory Search Interfaces
Hoeber, O. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716443 · Cited by 13 · see citing works
Urgo, K., & Arguello, J. (2025) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000084 · Cited by 14 · see citing works
Zhang, Q., & Lockee, B. B. (2022) · Journal of Formative Design in Learning · doi:10.1007/s41686-022-00068-y · Cited by 9 · see citing works
Li, Y., Crescenzi, A., Ward, A. R., & Capra, R. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24801 · Cited by 9 · see citing works
Self-citationSeeking Socially Responsible Consumers: Exploring the Intention-Search-Behaviour Gap
Azzopardi, L. A., & van der Sluis, F. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638324 · Cited by 9 · see citing works
Understanding Query Combination Behavior in Exploratory Searches
Li, P., Zhang, Y., & Zhang, B. (2022) · Applied Sciences · doi:10.3390/app12020706 · Cited by 7 · see citing works
Organizing Found Information in Public Digital Library Search
Orin, K., & Hoeber, O. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716471 · Cited by 7 · see citing works
Visual Collaboration - An Approach for Visual Analytical Collaborative Research
Blazevic, M., Sina, L. B., & Nazemi, K. (2022) · International Conference on Information Visualisation (IV) · doi:10.1109/iv56949.2022.00057 · Cited by 5 · see citing works
van der Sluis, F., Azzopardi, L. A., & Meier, F. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730347 · Cited by 2 · see citing works
Exploratory Search and Beyond: A Study of Complex Search Scenarios within a Public Digital Library
Boon, M., Hoeber, O., Hoeber, L., Storie, D., & Ramshaw, V. (2024) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1007 · Cited by 2 · see citing works
Türkmen, G., & Yıldırım, Z. (2025) · Journal of Museum Education · doi:10.1080/10598650.2025.2467497 · Cited by 2 · see citing works
Zhu, R. H., Cheng, C., Song, Y., & Cai, Z. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3699750 · Cited by 1 · see citing works
Contextualizing Session Resuming Reasons with Tasks Involving Expected Cross-session Searches
Li, Y., & Capra, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28032-0_32
Self-citationIntegrating Machine Learning in Visual Analytics for Supporting Collaboration in Science
Blazevic, M., Sina, L. B., Secco, C. A., & Nazemi, K. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-46549-9_12
Urgo, K., Arguello, J., & Capra, R. (2026) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3786304.3787890
Self-citationDoes It Still Make Sense? Organizing and Summarizing Resources in Cross-Session Aggregated Search
Momeni, M., & Hoeber, O. (2026) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3786304.3787873 · Cited by 1 · see citing works
- Authors
- Crescenzi, A., Ward, A. R., Li, Y., & Capra, R.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
- Venue
- Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2021-07-11
- Pages
- 1197-1207
- OpenAlex
- W3154112228
- Cited by
- 25
- References
- 47
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 6 · 2023: 4 · 2022: 5 · 2021: 1
- FWCI
- 3.32
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- Funders
- National Science Foundation (IIS-1552587.)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Social Sciences › Decision Sciences › Information Systems and Management › Personal Information Management and User Behavior
- Keywords
- metacognition, exploratory search, information synthesis, information organization, reflective learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
