Using Metrics of Curation to Evaluate Information-Based Ideation
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Evaluating creativity support environments is challenging. Some approaches address people’s experiences of creativity. The present method measures creativity, across conditions, in the products that people make. This research introduces information-based ideation(IBI), a paradigm for investigating open-ended tasks and activities in which users develop new ideas. IBI tasks span imagining, planning, and reflecting on a weekend, vacation, outfit, makeover, paper, internship, thesis, design, campaign, crisis response, career, or invention. What products do people create through engagement in IBI?Curation of digital media incorporates conceptualization, finding and choosing information objects, annotation, and synthesis. Through engagement in IBI tasks, people create curation products. This article formulates a quantitative methodology for evaluating IBI support tools, building on prior creative cognition research in engineering design to derive a battery of ideation metrics of curation. Elemental ideation metrics evaluate creativity within curated found objects.Holistic ideation metrics evaluate how a curation puts elements together. IBI support environments are characterized by their underlying medium of curation. Curation media include lists, such as listicles, and grids, such as the boards of Pinterest. An in-depth case study investigates information composition, an art-based medium representing a curation as a freeform visual semantic connected whole. We raise two creative cognition challenges for IBI. One challenge is overcoming fixation—for instance, when a person gets stuck in a counterproductive mental set. The other challenge is to bridge information visualization’s synthesis gap, by providing support for connecting findings. To address the challenges, we develop mixed-initiative information composition(MI 2 C), integrating human curation of information composition with automated agents of information retrieval and visualization. We hypothesize that MI 2 C generates provocative stimuli that help users overcome fixation to become more creative on IBI tasks. We hypothesize that MI 2 C’s integration of curation and visualization bridges the synthesis gap to help users become more creative. To investigate these hypotheses, we apply ideation metrics of curation to interpret results from experiments with 44 and 49 participants.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The associative basis of the creative process.
Mednick, S. (1962) · Psychological Review · doi:10.1037/h0048850 · Cited by 4,140 · cited 2×
“Variety provides opportunities for more remote associations, and more remote analogies, both of which are likely to lead to creative products [Mednick 1962].”
BackgroundGuilford, J. P. (1950) · American Psychologist · doi:10.1037/h0063487 · Cited by 3,588
Boden, M. A. (2004) · doi:10.4324/9780203508527 · Cited by 2,775
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Exploratory search: from finding to understanding
Marchionini, G. (2006) · Communications of the ACM · doi:10.1145/1121949.1121979 · Cited by 1,506
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
Ambiguity as a resource for design
Gaver, W. W., Beaver, J., & Benford, S. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642653 · Cited by 1,474 · cited 1×
“However, Sandbox uses arrows to explicitly express relationships among collected elements rather than composition’s implicit visual design features, which employ ambiguity to support flexible interpretation [Gaver et al. 2003].”
MethodIllich, I. (1973) · doi:10.1558/isbn.9780714521602 · Cited by 1,410
The Cambridge Handbook of Creativity
(2010) · doi:10.1017/cbo9780511763205 · Cited by 1,291
Jansson, D. G., & Smith, S. M. (1991) · Design Studies · doi:10.1016/0142-694x(91)90003-f · Cited by 1,087
Self-citation
Cited by
Mapping the Landscape of Creativity Support Tools in HCI
Frich, J., MacDonald Vermeulen, L., Remy, C., Biskjaer, M. M., & Dalsgaard, P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300619 · Cited by 410
“2018 [29, 30, 62, 73, 90, 111, 126, 162, 165, 171, 175, 184] 2017 [3, 7, 36, 43, 64, 82, 94, 98, 102, 118, 130, 134, 150, 158, 176, 183, 192] 2016 [9, 25, 35, 39, 57, 63, 66, 79, 81, 87, 101, 114, 117, 127, 129, 159, 167, 187] 2015 [38, 54, 89, 100, 123, 137, 160, 166, 168, 186] 2014 [14, 37, 55, 71, 83, 88, 97, 99, 1…”
The Creativity of Text-to-Image Generation
Oppenlaender, J. (2022) · Proceedings of the 25th International Academic Mindtrek Conference · doi:10.1145/3569219.3569352 · Cited by 290
Evaluating Creativity Support Tools in HCI Research
Remy, C., MacDonald Vermeulen, L., Frich, J., Biskjaer, M. M., & Dalsgaard, P. (2020) · Proceedings of the 2020 ACM Designing Interactive Systems Conference · doi:10.1145/3357236.3395474 · Cited by 96
“2018 [105, 142, 150, 30, 57, 29, 85, 115, 146, 66] 2017 [90, 5, 2, 152, 45, 164, 58, 79, 123, 120, 136, 99, 109, 36, 95, 139] 2016 [119, 59, 160, 78, 82, 107, 143, 76, 39, 98, 6, 108, 25, 35, 121] 2015 [51, 38, 159, 126, 140, 144, 97, 84, 112] 2014 [37, 156, 111, 96, 113, 163, 155, 52, 80, 94, 13, 83] 2013 [158, 81, 1…”
MethodTowards a Reflection in Creative Experience Questionnaire
Ford, C., & Bryan-Kinns, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581077 · Cited by 15
“CST researchers have thus developed their own objective metrics to measure aspects of creativity such as ideation [46] or mutual engagement [13].”
Dimensional Reasoning and Research Design Spaces
MacNeil, S., Okerlund, J., & Latulipe, C. (2017) · Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition · doi:10.1145/3059454.3059472 · Cited by 8
Metadata
- Authors
- Kerne, A., Webb, A. M., Smith, S. M., Linder, R., Lupfer, N., Qu, Y., Moeller, J., & Damaraju, S.
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2014-06
- Volume
- 21
- Issue
- 3
- Pages
- 1-48
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W2028261086
- Cited by
- 93
- References
- 16
- FWCI
- 10.13
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using Metrics of Curation to Evaluate Information-Based Ideation. https://app.cochat.ai/papers/10.1145/2591677
- The paper
- Kerne, A., Webb, A. M., Smith, S. M., Linder, R., Lupfer, N., Qu, Y., Moeller, J., & Damaraju, S. (2014). Using Metrics of Curation to Evaluate Information-Based Ideation. ACM Transactions on Computer-Human Interaction. https://doi.org/10.1145/2591677
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.