On the Dependability and Feasibility of Layperson Ratings of Divergent Thinking
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
A new system for subjective rating of responses to divergent thinking tasks was tested using raters recruited from Amazon Mechanical Turk. The rationale for the study was to determine if such raters could provide reliable (aka generalizable) ratings from the perspective of generalizability theory. To promote reliability across the Alternative Uses and Consequence task prompts often used by researchers as measures of Divergent Thinking, two parallel scales were developed to facilitate feasibility and validity of ratings performed by laypeople. Generalizability and dependability studies were conducted separately for two scoring systems: the average-rating system and the snapshot system. Results showed that it is difficult to achieve adequate reliability using the snapshot system, while good reliability can be achieved on both task families using the average-rating system and a specific number of items and raters. Additionally, the construct validity of the average-rating system is generally good, with less validity for certain Consequences items. Recommendations for researchers wishing to adopt the new scales are discussed, along with broader issues of generalizability of subjective creativity ratings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The social psychology of creativity: A componential conceptualization.
Amabile, T. M. (1983) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.45.2.357 · Cited by 4,034
Social psychology of creativity: A consensual assessment technique.
Amabile, T. M. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.43.5.997 · Cited by 2,278
Brennan, R. L. (2001) · doi:10.1007/978-1-4757-3456-0 · Cited by 1,121
Silvia, P. J., Winterstein, B. P., Willse, J. T., Barona, C. M., Cram, J. T., Hess, K. I., Martinez, J. L., & Richard, C. A. (2008) · Psychology of Aesthetics, Creativity, and the Arts · doi:10.1037/1931-3896.2.2.68 · Cited by 912 · cited 2×
“For example, following from (Silvia et al., 2008), raters were considered a random facet in the current study design, meaning that the raters in this study represent a random sample from an infinite universe of raters, all of which are interchangeable.”
BackgroundResultSelf-citationCan We Trust Creativity Tests? A Review of the Torrance Tests of Creative Thinking (TTCT)
Kim, K. H. (2006) · Creativity Research Journal · doi:10.1207/s15326934crj1801_2 · Cited by 830 · cited 1×
“For the Consequences task, the top anchor was designed to capture imagination and elaboration, the latter being a scoring system sometimes used for items from the Torrance Tests (Kim, 2006).”
MethodBeaty, R. E., & Silvia, P. J. (2012) · Psychology of Aesthetics, Creativity, and the Arts · doi:10.1037/a0029171 · Cited by 551
Self-citationTo create or to recall? Neural mechanisms underlying the generation of creative new ideas
Benedek, M., Jauk, E., Fink, A., Koschutnig, K., Reishofer, G., Ebner, F., & Neubauer, A. C. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.021 · Cited by 407
Neural Mechanisms of Episodic Retrieval Support Divergent Creative Thinking
Madore, K. P., Thakral, P. P., Beaty, R. E., Addis, D. R., & Schacter, D. L. (2019) · Cerebral Cortex · doi:10.1093/cercor/bhx312 · Cited by 137
Cited by
Scoring divergent thinking tests: A review and systematic framework.
Reiter-Palmon, R., Forthmann, B., & Barbot, B. (2019) · Psychology of Aesthetics, Creativity, and the Arts · doi:10.1037/aca0000227 · Cited by 391
Automating creativity assessment with SemDis: An open platform for computing semantic distance
Beaty, R. E., & Johnson, D. R. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01453-w · Cited by 341
“A popular approach to originality scoring is the subjective scoring method (Hass, Rivera, & Silvia, 2018; Silvia et al., 2008).”
BackgroundMethodOrganisciak, P., Acar, S., Dumas, D., & Berthiaume, K. (2023) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2023.101356 · Cited by 202
Naming unrelated words predicts creativity
Olson, J. A., Nahas, J., Chmoulevitch, D., Cropper, S. J., & Webb, M. E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2022340118 · Cited by 182
Metadata
- Authors
- Hass, R. W., Rivera, M., & Silvia, P. J.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2018-08-13
- Volume
- 9
- Pages
- 1343-1343
- ISSN
- 1664-1078 (ISSN-L)
- OpenAlex
- W2885069892
- Cited by
- 41
- References
- 8
- FWCI
- 5.55
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On the Dependability and Feasibility of Layperson Ratings of Divergent Thinking. https://app.cochat.ai/papers/10.3389/fpsyg.2018.01343
- The paper
- Hass, R. W., Rivera, M., & Silvia, P. J. (2018). On the Dependability and Feasibility of Layperson Ratings of Divergent Thinking. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2018.01343
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.