The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
The jingle and jangle of emotion assessment: Imprecise measurement, casual scale usage, and conceptual fuzziness in emotion research.
Abstract
Although affective science has seen an explosion of interest in measuring subjectively experienced distinct emotional states, most existing self-report measures tap broad affect dimensions and dispositional emotional tendencies, rather than momentary distinct emotions. This raises the question of how emotion researchers are measuring momentary distinct emotions in their studies. To address this question, we reviewed the self-report measurement practices regularly used for the purpose of assessing momentary distinct emotions, by coding these practices as observed in a representative sample of articles published in Emotion from 2001-2011 (n = 467 articles; 751 studies; 356 measurement instances). This quantitative review produced several noteworthy findings. First, researchers assess many purportedly distinct emotions (n = 65), a number that differs substantially from previously developed emotion taxonomies. Second, researchers frequently use scales that were not systematically developed, and that include items also used to measure at least 1 other emotion on a separate scale in a separate study. Third, the majority of scales used include only a single item, and had unknown reliability. Together, these tactics may create ambiguity regarding which emotions are being measured in empirical studies, and conceptual inconsistency among measures of purportedly identical emotions across studies. We discuss the implications of these problematic practices, and conclude with recommendations for how the field might improve the way it measures emotions. (PsycINFO Database Record
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · cited 1×
“…perspective, a dimensionalist perspective (e.g., examining emotions at the level of positive and negative affect or valence and arousal; Russell, 1980; Watson & Tellegen, 1985), or a social or psychological constructivist perspective (i.e., treating distinct emotions as cognitively…”
BackgroundThe genetical evolution of social behaviour. I
Hamilton, W. D. (1964) · Journal of Theoretical Biology · doi:10.1016/0022-5193(64)90038-4 · Cited by 15,725
Happiness is everything, or is it? Explorations on the meaning of psychological well-being.
Ryff, C. D. (1989) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.57.6.1069 · Cited by 14,847
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Subjective well-being. The science of happiness and a proposal for a national index
Diener, E. (2000) · American Psychologist · doi:10.1037/0003-066x.55.1.34 · Cited by 7,121
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Constructing validity: Basic issues in objective scale development.
Clark, L. A., & Watson, D. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.3.309 · Cited by 6,567
The development of markers for the Big-Five factor structure
Goldberg, L. R. (1992) · Psychological Assessment · doi:10.1037/1040-3590.4.1.26 · Cited by 5,530
Constants across cultures in the face and emotion.
Ekman, P., & Friesen, W. V. (1971) · Journal of Personality and Social Psychology · doi:10.1037/h0030377 · Cited by 5,509
The dark triad of personality: Narcissism, Machiavellianism, and psychopathy
Paulhus, D. L., & Williams, K. M. (2002) · Journal of Research in Personality · doi:10.1016/s0092-6566(02)00505-6 · Cited by 5,265 · cited 1×
“…in the Big Five's coverage of human personality and developed theories about traits that lie outside the Big Five, such as the Dark Triad (e.g., Paulhus & Williams, 2002), Honest-Humility (e.g., Ashton, Lee, & DeVries, 2014), and motives such as Need for Achievement (e.g., McClelland, 1987;…”
BackgroundRammstedt, B., & John, O. P. (2007) · Journal of Research in Personality · doi:10.1016/j.jrp.2006.02.001 · Cited by 4,244 · cited 1×
“…frequency with which researchers rely on different scales to measure the same emotion across studies—without knowing whether those scales contain convergent sets of items—which is in turn likely to improve the rate at which researchers measure a given emotion with items are unique to that emotion.”
BackgroundA Measure of Subjective Happiness: Preliminary Reliability and Construct Validation
Lyubomirsky, S., & Lepper, H. S. (1999) · Social Indicators Research · doi:10.1023/a:1006824100041 · Cited by 4,071
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
High income improves evaluation of life but not emotional well-being
Kahneman, D., & Deaton, A. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1011492107 · Cited by 3,162
Robins, R. W., Hendin, H. M., & Trzesniewski, K. H. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/0146167201272002 · Cited by 3,041
Compassion: an evolutionary analysis and empirical review
Goetz, J. L., Keltner, D., & Simon-Thomas, E. (2010) · Psychological Bulletin · doi:10.1037/a0018807 · Cited by 2,200
Mauss, I. B., & Robinson, M. D. (2009) · Cognition and Emotion · doi:10.1080/02699930802204677 · Cited by 2,078
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Barrett, L. F. (2006) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2006.00003.x · Cited by 1,859 · cited 1×
“We coded articles for evidence of researchers having adopted one of these three approaches, in particular, because they represent the three most prevalent current theoretical orientations toward emotion research (see Barrett, 2014; Russell, 2014; Tracy, 2014).”
MethodP-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500
Rethinking the mood and anxiety disorders: A quantitative hierarchical model for DSM-V.
Watson, D. (2005) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.114.4.522 · Cited by 1,332 · cited 2×
“EMOTION MEASUREMENT Table 2 (continued)”
BackgroundGoldin, P. R., & Gross, J. J. (2010) · Emotion · doi:10.1037/a0018441 · Cited by 1,242 · cited 1×
“Specifically, for studies with hypotheses that solely involved affect dimensions (e.g., Goldin & Gross, 2010; Storbeck & Clore, 2008), we coded the authors as taking a dimensionalist theoretical approach.”
BackgroundHeart Strings and Purse Strings: Carryover Effects of Emotions on Economic Decisions
Lerner, J. S., Small, D. A., & Loewenstein, G. (2004) · Psychological Science · doi:10.1111/j.0956-7976.2004.00679.x · Cited by 959
Factor analysis and scale revision.
Reise, S. P., Waller, N. G., & Comrey, A. L. (2000) · Psychological Assessment · doi:10.1037/1040-3590.12.3.287 · Cited by 855
Beat them or ban them: the characteristics and social functions of anger and contempt
Fischer, A. H., & Roseman, I. J. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.93.1.103 · Cited by 723
Signs of appeasement: Evidence for the distinct displays of embarrassment, amusement, and shame.
Keltner, D. (1995) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.68.3.441 · Cited by 718 · cited 1×
“…nonverbal emotion expressions have identified 10 emotions that are reliably associated with distinct, cross-culturally recognized expressions (anger, contempt, disgust, embarrassment, fear, happiness, pride, sadness, shame, and surprise; Ekman & Friesen, 1971; Keltner, 1995; Tracy & Robins, 2008).”
BackgroundSynchrony and the social tuning of compassion.
Valdesolo, P., & DeSteno, D. (2011) · Emotion · doi:10.1037/a0021302 · Cited by 654
Two replicable suppressor situations in personality research
Paulhus, D. L., Robins, R. W., Trzesniewski, K. H., & Tracy, J. L. (2004) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3902_7 · Cited by 631
Self-citationThe moral emotions: A social–functionalist account of anger, disgust, and contempt.
Hutcherson, C. A., & Gross, J. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022408 · Cited by 609
The many meanings/aspects of emotion: definitions, functions, activation, and regulation
Izard, C. E. (2010) · Emotion Review · doi:10.1177/1754073910374661 · Cited by 574
Measurement error in psychological research: Lessons from 26 research scenarios.
Schmidt, F. L., & Hunter, J. E. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.199 · Cited by 567 · cited 2×
“…the size of observed effects between those unreliable measures and measures of other constructs (Kashy, Donnellan, Ackerman, & Russell, 2009; Schmidt & Hunter, 1996), which is particularly problematic in light of the field's current emphasis on more precise estimation of effect sizes (e.g.,…”
BackgroundHodson, G., & Costello, K. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01962.x · Cited by 497
Elevation Leads to Altruistic Behavior
Schnall, S., Roper, J., & Fessler, D. M. T. (2010) · Psychological Science · doi:10.1177/0956797609359882 · Cited by 497
Ketelaar, T., & Tung Au, W. (2003) · Cognition and Emotion · doi:10.1080/02699930143000662 · Cited by 482
Counteracting Loneliness: On the Restorative Function of Nostalgia
Zhou, X., Sedikides, C., Wildschut, T., & Gao, D.-G. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02194.x · Cited by 468
Odyssey's end: Lay conceptions of nostalgia reflect its original Homeric meaning
Hepper, E. G., Ritchie, T. D., Sedikides, C., & Wildschut, T. (2012) · Emotion · doi:10.1037/a0025167 · Cited by 467
Romantic love and sexual desire in close relationships
Gonzaga, G. C., Turner, R. A., Keltner, D., Campos, B., & Altemus, M. (2006) · Emotion · doi:10.1037/1528-3542.6.2.163 · Cited by 292
Amodio, D. M., Devine, P. G., & Harmon-Jones, E. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01933.x · Cited by 274
Roseman, I. J. (2011) · Emotion Review · doi:10.1177/1754073911410744 · Cited by 200 · cited 1×
“Distinct emotions are constructs that generally involve consistent and specific patterns of subjective feelings, physiological changes, neural activity, cognitive appraisals, and motivated action tendencies (Ekman, 1992; Kragel & LaBar, 2014; Roseman, 2011; see Tracy & Randles, 2011).”
BackgroundLelieveld, G.-J., Van Dijk, E., Van Beest, I., & Van Kleef, G. A. (2012) · Personality and Social Psychology Bulletin · doi:10.1177/0146167212446938 · Cited by 129
The ‘Short Five’ (S5): Measuring Personality Traits Using Comprehensive Single Items
Konstabel, K., Lönnqvist, J., Walkowitz, G., Konstabel, K., & Verkasalo, M. (2012) · European Journal of Personality · doi:10.1002/per.813 · Cited by 121
Cited by
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920952393 · Cited by 969
Construct Validation in Social and Personality Research: Current Practice and Recommendations
Flake, J. K., Pek, J., & Hehman, E. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693063 · Cited by 697
Social Presence: Conceptualization and Measurement
Kreijns, K., Xu, K., & Weidlich, J. (2022) · Educational Psychology Review · doi:10.1007/s10648-021-09623-8 · Cited by 401
The impact of a view from a window on thermal comfort, emotion, and cognitive performance
Ko, W. H., Schiavon, S., Zhang, H., Graham, L. T., Brager, G., Mauss, I., & Lin, Y.-W. (2020) · Building and Environment · doi:10.1016/j.buildenv.2020.106779 · Cited by 342
Cognitive reappraisal and acceptance: Effects on emotion, physiology, and perceived cognitive costs.
Troy, A. S., Shallcross, A. J., Brunner, A., Friedman, R., & Jones, M. C. (2018) · Emotion · doi:10.1037/emo0000371 · Cited by 322
Metadata
- Authors
- Weidman, A. C., Steckler, C. M., & Tracy, J. L.
- Venue
- Emotion
- Published
- 2017
- Volume
- 17
- Issue
- 2
- Pages
- 267-295
- ISSN
- 1528-3542 (ISSN-L)
- 1931-1516
- OpenAlex
- W2523148522
- Cited by
- 217
- References
- 43
- FWCI
- 12.77
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.