The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Reliability of Nominal Data Based on Qualitative Judgments
Abstract
Most research related to the reliability and validity of marketing measures has focused on multi-item quantitative scales. In contrast, little attention has been given to the quality of nominal scale data developed from qualitative judgments. Judgment-based (“coded”) nominal scale data are important and frequently used in marketing research-for example, in analysis of consumer responses to open-ended survey questions, in cognitive response research, in meta-analysis, and in content analysis. The authors address opportunities and challenges involved in evaluating and improving the quality of judgment-based nominal scale data, with specific emphasis on the use of multiple judges. They review approaches commonly used in other disciplines, then develop a new index of reliability that is more appropriate for the type of interjudge data typically found in marketing studies. Data from a cognitive response experiment are used to illustrate the new index and compare it with other common measures. The authors conclude with suggestions on how to improve the design of studies that rely on judgment-coded data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“When the described coding procedures were applied, the intercoder reliability according to Perreault and Leigh (1989) was .93, which exceeded the required threshold of .8.”
From A Meta-Analysis of Organizational Complaint Handling and Customer Responses (2011) · cites this paper as Perreault and Leigh (1989)
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
A Paradigm for Developing Better Measures of Marketing Constructs
Churchill, G. A. (1979) · Journal of Marketing Research · doi:10.1177/002224377901600110 · Cited by 16,549
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Coefficient Kappa: Some Uses, Misuses, and Alternatives
Brennan, R. L., & Prediger, D. J. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100307 · Cited by 1,676
Reliability: A Review of Psychometric Basics and Recent Marketing Practices
Peter, J. P. (1979) · Journal of Marketing Research · doi:10.2307/3150868 · Cited by 1,577
Large sample standard errors of kappa and weighted kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969) · Psychological Bulletin · doi:10.1037/h0028106 · Cited by 1,536
Bettman, J. R., & Park, C. W. (1980) · Journal of Consumer Research · doi:10.1086/208812 · Cited by 1,344
Construct Validity: A Review of Basic Issues and Marketing Practices
Peter, J. P. (1981) · Journal of Marketing Research · doi:10.2307/3150948 · Cited by 1,189
Measures of response agreement for qualitative data: Some generalizations and alternatives.
Light, R. J. (1971) · Psychological Bulletin · doi:10.1037/h0031643 · Cited by 747
Attributions in the Board Room: Causal Reasoning in Corporate Annual Reports
Bettman, J. R., & Weitz, B. A. (1983) · Administrative Science Quarterly · doi:10.2307/2392616 · Cited by 612
Correspondence Analysis: Graphical Representation of Categorical Data in Marketing Research
Hoffman, D. L., & Franke, G. R. (1986) · Journal of Marketing Research · doi:10.1177/002224378602300302 · Cited by 576
The Determinants of Salesperson Performance: A Meta-Analysis
Churchill, G. A., Ford, N. M., Hartley, S. W., & Walker, O. C. (1985) · Journal of Marketing Research · doi:10.1177/002224378502200201 · Cited by 559
Measuring Agreement between Two Judges on the Presence or Absence of a Trait
Fleiss, J. L. (1975) · Biometrics · doi:10.2307/2529549 · Cited by 506
How Advertising Affects Sales: Meta-Analysis of Econometric Results
Assmus, G., Farley, J. U., & Lehmann, D. R. (1984) · Springer eBooks · doi:10.1007/978-3-031-88060-5_9 · Cited by 502
The Cognitive Processes Mediating Acceptance of Advertising
Wright, P. L. (1973) · Journal of Marketing Research · doi:10.2307/3149409 · Cited by 472
Mitchell, S. K. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.376 · Cited by 445
The Reliability of Clinical Methods, Data and Judgments
Koran, L. M. (1975) · New England Journal of Medicine · doi:10.1056/nejm197510022931405 · Cited by 402
Ramifications of a Population Model for κ as a Coefficient of Reliability
KRAEMER, H. C. (1979) · Psychometrika · doi:10.1007/bf02296208 · Cited by 295
Children's Use of Cognitive Defenses Against Television Advertising: A Cognitive Response Approach
Brucks, M. L., Armstrong, G. M., & Goldberg, M. E. (1988) · Journal of Consumer Research · doi:10.1086/209129 · Cited by 265
Communications Through Limited Response Questioning
Bennett, E., Alpert, R., & Goldstein, A. C. (1954) · Public Opinion Quarterly · doi:10.1086/266520 · Cited by 254
Bivariate Agreement Coefficients for Reliability of Data
Krippendorff, K. (1970) · Sociological Methodology · doi:10.2307/270787 · Cited by 251
Coefficients of Agreement Between Observers and Their Interpretation
Maxwell, A. E. (1977) · The British Journal of Psychiatry · doi:10.1192/bjp.130.1.79 · Cited by 196
Judgment of counseling process: Reliability, agreement, and error.
Lawlis, G. F., & Lu, E. (1972) · Psychological Bulletin · doi:10.1037/h0032935 · Cited by 174
Reliability of Psychiatric Diagnosis
Helzer, J. E. (1977) · Archives of General Psychiatry · doi:10.1001/archpsyc.1977.01770140019001 · Cited by 158
MOMENTS OF THE STATISTICS KAPPA AND WEIGHTED KAPPA
Everitt, B. S. (1968) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1968.tb00400.x · Cited by 156
Hubert, L. J. (1977) · Psychological Bulletin · doi:10.1037/0033-2909.84.2.289 · Cited by 153
Effectiveness of Multiple Request Strategies: A Synthesis of Research Results
Fern, E. F., Monroe, K. B., & Avila, R. A. (1986) · Journal of Marketing Research · doi:10.1177/002224378602300206 · Cited by 151
Peter, J. P., & Churchill, G. A. (1986) · Journal of Marketing Research · doi:10.1177/002224378602300101 · Cited by 126
Dewey, M. E. (1983) · The British Journal of Psychiatry · doi:10.1192/bjp.143.5.487 · Cited by 102
On the Analysis of Qualitative Data in Marketing Research
Green, P. E., Carmone, F. J., & Wachspress, D. P. (1977) · Journal of Marketing Research · doi:10.1177/002224377701400106 · Cited by 102
Coding Reliability and Validity of Interview Data
Crittenden, K. S., & Hill, R. J. (1971) · American Sociological Review · doi:10.2307/2093766 · Cited by 79
A Model-Free Approach for Analysis of Complex Contingency Data in Survey Research
Perreault, W. D., & Barksdale, H. C. (1980) · Journal of Marketing Research · doi:10.1177/002224378001700409 · Cited by 64
Self-citationAlternating Least Squares Optimal Scaling: Analysis of Nonmetric Data in Marketing Research
Perreault, W. D., & Young, F. W. (1980) · Journal of Marketing Research · doi:10.1177/002224378001700101 · Cited by 54
Self-citationA Probabilistic Latent Class Model for Assessing Inter-Judge Reliability
Dillon, W. R., & Mulani, N. (1984) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr1904_5 · Cited by 51
A Cross-Sectional Optimal Scaling Analysis of the Index of Consumer Sentiment
Didow, N. M., Perreault, W. D., & Williamson, N. C. (1983) · Journal of Consumer Research · doi:10.1086/208973 · Cited by 30
Self-citationEffects of choice and sales message on customer-salesman interaction.
Farley, J. U., & Swinth, R. L. (1967) · Journal of Applied Psychology · doi:10.1037/h0024473 · Cited by 23
Experiences with the Bettman-Park Verbal-Protocol Coding Scheme
Biehal, G. J., & Chakravarti, D. (1982) · Journal of Consumer Research · doi:10.1086/208884 · Cited by 22
An Extension of the Random Error Coefficient of Agreement to N x N Tables
Janes, C. L. (1979) · The British Journal of Psychiatry · doi:10.1192/bjp.134.6.617 · Cited by 17
An Evaluation of the Children's Advertising Review Unit
Armstrong, G. M. (1984) · Journal of Public Policy & Marketing · doi:10.1177/074391568400300103 · Cited by 17
Cited by
Sorted by citations
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225
Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
A Meta-Analysis of Cronbach's Coefficient Alpha
Peterson, R. A. (1994) · Journal of Consumer Research · doi:10.1086/209405 · Cited by 3,028
Meuter, M. L., Ostrom, A. L., Roundtree, R. I., & Bitner, M. J. (2000) · Journal of Marketing · doi:10.1509/jmkg.64.3.50.18024 · Cited by 3,035
Validation Guidelines for IS Positivist Research
Straub, D., & Gefen, D. (2004) · Communications of the Association for Information Systems · doi:10.17705/1cais.01324 · Cited by 2,797
Metadata
- Authors
- Perreault, W. D., & Leigh, L. E.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Birmingham–Southern College (US)
- Venue
- Journal of Marketing Research
- Publisher
- JSTOR
- Published
- 1989-05
- Volume
- 26
- Issue
- 2
- Pages
- 135
- ISSN
- 0022-2437 (ISSN-L)
- OpenAlex
- W2069585723
- Cited by
- 1,575
- References
- 42
- Citations by year
- 2026: 19 · 2025: 35 · 2024: 43 · 2023: 44 · 2022: 56 · 2021: 59 · 2020: 60 · 2019: 58 · 2018: 52 · 2017: 39 · 2016: 48 · 2015: 89 · 2014: 77 · 2013: 88 · 2012: 86
- FWCI
- 12.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Marketing Research (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- reliability statistics, nominal data, qualitative judgment, marketing activities, measurement validity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review