The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Justifying Aggregation With Consensus-Based Constructs: A Review and Examination of Cutoff Values for Common Aggregation Indices
Abstract
Management researchers often use consensus-based composition models to examine the antecedents and effects of higher-level constructs. Typically, researchers present three indices, r wg, ICC(1), and ICC(2), to demonstrate agreement and consistency among lower-level units when justifying aggregation. Nevertheless, researchers debate what values for these indices are sufficient. This study examines the distributional characteristics of ICCs and r wg values from three sources: the multilevel literature, a large multinational sample of student teams, and a large sample of randomly generated “pseudo teams.” Our results support existing cutoff criteria for ICCs but suggest that generally accepted values for r wg may, under certain circumstances, reflect pseudo-agreement (i.e., agreement observed among two raters not attributable to the same target). Thus, when there is minimal between-group variance (i.e., low ICCs), it is difficult to determine whether high r wg values reflect agreement or pseudo-agreement. Based on these findings, we provide recommendations to help researchers interpret aggregation indices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating within-group interrater reliability with and without response bias
James, L. R., Demaree, R. G., & Wolf, G. (1984) · Journal of Applied Psychology · doi:10.1037/0021-9010.69.1.85 · Cited by 4,774
A general multilevel SEM framework for assessing multilevel mediation
Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010) · Psychological Methods · doi:10.1037/a0020141 · Cited by 3,796
Answers to 20 questions about interrater reliability and interrater agreement
LeBreton, J. M., & Senter, J. L. (2008) · Organizational Research Methods · doi:10.1177/1094428106296642 · Cited by 3,608 · cited 17×
“This could include such things as the types of decisions being made (e.g., high vs. low stakes), the quality of their measures, restrictions of range in their data, the novelty of their construct, and the characteristics of their sample, such as group size (LeBreton & Senter, 2008).”
BackgroundMethodResultInfluentialTask versus relationship conflict, team performance, and team member satisfaction: A meta-analysis.
De Dreu, C. K. W., & Weingart, L. R. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.4.741 · Cited by 3,071 · cited 1×
“Also, the negative association between both forms of conflict and a team’s satisfaction and cohesion has been supported across a number of studies (De Dreu & Weingart, 2003; de Wit et al., 2012).”
BackgroundTHE DYNAMIC NATURE OF CONFLICT: A LONGITUDINAL STUDY OF INTRAGROUP CONFLICT AND GROUP PERFORMANCE.
Jehn, K. A., & Mannix, E. A. (2001) · Academy of Management Journal · doi:10.2307/3069453 · Cited by 2,487
Aggregation bias in estimates of perceptual agreement.
James, L. R. (1982) · Journal of Applied Psychology · doi:10.1037/0021-9010.67.2.219 · Cited by 2,139
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053
Levels issues in theory development, data collection and analysis
Klein, K. J., Dansereau, F., & Hall, R. J. (1994) · The Academy of Management Review · doi:10.2307/258703 · Cited by 1,607 · cited 1×
“When employing composition models, whether they are referent-shift or direct consensus, researchers must ensure that their higher-level constructs are grounded in a strong theoretical framework (Klein et al., 1994, 1999; Klein & Kozlowski, 2000a).”
MethodPeterson, R. S., Smith, D. B., Martorana, P. V., & Owens, P. D. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.5.795 · Cited by 654
Group Size, ICC Values, and Group-Level Correlations: A Simulation
Bliese, P. D. (1998) · Organizational Research Methods · doi:10.1177/109442819814001 · Cited by 636 · cited 2×
“This consistency is often assessed via two measures—ICC(1) and ICC(2) (Bliese, 1998, 2000).”
BackgroundMethod
Cited by
Metacritiques of Upper Echelons Theory: Verdicts and Recommendations for Future Research
Neely, B. H., Lovelace, J. B., Cowen, A. P., & Hiller, N. J. (2020) · Journal of Management · doi:10.1177/0149206320908640 · Cited by 400
Careless responding in questionnaire measures: Detection, impact, and remedies
Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101384 · Cited by 200
“…raters show high agreement with their co-raters on the target (indicated by a high r wg(j) value [e.g., above 0.70] and/or when the average within-group variance of the real groups is smaller than that of the randomly drawn pseudo-groups) (e.g., Woehr, Loignon, Schmidt, Loughry, & Ohland, 2015).”
BackgroundInfluentialBardus, M., Ali, A., Demachkieh, F., & Hamadeh, G. (2019) · JMIR mHealth and uHealth · doi:10.2196/mhealth.9836 · Cited by 44
(2017) · Saint Mary's University Institutional Repository (Saint Mary's University) · Cited by 10
Metadata
- Authors
- Woehr, D. J., Loignon, A. C., Schmidt, P. B., Loughry, M. L., & Ohland, M. W.
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2015-05-05
- Volume
- 18
- Issue
- 4
- Pages
- 704-737
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W1925128518
- Cited by
- 218
- References
- 10
- FWCI
- 8.97
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Justifying Aggregation With Consensus-Based Constructs: A Review and Examination of Cutoff Values for Common Aggregation Indices. https://app.cochat.ai/papers/10.1177/1094428115582090
- The paper
- Woehr, D. J., Loignon, A. C., Schmidt, P. B., Loughry, M. L., & Ohland, M. W. (2015). Justifying Aggregation With Consensus-Based Constructs: A Review and Examination of Cutoff Values for Common Aggregation Indices. Organizational Research Methods. https://doi.org/10.1177/1094428115582090
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.