The Other Side of Method Bias: The Perils of Distinct Source Research Designs
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Common source bias has been the focus of much attention. To minimize the problem, researchers have sometimes been advised to take measurements of predictors from one observer and measurements of outcomes from another observer or to use separate occasions of measurement. We propose that these efforts to eliminate biases due to common source variance create serious problems. To demonstrate the problems of using what we term the "distinct sources" measurement design, we provide an integrative review of the literature regarding both contamination and deficiency of measures. Building on this theme, the article uses simulated data to demonstrate how using data from distinct observers or occasions of measurement can distort estimates of predictor importance at least as much as common source variance. Alternative multisource designs are advocated and examined for tractability by simulating various numbers of observations and sources in the research design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.5.879 · Cited by 82,194
Flawed Self-Assessment: Implications for Health, Education, and the Workplace
Dunning, D., Heath, C., & Suls, J. M. (2004) · Psychological Science in the Public Interest · doi:10.1111/j.1529-1006.2004.00018.x · Cited by 1,865
Janssen, O., & Van Yperen, N. W. (2004) · Academy of Management Journal · doi:10.2307/20159587 · Cited by 1,623 · cited 1×
“One primary use of DS design has been a combination of self-reports of personality or knowledge alongside supervisors rating their subordinates' behavior, clinicians rating patient outcomes, or teachers rating students (eg, Janssen & Van Yperen, 2004 ; Mayer & Gavin, 2005 ; Piccolo & Colquitt, 2006 ; Rubin, Munz, & Bo…”
BackgroundCommon methods bias: does common methods variance really bias results?
Doty, D. H., & Glick, W. H. (1998) · Organizational Research Methods · doi:10.1177/109442819814002 · Cited by 1,380 · cited 4×
“Meta-analytic research by Doty and Glick (1998) found 32% of variance in scores across a variety of measures could be attributed to sources of measurement.”
BackgroundMethodResultInfluentialTrust in Management and Performance: Who Minds the Shop While the Employees Watch the Boss?
Mayer, R. C., & Gavin, M. B. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.18803928 · Cited by 1,200
Prognostic Value of Ambulatory Blood-Pressure Recordings in Patients with Treated Hypertension
Clement, D. L., De Buyzere, M. L., De Bacquer, D. A., de Leeuw, P. W., Duprez, D. A., Fagard, R. H., Gheeraert, P. J., Missault, L. H., Braun, J. J., Six, R. O., et al. (2003) · New England Journal of Medicine · doi:10.1056/nejmoa022273 · Cited by 1,085 · cited 1×
“…distorted due to time of day or nervousness on the part of the patient; patients in one study were fitted with an ambulatory monitoring device that recorded their blood pressure automatically every 30 to 60 min over the course of a day to show “trait” blood pressure levels (Clement et al., 2003).”
BackgroundCommittee on Quality Improvement, S. O. A.-D. D. (2000) · Pediatrics · doi:10.1542/peds.105.5.1158 · Cited by 1,076
Rubin, R. S., Munz, D. C., & Bommer, W. H. (2005) · Academy of Management Journal · doi:10.5465/amj.2005.18803926 · Cited by 576
Service with A Smile and Encounter Satisfaction: Emotional Contagion and Appraisal Mechanisms
Barger, P. B., & Grandey, A. A. (2006) · Academy of Management Journal · doi:10.5465/amj.2006.23478695 · Cited by 503 · cited 2×
“For example, it is standard to have the quality of service interactions (eg, server friendliness) rating by multiple independent coders (eg, Barger & Grandey, 2006 ; Doucet, 2004 ; Tan, Foo, & Kwek, 2004 ), and in some studies repeated measures of attitudes are related to turnover behavior ( Kammeyer-Mueller, Wanberg,…”
BackgroundImplementing workplace‐based assessment across the medical specialties in the United Kingdom
Wilkinson, J. R., Crossley, J. G. M., Wragg, A., Mills, P., Cowan, G., & Wade, W. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03010.x · Cited by 272
Cited by
Taras, V., Kirkman, B. L., & Steel, P. (2010) · Journal of Applied Psychology · doi:10.1037/a0018938 · Cited by 1,160
Self-citationHulland, J., Baumgartner, H., & Smith, K. M. (2018) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-017-0532-y · Cited by 1,067
“For example, Kammeyer-Mueller et al. (2010) show using simulated data that under some conditions using distinct data sources can distort estimation.”
MethodAcademic procrastination: psychological antecedents revisited
Steel, P., & Klingsieck, K. B. (2016) · Australian Psychologist · doi:10.1111/ap.12173 · Cited by 534
Self-citationGeorge, B., & Pandey, S. K. (2017) · Review of Public Personnel Administration · doi:10.1177/0734371x17698189 · Cited by 421
Lee, A., Lyubovnikova, J., Tian, A. W., & Knight, C. (2020) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12265 · Cited by 347
Metadata
- Authors
- Kammeyer-Mueller, J., Steel, P. D. G., & Rubenstein, A.
- Venue
- Multivariate Behavioral Research
- Publisher
- Informa UK Limited
- Published
- 2010-03-31
- Volume
- 45
- Issue
- 2
- Pages
- 294-321
- ISSN
- 0027-3171 (ISSN-L)
- 1532-7906
- OpenAlex
- W2056241146
- Cited by
- 43
- References
- 11
- FWCI
- 2.62
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Other Side of Method Bias: The Perils of Distinct Source Research Designs. https://app.cochat.ai/papers/10.1080/00273171003680278
- The paper
- Kammeyer-Mueller, J., Steel, P. D. G., & Rubenstein, A. (2010). The Other Side of Method Bias: The Perils of Distinct Source Research Designs. Multivariate Behavioral Research. https://doi.org/10.1080/00273171003680278
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.