Cited by 344Statistical Methods and Bayesian InferencePsychometric Methodologies and TestingSurvey Methodology and Nonresponse

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

Missing Data in Multiple Item Scales: A Monte Carlo Analysis of Missing Data Techniques

Roth, Philip L., Switzer, Fred S., Switzer, Deborah M.·1999·Organizational Research Methods

DOI 10.1177/109442819923001

Open this paper in CoChat — free account

Abstract

Researchers in many fields use multiple item scales to measure important variables such as attitudes and personality traits, but find that some respondents failed to complete certain items. Past missing data research focuses on missing entire instruments, and is of limited help because there are few variables to help impute missing scores and the variables are often not highly related to each other. Multiple item scales offer the unique opportunity to impute missing values from other correlated items designed to measure the same construct. A Monte Carlo analysis was conducted to compare several missing data techniques. The techniques included listwise deletion, regression imputation, hot-deck imputation, and two forms of mean substitution. Results suggest that regression imputation and substituting the mean response of a person to other items on a scale are very promising approaches. Furthermore, the imputation techniques often outperformed listwise deletion.

Integrity signals

Registry

Crossref · 10.1177/109442819923001

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Using multivariate statistics
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 79,882
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META‐ANALYSIS
BARRICK, M. R., & MOUNT, M. K. (1991) · Personnel Psychology · doi:10.1111/j.1744-6570.1991.tb00688.x · Cited by 8,689
Job Satisfaction and Organizational Commitment as Predictors of Organizational Citizenship and In-Role Behaviors
Williams, L. J., & Anderson, S. E. (1991) · Journal of Management · doi:10.1177/014920639101700305 · Cited by 6,380
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Statistical Analysis With Missing Data
Ghosh, S., Little, R. J. A., & Rubin, D. B. (1988) · Technometrics · doi:10.2307/1269814 · Cited by 4,730
Maximum likelihood estimation from incomplete data via the EM algorithm
Dempster, A. P. (1977) · Journal of the Royal Statistical Society Series A (Statistics in Society) · Cited by 4,543
Role Conflict and Ambiguity in Complex Organizations
Rizzo, J. R., House, R. J., & Lirtzman, S. I. (1970) · Administrative Science Quarterly · doi:10.2307/2391486 · Cited by 3,700
A Review of Scale Development Practices in the Study of Organizations
Hinkin, T. R. (1995) · Journal of Management · doi:10.1177/014920639502100509 · Cited by 3,193

Show all 39 references

Research works printed but not linked: 12

Bailar, B. A., 1982, Incomplete data in sample surveys: Proceedings of the symposium, 299
Baker, K., 1989, Journal of Market Research Society, 31, 153
Dawes, R. M., 1983, Judgment under uncertainty: Heuristics & biases
Ford, B. L., 1983, Incomplete data in sample surveys. Volume II: Theory and bibliographies, 185
Goodman, J. S., 1996, Journal of Management-Research Methods & Analysis, 22, 627
Hackman, J. R., 1980, Work redesign
Hunter, J. E., Ninth Annual Conference for Industrial and Organizational Psychology
Madow, W. G., 1983, Incomplete data in sample surveys. Volume I: Report and case studies
Madow, W. G., 1983, Incomplete data in sample surveys. Volume I: Report and case studies, 3
McDonald, R. A., Southern Academy of Management meetings
Rizvi, M. H., 1983, Incomplete data in sample surveys. Volume III: Proceedings of the symposium, 351
Roth, P. L., Journal of Business & Psychology

Sorted by citations

Psychometric validity of the 22‐item Sinonasal Outcome Test
Hopkins, C. L., Gillett, S., Slack, R., Lund, V. J., & Browne, J. P. (2009) · Clinical Otolaryngology · doi:10.1111/j.1749-4486.2009.01995.x · Cited by 1,647
Missing Data: Five Practical Guidelines
Newman, D. A. (2014) · Organizational Research Methods · doi:10.1177/1094428114548590 · Cited by 1,411
“This is like the practice that Roth, Switzer, and Switzer (1999) recommend, which they referred to as ‘‘mean substitution across items (and within an individual)’’ or ‘‘meanperson imputation’’ (pp. 214, 222; also see Downey & King, 1998), although using the technique as I am describing it here…”
BackgroundMethod
Clarifying the conditions and limits of the contributions of ordinary and dynamic capabilities to relative firm performance
Drnevich, P. L., & Kriauciunas, A. P. (2011) · Strategic Management Journal · doi:10.1002/smj.882 · Cited by 681
Research Report: Richness Versus Parsimony in Modeling Technology Adoption Decisions—Understanding Merchant Adoption of a Smart Card-Based Payment System
Plouffe, C. R., Hulland, J. S., & Vandenbosch, M. (2001) · Information Systems Research · doi:10.1287/isre.12.2.208.9697 · Cited by 690
“To retain these cases for further analysis, we followed the data imputation procedures recommended by Hair et al. (1995) and Roth et al. (1999).”
Method
Absorptive capacity, its determinants, and influence on innovation output: Cross-cultural validation of the structural model
Murovec, N., & Prodan, I. (2009) · Technovation · doi:10.1016/j.technovation.2009.05.010 · Cited by 474
Goodbye, Listwise Deletion: Presenting Hot Deck Imputation as an Easy and Effective Tool for Handling Missing Data
Myers, T. A. (2011) · Communication Methods and Measures · doi:10.1080/19312458.2011.624490 · Cited by 459
“…(2) imputations will not be outside the range of possible values (as might happen with multiple imputations, see He, 2010); and (3) it is not necessary to deﬁne an explicit model for the distribution of the missing val-ues” (p. 84; see also Andridge & Little, 2010; Roth, Switzer, & Switzer, 1999).”
Background
Racial differences in employee retention: Are diversity climate perceptions the key?
McKAY, P. F., AVERY, D. R., TONIDANDEL, S., MORRIS, M. A., HERNANDEZ, M., & HEBL, M. R. (2007) · Personnel Psychology · doi:10.1111/j.1744-6570.2007.00064.x · Cited by 561
Engaging the aging workforce: The relationship between perceived age similarity, satisfaction with coworkers, and employee engagement.
Avery, D. R., McKay, P. F., & Wilson, D. C. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.6.1542 · Cited by 422
MEAN RACIAL‐ETHNIC DIFFERENCES IN EMPLOYEE SALES PERFORMANCE: THE MODERATING ROLE OF DIVERSITY CLIMATE
Mckay, P. F., Avery, D. R., & Morris, M. A. (2008) · Personnel Psychology · doi:10.1111/j.1744-6570.2008.00116.x · Cited by 425
Handling missing data in self-report measures.
Fox-Wasylyshyn, S. M., & El-Masri, M. (2005) · Research in Nursing and Health · doi:10.1002/nur.20100 · Cited by 389
Imputing Cross-Sectional Missing Data: Comparison of Common Techniques
Hawthorne, G., Hawthorne, G., & Elliott, P. (2005) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/j.1440-1614.2005.01630.x · Cited by 382
“The findings suggest that hot deck imputation and person mean substitution (with approximately half the items present) are superior to the other procedures – a finding consistent with that of earlier researchers [38].”
Result
In the Eyes of the Beholder
Gooty, J., Gavin, M., Johnson, P. D., Frazier, M. L., & Snow, D. B. (2009) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051809332021 · Cited by 398
BREACH AND FULFILLMENT OF THE PSYCHOLOGICAL CONTRACT: A COMPARISON OF TRADITIONAL AND EXPANDED VIEWS
Lambert, L. S., Edwards, J. R., & Cable, D. M. (2003) · Personnel Psychology · doi:10.1111/j.1744-6570.2003.tb00244.x · Cited by 345
“When a psychological contract is violated, employees may experience anger, distrust, reduced loyalty and commitment, and increased propensity to leave the organization (Robinson, 1996; Robinson & Morrison, 2o00, Robinson & Rousseau, 1994 ).”
BackgroundMethodResultInfluential
An Empirical Study of Consumer Switching from Traditional to Electronic Channels: A Purchase-Decision Process Perspective
Gupta, A., Su, B.-C., & Walter, Z. D. (2004) · International Journal of Electronic Commerce · doi:10.1080/10864415.2004.11044302 · Cited by 331
“94 for factors measured by three to seven items [80].”
Background
A review of techniques for treating missing data in OM survey research
Tsikriktsis, N. (2005) · Journal of Operations Management · doi:10.1016/j.jom.2005.03.001 · Cited by 351
“Second, missing data may result in biased estimates (Madlow et al., 1983; Roth et al., 1999) in several ways.”
BackgroundMethodInfluential
Global-scale drought risk assessment for agricultural systems
Meza, I., Siebert, S., Döll, P., Kusche, J., Herbert, C., Rezaei, E. E., Nouri, H., Gerdener, H., Popat, E., Frischen, J., et al. (2020) · Natural hazards and earth system sciences · doi:10.5194/nhess-20-695-2020 · Cited by 283
Relationships at school and stage‐environment fit as resources for adolescent engagement and achievement
Zimmer‐Gembeck, M. J., Chipuer, H. M., Hanisch, M., Creed, P. A., & McGregor, L. (2006) · Journal of Adolescence · doi:10.1016/j.adolescence.2006.04.008 · Cited by 294
“In one simulation study using Monte Carlo analysis to vary the number of items on measures, the correlation between items, and the pattern of missing data (Roth et al., 1999), replacement of missing data with the mean of items on the scale (‘‘mean substitution across individuals’’, p.”
Method
Terror management and religion: Evidence that intrinsic religiousness mitigates worldview defense following mortality salience.
Jonas, E., & Fischer, P. (2006) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.91.3.553 · Cited by 367
The role of self-care on compassion satisfaction, burnout and secondary trauma among child welfare workers
Salloum, A., Kondrat, D. C., Johnco, C. J., & Olson, K. R. (2015) · Children and Youth Services Review · doi:10.1016/j.childyouth.2014.12.023 · Cited by 304
Missing data in a multi-item instrument were best handled by multiple imputation at the item score level
Eekhout, I., de Vet, H. C. W., Twisk, J. W. R., Brand, J. P. L., de Boer, M. R., & Heymans, M. W. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.09.009 · Cited by 302
The role of control motivation in mortality salience effects on ingroup support and defense
Fritsche, I., Jonas, E., & Fankhänel, T. (2008) · Journal of Personality and Social Psychology · doi:10.1037/a0012666 · Cited by 252
A TALE OF TWO CLIMATES: DIVERSITY CLIMATE FROM SUBORDINATES’ AND MANAGERS’ PERSPECTIVES AND THEIR ROLE IN STORE UNIT SALES PERFORMANCE
Mckay, P. F., Avery, D. R., & Morris, M. A. (2009) · Personnel Psychology · doi:10.1111/j.1744-6570.2009.01157.x · Cited by 244
Multiple Imputation for Missing Data: Making the Most of What You Know
Fichman, M., & Cummings, J. N. (2003) · Organizational Research Methods · doi:10.1177/1094428103255532 · Cited by 229
“hold for most other statistical techniques in wide use today (Rubin, 1987; Schafer, 1997). Of course, it is always best to have no missing data and design research to minimize the probability of missing data. However, even with best efforts, missing data can and will occur. We also must always understand that there is…”
Background
Stigma as a barrier to recognizing personal mental illness and seeking help: a prospective study among untreated persons with mental illness
Schomerus, G., Stolzenburg, S., Freitag, S., Speerforck, S., Janowitz, D., Evans-Lacko, S., Muehlan, H., & Schmidt, S. (2019) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-018-0896-0 · Cited by 214
“For all scales, missing values of total scores were imputed by individual participant mean if no more than 25% of items were missing [42,43], whch was necessary in 1-4 participants/scale.”
Method
Does Diversity Climate Lead to Customer Satisfaction? It Depends on the Service Climate and Business Unit Demography
McKay, P. F., Avery, D. R., Liao, H., & Morris, M. A. (2010) · Organization Science · doi:10.1287/orsc.1100.0550 · Cited by 207
“This imputation method provides robust results with up to 20% missing responses (Roth et al. 1999).”
Method
Empirically supported ethical research practice: The costs and benefits of research from the participants’ view
Newman, E., Willard, T., Sinclair, R. R., & Kaloupek, D. G. (2001) · Accountability in Research · doi:10.1080/08989620108573983 · Cited by 180
Organizational Justice, Trustworthiness, and Trust: A Multifoci Examination
Frazier, M. L., Johnson, P. D., Gavin, M., Gooty, J., & Snow, D. B. (2009) · Group & Organization Management · doi:10.1177/1059601109354801 · Cited by 162
Exploratory Factor Analysis With Small Samples and Missing Data
McNeish, D. (2016) · Journal of Personality Assessment · doi:10.1080/00223891.2016.1252382 · Cited by 152
“Roth, Switzer, and Switzer (1999) noted this deficiency over 15 years ago by stating, “We strongly urge future research on varying sample sizes … we wonder if the quality of estimates of missing data would change for smaller Ns and comparisons of imputation techniques to deletion techniques would…”
Background
Stress, Marital Satisfaction, and Psychological Distress Among African Americans
Lincoln, K. D., & Chae, D. H. (2010) · Journal of Family Issues · doi:10.1177/0192513x10365826 · Cited by 141
Stewardship Climate Scale
Neubaum, D. O., Thomas, C. H., Dibrell, C., & Craig, J. B. L. (2016) · Family Business Review · doi:10.1177/0894486516673701 · Cited by 124
Minority Stressors, Rumination, and Psychological Distress in Lesbian, Gay, and Bisexual Individuals
Timmins, L., Rimes, K. A., & Rahman, Q. (2019) · Archives of Sexual Behavior · doi:10.1007/s10508-019-01502-2 · Cited by 141
An examination of the 3‐factor model and structural invariance across racial/ethnic groups for the FACIT‐Sp: a report from the American Cancer Society's Study of Cancer Survivors‐II (SCS‐II)
Murphy, P. E., Canada, A. L., Fitchett, G. A., Stein, K., Portier, K. M., Crammer, C., & Peterman, A. H. (2009) · Psycho-Oncology · doi:10.1002/pon.1559 · Cited by 120
“In the analysis of a scale like the FACIT-Sp, which uses multiple items to measure constructs, and with a large sample, imputation of missing data by regressing the missing item on the other items in a given scale yields good parameter estimates [13].”
The association between education outside the classroom and students’ school motivation: Results from a one-school-year quasi-experiment
Bølling, M., Otte, C. R., Elsborg, P., Nielsen, G., & Bentsen, P. (2018) · International Journal of Educational Research · doi:10.1016/j.ijer.2018.03.004 · Cited by 143
Salesperson social media use in business-to-business relationships: An empirical test of an integrative framework linking antecedents and consequences
Bill, F., Feurer, S., & Klarmann, M. (2020) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00708-z · Cited by 110
“approach often outperforms list-wise deletion and other techniques (Roth et al. 1999).”
MethodInfluential
Doing More with Less: The Disordinal Implications of Firm Age for Leveraging Capabilities for Innovation Activity*
Withers, M. C., Drnevich, P. L., & Marino, L. (2011) · Journal of Small Business Management · doi:10.1111/j.1540-627x.2011.00334.x · Cited by 104
Racial Discrimination and Racial Identity Attitudes in Relation to Self-Rated Health and Physical Pain and Impairment Among Two-Spirit American Indians/Alaska Natives
Chae, D. H., & Walters, K. L. (2009) · American Journal of Public Health · doi:10.2105/ajph.2007.126003 · Cited by 106
“Analyses Missing data on items assessing discrimination and actualization were first handled with a mean-substitution technique: the withinparticipant mean value of remaining items was used to substitute for missing values in cases in which at least 80% of items were complete.(61) After we applied this technique, the…”
Associations of Cigarette Smoking With Subclinical Inflammation and Atherosclerosis: ELSA‐Brasil (The Brazilian Longitudinal Study of Adult Health)
Kianoush, S., Yakoob, M. Y., Al‐Rifai, M., DeFilippis, A. P., Bittencourt, M. S., Duncan, B. B., Benseñor, I. M., Bhatnagar, A., Lotufo, P. A., & Blaha, M. J. (2017) · Journal of the American Heart Association · doi:10.1161/jaha.116.005088 · Cited by 104
“In sensitivity analyses, we performed Monte Carlo multiple imputation for missing observations of continuous covariates and missing indicator terms for categorical covariates.(35) We also repeated the analyses defining the ABI outcome as ABI ≤1.”
Method
Discrimination, attribution, and racial group identification: Implications for psychological distress among Black Americans in the National Survey of American Life (2001–2003).
Chae, D. H., Lincoln, K. D., & Jackson, J. S. (2011) · American Journal of Orthopsychiatry · doi:10.1111/j.1939-0025.2011.01122.x · Cited by 104
ADHD Symptoms, Entrepreneurial Orientation (EO), and Firm Performance
Yu, W., Wiklund, J., & Pérez‐Luño, A. (2019) · Entrepreneurship Theory and Practice · doi:10.1177/1042258719892987 · Cited by 112
“To make maximum use of the data, we imputed missing values if a maximum of two item values were missing for the ADHD and EO scale and if a maximum of one item was missing for the subjective performance scale (Downey & King, 1998; Roth et al., 1999).”
Method
Predictors of secondary traumatic stress among social workers: Supervision, income, and caseload size
Quinn, A. E., Ji, P., & Nackerud, L. G. (2018) · Journal of Social Work · doi:10.1177/1468017318762450 · Cited by 106
Using Market Basket Analysis in Management Research
Aguinis, H., Forcum, L. E., & Joo, H. (2012) · Journal of Management · doi:10.1177/0149206312466147 · Cited by 97
“For example, Russell et al. (1999) pointed out that the use of MBA has allowed marketing researchers to build theoretical models of purchasing decisions involving products in more than one category (i.”
Background
Psychometric properties of the Functional Assessment of Chronic Illness Therapy—Spiritual Well‐being (FACIT‐Sp) in an Arabic‐speaking, predominantly Muslim population
Lazenby, M., Khatib, J., Al-Khair, F. A., & Neamat, M. (2011) · Psycho-Oncology · doi:10.1002/pon.2062 · Cited by 92
“Data were cleaned; of those subscales missing one item, the missing data were imputed by summing the other items of the same subscale and using the mean, which in a scale that uses multiple items to measure constructs, yields good parameter estimates [19].”
Method
Consumer Racial Discrimination in Tipping: A Replication and Extension
Lynn, M., Sturman, M. C., Ganley, C., Adams, E. A., Douglas, M., & McNeil, J. (2008) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2008.00338.x · Cited by 86
“This technique (Mean SubstitutionPerson) has been shown to be a highly effective missing data technique, and is therefore appropriate given our desire to maintain power and variability of the measures (Roth et al., 1999).”
Method
Evaluation of the principal-component and expectation-maximization methods for estimating missing data in morphometric studies
Strauss, R. E., Atanassov, M. N., & de Oliveira, J. A. (2003) · Journal of Vertebrate Paleontology · doi:10.1671/0272-4634(2003)023[0284:eotpae]2.0.co;2 · Cited by 88
“Although some comparative Monte Carlo studies have been done to compare methods, these studies generally (1) concern types of data (e.g., Roth et al., 1999) or methods (e.g., Bello, 1993a, b; Liu, 1997) that are not relevant to morphometric studies; (2) have been aimed at evaluating the effects of…”
Method
Association of Education Outside the Classroom and Pupils' Psychosocial Well‐Being: Results From a School Year Implementation
Bølling, M., Niclasen, J., Bentsen, P., & Nielsen, G. (2019) · Journal of School Health · doi:10.1111/josh.12730 · Cited by 109
A review on missing hydrological data processing
Gao, Y., Merz, C., Lischeid, G., & Schneider, M. (2018) · Environmental Earth Sciences · doi:10.1007/s12665-018-7228-6 · Cited by 86
Conceptualising market orientation in non-profit organisations: definition, performance, and preliminary construction of a scale
Modi, P., & Mishra, D. (2010) · Journal of Marketing Management · doi:10.1080/02672570903485113 · Cited by 82
“Roth, Switzer, and Switzer ( 1999 ) recommend using either the regression technique or the technique of using a person's mean response for other items on a scale to impute values to missing data.”
Method
Determining best methods to screen for religious/spiritual distress
King, S. D. W., Fitchett, G. A., Murphy, P. E., Pargäment, K. I., Harrison, D. A., & Loggers, E. T. (2016) · Supportive Care in Cancer · doi:10.1007/s00520-016-3425-6 · Cited by 85
“For these cases, regression equations were used to predict parameter estimates for each of the missing items based on the responses to the remaining NRC items [35].”
Transition Readiness Skills Acquisition in Adolescents and Young Adults with Inflammatory Bowel Disease
Gray, W. N., Holbrook, E., Morgan, P. J., Saeed, S. A., Denson, L. A., & Hommel, K. A. (2015) · Inflammatory Bowel Diseases · doi:10.1097/mib.0000000000000352 · Cited by 82
Maternal Dysphoric Mood, Stress, and Parenting Practices in Mothers of Head Start Preschoolers: The Role of Experiential Avoidance
Shea, S. E., & Coyne, L. W. (2011) · Child & Family Behavior Therapy · doi:10.1080/07317107.2011.596004 · Cited by 88
Spiritual or religious struggle in hematopoietic cell transplant survivors
King, S. D. W., Fitchett, G. A., Murphy, P. E., Pargäment, K. I., Martin, P. J., Johnson, R. H., Harrison, D. A., & Loggers, E. T. (2015) · Psycho-Oncology · doi:10.1002/pon.4029 · Cited by 80
“For these cases, regression equations were used to predict parameter estimates for each of the missing items based on the responses to the remaining NRC items [36].”
Method
LUMP‐SUM BONUS SATISFACTION: TESTING THE CONSTRUCT VALIDITY OF A NEW PAY SATISFACTION DIMENSION
Sturman, M. C., & Short, J. C. (2000) · Personnel Psychology · doi:10.1111/j.1744-6570.2000.tb00218.x · Cited by 79
Discrimination, mental health, and leukocyte telomere length among African American men
Chae, D. H., Epel, E. S., Nuru‐Jeter, A., Lincoln, K. D., Taylor, R. J., Lin, J., Blackburn, E. H., & Thomas, S. B. (2015) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2015.09.001 · Cited by 79
“Mean substitution in cases where at least 80% of items are completed is considered to be a robust technique for handling missing data on multi-item scales (Roth et al., 1999).”
Method
Probabilistic neural network based categorical data imputation
Nishanth, K. J., & Ravi, V. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.08.044 · Cited by 78
The association of depressive symptoms and diabetes distress with glycaemic control and diabetes complications over 2 years in newly diagnosed type 2 diabetes: a prospective cohort study
Ismail, K., Moulton, C. D., Winkley, K., Pickup, J. C., Thomas, S. M., Sherwood, R. A., Stahl, D. R., & Amiel, S. A. (2017) · Diabetologia · doi:10.1007/s00125-017-4367-3 · Cited by 82
“Case means substitution has been found to be robust in handling item-level missingness when 20% of data are missing, whether in random or systematic patterns [35].”
Background
How Do Missing Data Bias Estimates of Within-Group Agreement? Sensitivity of SD WG, CVWG, rWG(J), rWG(J) * , and ICC to Systematic Nonresponse
Newman, D., & Sin, H.-P. (2009) · Organizational Research Methods · doi:10.1177/1094428106298969 · Cited by 94
Patient activation with knowledge, self‐management and confidence in chronic kidney disease
Johnson, M. L., Zimmerman, L. M., Welch, J. L., Hertzog, M. A., Pozehl, B., & Plumb, T. J. (2015) · Journal of Renal Care · doi:10.1111/jorc.12142 · Cited by 75
Imputing Missing Repeated Measures Data: How Should We Proceed?
Elliott, P., & Hawthorne, G. (2005) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/j.1440-1614.2005.01629.x · Cited by 72
“Closest match replaces a case i’s missing datum with a value obtained from one or more participants who have similar scores on the same measure assessed at other timepoints [20,22,23].”
Background
Missing data: an introductory conceptual overview for the novice researcher.
El‐Masri, M. M., & Fox-Wasylyshyn, S. M. (2005) · PubMed · Cited by 71
Immigration status and bullying victimization: Associations across national and school contexts
Stevens, G. W. J. M., Boer, M., Titzmann, P. F., Cosma, A. P., & Walsh, S. D. (2019) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2019.101075 · Cited by 83
Imputation methods for recovering streamflow observation: A methodological review
Razali, S. F. M., Razali, S. F. M., Razali, S. F. M., Jaafar, O., & Jamil, N. A. (2020) · Sustainable Environment · doi:10.1080/23311843.2020.1745133 · Cited by 69
“These missing value(s) in the time series usually lessens the ability and the accuracy of statistical analysis approaches (Roth et al., 1999) and initiate biased estimations of the relationship among variables (Pigott, 2001).”
Background
Missing data analysis with fuzzy C-Means: A study of its application in a psychological scenario
Nuovo, A. G. D. (2010) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.12.067 · Cited by 69
An investigation of project complexity's influence on team communication using Monte Carlo simulation
Kennedy, D., McComb, S., & Vozdolska, R. R. (2011) · Journal of Engineering and Technology Management · doi:10.1016/j.jengtecman.2011.03.001 · Cited by 60
Individuals with currently untreated mental illness: causal beliefs and readiness to seek help
Stolzenburg, S., Freitag, S., Evans-Lacko, S. E., Speerforck, S., Schmidt, S., & Schomerus, G. (2018) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796017000828 · Cited by 65
“For all scales, missing values of total scores were imputed by individual participant mean if no more than 25% of items were missing (Downey & King, 1998; Roth et al. 1999).”
Method
Effects of different types of exercise on muscle mass, strength, function and well‐being in elderly
Solberg, P. A., Kvamme, N. H., Raastad, T., Ommundsen, Y., Tomten, S. E., Halvari, H., Loland, N. W., & Hallén, J. (2011) · European Journal of Sport Science · doi:10.1080/17461391.2011.617391 · Cited by 70
“The few missing values in the psychological measurements ( 1999 ).”
Background
What drives continuous improvement project success in healthcare?
Stelson, P., Hille, J. D., Eseonu, C. I., & Doolen, T. L. (2017) · International Journal of Health Care Quality Assurance · doi:10.1108/ijhcqa-03-2016-0035 · Cited by 74
“This approach provides accurate scale averages and does not introduce significant bias due to missing data (Brick and Kalton, 1996; Roth et al., 1999).”
Background
Factors associated with safe patient handling behaviors among critical care nurses
Lee, S., Faucett, J. A., Gillen, M., Krause, N., & Landry, L. G. (2010) · American Journal of Industrial Medicine · doi:10.1002/ajim.20843 · Cited by 60
The Relationships Between Self-Compassion, Attachment and Interpersonal Problems in Clinical Patients with Mixed Anxiety and Depression and Emotional Distress
Mackintosh, K., Power, K., Schwannauer, M., & Chan, S. W. Y. (2017) · Mindfulness · doi:10.1007/s12671-017-0835-6 · Cited by 60
“Various methods for imputing missing data have been described in the literature (e.g. Fox-Wasylyshyn and El-Masri 2005; Roth et al. 1999; Shrive et al. 2006).”
Method
Factors in the practice environment of nurses working in inpatient mental health: A partial least squares path modeling approach
Roche, M. A., Duffield, C. M., & White, E. G. (2011) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.07.001 · Cited by 59
The potential role of actively open-minded thinking in preventing motivated reasoning about controversial science
Stenhouse, N., Myers, T. A., Vraga, E. K., Kotcher, J., Beall, L., & Maibach, E. W. (2018) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2018.06.001 · Cited by 67
Explanations of service gratuities and tipping: Evidence from individual differences in tipping motivations and tendencies
Lynn, M. (2015) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2015.01.002 · Cited by 57
Racial/ethnic differences in spiritual well-being among cancer survivors
Canada, A. L., Fitchett, G. A., Murphy, P. E., Stein, K., Portier, K. M., Crammer, C., & Peterman, A. H. (2012) · Journal of Behavioral Medicine · doi:10.1007/s10865-012-9439-8 · Cited by 61
Recognizing vocal expressions of emotion in patients with social skills deficits following traumatic brain injury
Dimoska, A., McDonald, S., Pell, M. D., Tate, R. L., & James, C. (2010) · Journal of the International Neuropsychological Society · doi:10.1017/s1355617709991445 · Cited by 60
The intersection of professionalism and interprofessional care: development and initial testing of the interprofessional professionalism assessment (IPA)
Frost, J. S., Hammer, D. P., Nunez, L. M., Adams, J. L., Chesluk, B. J., GRUS, C. L., Harvison, N., McGuinn, K., Mortensen, L. H., Nishimoto, J. H., et al. (2018) · Journal of Interprofessional Care · doi:10.1080/13561820.2018.1515733 · Cited by 56
A Prospective Study on Structural and Attitudinal Barriers to Professional Help-Seeking for Currently Untreated Mental Health Problems in the Community
Tomczyk, S., Schmidt, S., Muehlan, H., Stolzenburg, S., & Schomerus, G. (2019) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-019-09662-8 · Cited by 65
“Statistical analysis If a person had up to 25% missing values per scale, scale values were imputed by the participant’s individual mean.(39) To evaluate the impact of perceived structural barriers over and above attitudinal factors, the researchers computed hierarchical regression models for each source of help.”
Method
Why Missing Data Matter in the Longitudinal Study of Adolescent Development: Using the 4-H Study to Understand the Uses of Different Missing Data Methods
Jeličić, H., Phelps, E., & Lerner, R. M. (2010) · Journal of Youth and Adolescence · doi:10.1007/s10964-010-9542-5 · Cited by 55
“The literature also suggests that calculating a scale based on 60% of available items will yield more efficient and unbiased results than deletion of cases based on missing data on any scale item(s) if the multiple item scale has relatively high reliability (.50) and represents a unidimensional construct ( Roth et al.…”
Background
A Monte Carlo Study of Missing Item Methods
McDonald, R., Thurston, P. W., & Nelson, M. R. (2000) · Organizational Research Methods · doi:10.1177/109442810031003 · Cited by 54
“Some research findings showed that item mean substitution results in less dispersion around true correlations than listwise deletion (Chan & Dunn, 1972; Roth & Switzer, 1995).”
BackgroundMethodInfluential
Effects of three training types on vitality among older adults: A self-determination theory perspective
Solberg, P. A., Hopkins, W. G., Ommundsen, Y., & Halvari, H. (2012) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2012.01.006 · Cited by 59
Perceived Stress After Acute Myocardial Infarction: A Comparison Between Young and Middle-Aged Women Versus Men
Xu, X. J., Bao, H., Strait, K. M., Edmondson, D. E., Davidson, K. W., Beltrame, J. F., Bueno, H., Lin, H., Dreyer, R. P., Brush, J. E., et al. (2016) · Psychosomatic Medicine · doi:10.1097/psy.0000000000000429 · Cited by 58
Safe patient handling behaviors and lift use among hospital nurses: A cross-sectional study
Lee, S., & Lee, J. H. (2017) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2017.06.002 · Cited by 62
The Stigma of Mental Illness as a Barrier to Self Labeling as Having a Mental Illness
Stolzenburg, S., Freitag, S., Evans-Lacko, S. E., Muehlan, H., Schmidt, S., & Schomerus, G. (2017) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000000756 · Cited by 62
“We computed the total scores of the different measures, imputing missing values using individual mean participant response of that scale, if no more than 25% of values were missing (Downey and King, 1998; Roth et al., 1999).”
Method
The Role of Racial Identity and Implicit Racial Bias in Self-Reported Racial Discrimination: Implications for Depression Among African American Men
Chae, D., Powell, W., Nuru-Jeter, A. M., Smith-Bynum, M. A., Seaton, E. K., Forman, T. A., Turpin, R., & Sellers, R. (2017) · The Journal of Black Psychology · doi:10.1177/0095798417690055 · Cited by 51
“Using mean substitution for multi-item scales with at least 80% complete data has been shown to be an acceptable technique with little impact on statistical inferences (Roth, 1999).”
Method
Handling Missing Values in Information Systems Research: A Review of Methods and Assumptions
Peng, J., Hahn, J., & Huang, K.-W. (2022) · Information systems research · doi:10.1287/isre.2022.1104 · Cited by 17
Implicit Racial Bias as a Moderator of the Association Between Racial Discrimination and Hypertension: A Study of Midlife African American Men
Chae, D. H., Nuru-Jeter, A. M., & Adler, N. E. (2012) · Psychosomatic Medicine · doi:10.1097/psy.0b013e3182733665 · Cited by 49
“Mean substitution for multi-item scales with less than 20%missing data has little effect on the validity of statistical inferences (30).”
Background
Factors Associated with Mental Health Court Nonparticipation and Negative Termination
Dirks‐Linhorst, P. A., Kondrat, D. C., Linhorst, D. M., & Morani, N. M. (2011) · Justice Quarterly · doi:10.1080/07418825.2011.615756 · Cited by 49
Tough on Crime or Beating the System
Dirks‐Linhorst, P. A., & Kondrat, D. C. (2012) · Homicide Studies · doi:10.1177/1088767912438711 · Cited by 49
The Broader Autism Phenotype and Friendships in Non-clinical Dyads
Wainer, A., Block, N., Donnellan, M. B., & Ingersoll, B. R. (2013) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-013-1789-8 · Cited by 54
“Missing data was estimated by substituting the mean of all other items on the same scale for any missing item values (Roth et al. 1999).”
Method
Treatment of post‐traumatic stress disorder in children using cognitive behavioural writing therapy
der Oord, S. V., Lucassen, S., van Emmerik, A., & Emmelkamp, P. M. G. (2009) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.670 · Cited by 48
Mindfulness and self-compassion as tools in health behavior change: An evaluation of a workplace intervention pilot study
Horan, K. A., & Taylor, M. B. (2018) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2018.02.003 · Cited by 57
Is knowledge “half the battle”? The role of depression literacy in help-seeking among a non-clinical sample of adults with currently untreated mental health problems
Tomczyk, S., Muehlan, H., Freitag, S., Stolzenburg, S., Schomerus, G., & Schmidt, S. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.05.059 · Cited by 46
Performance implications of technological uncertainty, age, and size for small businesses
Drnevich, P. L., & West, J. (2021) · Journal of Small Business Management · doi:10.1080/00472778.2020.1867733 · Cited by 49
“For missing data (which ranged from approximately 2 percent to 5 percent depending upon the item), we used a means substitution technique for replacement (Roth, 1994; Roth et al., 1999).”
Method
Stepping into my shoes: generativity as a mediator of the relationship between business owners' age and family succession
Zacher, H., Schmitt, A., & Gielnik, M. M. (2011) · Ageing and Society · doi:10.1017/s0144686x11000547 · Cited by 46
The 14-item Paediatric Throat Disorders Outcome Test: a valid, sensitive, reliable, parent-reported outcome measure for paediatric throat disorders
Hopkins, C. L., Fairley, J. W., Yung, M., Hore, I., Balasubramaniam, S., & Haggard, M. P. (2010) · The Journal of Laryngology & Otology · doi:10.1017/s0022215109992386 · Cited by 43
Maternal Psychological Factors and Controlled Child Feeding Practices in Relation to Child Body Mass Index
Gemmill, A. W., Worotniuk, T., Holt, C. J., Skouteris, H., & Milgrom, J. (2013) · Childhood Obesity · doi:10.1089/chi.2012.0135 · Cited by 43
“Previous work has shown this approach to be robust.(51) Variables were assessed for normality, linearity, homoscedasticity, and univariate outliers.”
Implications of Discrimination Based on Sexuality, Gender, and Race/Ethnicity for Psychological Distress among Working-Class Sexual Minorities: The United for Health Study, 2003–2004
Chae, D. H., Krieger, N., Bennett, G. G., Lindsey, J. C., Stoddard, A. M., & Barbeau, E. M. (2010) · International Journal of Health Services · doi:10.2190/hs.40.4.b · Cited by 45
“” In cases where one of the six items was missing (fewer than 20% of the items), missing values were substituted with the within-participant mean for the remaining items (38).”
Method
Hot Deck Methods for Imputing Missing Data
Joenssen, D. W., & Bankhofer, U. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31537-4_6 · Cited by 41
The Performance of Multiple Imputation for Likert-type Items with Missing Data
Leite, W. L., & Beretvas, S. N. (2010) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1272686820 · Cited by 41
Emotional and behavioral functioning of typically-developing sisters of children with autism spectrum disorder: The roles of ASD severity, parental stress, and marital status
Tomeny, T. S., Baker, L. K., Barry, T. D., Eldred, S. W., & Rankin, J. A. (2016) · Research in Autism · doi:10.1016/j.rasd.2016.09.008 · Cited by 45
Self-labeling as having a mental or physical illness: the effects of stigma and implications for help-seeking
Horsfield, P., Stolzenburg, S., Hahm, S., Tomczyk, S., Muehlan, H., Schmidt, S., & Schomerus, G. (2019) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-019-01787-7 · Cited by 50
Perceived Stress, Severity of Asthma, and Quality of Life in Young Adults with Asthma
Kimura, T., Yokoyama, A., Kohno, N., Nakamura, H., & Eboshida, A. (2009) · Allergology International · doi:10.2332/allergolint.o-07-531 · Cited by 39

Authors: Roth, P. L., Switzer, F. S., & Switzer, D. M.
Affiliations: Clemson University (US)
ORCID: Roth, Philip L.
Switzer, Fred S.
Venue: Organizational Research Methods
Publisher: SAGE Publications
Published: 1999-07
Volume: 2
Issue: 3
Pages: 211-232
ISSN: 1094-4281 (ISSN-L)
1552-7425
DOI: 10.1177/109442819923001
OpenAlex: W2122169890
Cited by: 344
References: 39
Citations by year: 2026: 5 · 2025: 7 · 2024: 11 · 2023: 8 · 2022: 17 · 2021: 14 · 2020: 16 · 2019: 20 · 2018: 28 · 2017: 17 · 2016: 15 · 2015: 19 · 2014: 19 · 2013: 21 · 2012: 26
FWCI: 2.99
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Organizational Research Methods (publishedVersion)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Keywords: missing data, regression imputation, hot deck imputation, mean substitution, listwise deletion, imputation accuracy, attitudes, personality traits

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Missing Data in Multiple Item Scales: A Monte Carlo Analysis of Missing Data Techniques

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks