How valid and reliable are patient satisfaction data? An analysis of 195 studies
The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To assess the properties of validity and reliability of instruments used to assess satisfaction in a broad sample of health service user satisfaction studies, and to assess the level of awareness of these issues among study authors. DESIGN: Examination and analysis of 195 papers published in 1994 in 139 journals. The following databases were searched: British Nursing Index, CINAHL, EMBASE, MedLine, Popline, and PsycLIT. MAIN MEASURES: Number and types of strategies used for content, criterion, and construct validity, and for stability and internal consistency. Associations between validity/reliability and other study characteristics. RESULTS: Eighty-nine (46%) of the 195 studies reported some validity or reliability data; 76 reported some element of content validity; 14 reported criterion validity, with patient's intent to return the most commonly used criterion; four reported construct validity. Thirty-four studies reported internal consistency reliability, 31 of which used Cronbach's coefficient alpha; eight studies reported test-retest reliability. Only 11 studies (6% of the 181 quantitative studies) reported content validity and criterion or construct validity and reliability. 'New' instruments designed specifically for the reported study demonstrated significantly less evidence for reliability/validity than did 'old' instruments. CONCLUSION: With few exceptions, the study instruments in this sample demonstrated little evidence of reliability or validity. Moreover, study authors exhibited a poor understanding of the importance of these properties in the assessment of satisfaction. Researchers must be aware that this is poor research practice, and that lack of a reliable and valid assessment instrument casts doubt on the credibility of satisfaction findings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Assessment of client/patient satisfaction: development of a general scale
Larsen, D. L., Attkisson, C. C., Hargreaves, W. A., & Nguyen, T. D. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90094-6 · Cited by 2,887
The definition of quality and approaches to its assessment
Donabedian, A. (1980) · Health Administration Press eBooks · Cited by 2,605
A Brief Measure of Social Support: Practical and Theoretical Implications
Sarason, I. G., Sarason, B. R., Shearin, E. N., & Pierce, G. R. (1987) · Journal of Social and Personal Relationships · doi:10.1177/0265407587044007 · Cited by 1,977
The client satisfaction questionnaire
Attkisson, C. C., & Zwick, R. (1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90074-x · Cited by 1,743
Patient satisfaction: A review of issues and concepts
Sitzia, J., & Wood, N. (1997) · Social Science & Medicine · doi:10.1016/s0277-9536(97)00128-7 · Cited by 1,690
Self-citationNursing research : the application of qualitative approaches
Morse, J. M., & Field, P. (1996) · Medical Entomology and Zoology · Cited by 1,475
Patient satisfaction: A valid concept?
Williams, B. (1994) · Social Science & Medicine · doi:10.1016/0277-9536(94)90247-x · Cited by 1,145
Hall, J. A., & Dornan, M. C. (1990) · Social Science & Medicine · doi:10.1016/0277-9536(90)90205-7 · Cited by 945
Consumer satisfaction and perceived quality: Complementary or divergent constructs?
Gotlieb, J. B., Grewal, D., & Brown, S. W. (1994) · Journal of Applied Psychology · doi:10.1037//0021-9010.79.6.875 · Cited by 719
The psychosocial adjustment to illness scale (PAIS)
Derogatis, L. R. (1986) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(86)90069-3 · Cited by 621
Expectations as Determinants of Patient Satisfaction: Concepts, Theory and Evidence
THOMPSON, A. G. H., & SUNOL, R. (1995) · International Journal for Quality in Health Care · doi:10.1093/intqhc/7.2.127 · Cited by 572
Patients Evaluate Their Hospital Care: A National Survey
Cleary, P. D., Edgman-Levitan, S., Roberts, M., Moloney, T. W., McMullen, W., Walker, J. D., & Delbanco, T. L. (1991) · Health Affairs · doi:10.1377/hlthaff.10.4.254 · Cited by 554
The strengths and weaknesses of quantitative and qualitative research: what method for nursing?
Carr, L. T. (1994) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.1994.20040716.x · Cited by 376
Risser, N. L. (1975) · Nursing Research · doi:10.1097/00006199-197501000-00011 · Cited by 366
Nursing research: The application of qualitative approaches
Redfern, S. J. (1986) · Nurse Education Today · doi:10.1016/0260-6917(86)90011-0 · Cited by 364
Analysing qualitative interview data: addressing issues of validity andreliability
Appleton, J. V. (1995) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.1995.tb02653.x · Cited by 309
Analyzing Patient Satisfaction: A Multianalytic Approach
Abramowitz, S., Coté, A. A., & Berry, E. (1987) · QRB - Quality Review Bulletin · doi:10.1016/s0097-5990(16)30118-x · Cited by 284
Home-Based Versus Hospital-Based Care for People with Serious Mental Illness
Marks, I. M., Connolly, J., Muijen, M., Audini, B., McNamee, G., & Lawrence, R. E. (1994) · British Journal of Psychiatry · doi:10.1192/bjp.165.2.179 · Cited by 258
Key Determinants of Consumer Satisfaction With General Practice
WILLIAMS, S. J., & CALNAN, M. (1991) · Family Practice · doi:10.1093/fampra/8.3.237 · Cited by 234
Development of a patient satisfaction scale
La Monica, E. L., Oberst, M. T., Madea, A. R., & Wolf, R. M. (1986) · Research in Nursing & Health · doi:10.1002/nur.4770090108 · Cited by 218
Can Patients Evaluate the Quality of Hospital Care?
Rubin, H. R. (1990) · Medical Care Review · doi:10.1177/107755879004700302 · Cited by 205
Physician role performance and patient satisfaction
Larsen, D. E., & Rootman, I. (1976) · Social Science & Medicine (1967) · doi:10.1016/0037-7856(76)90136-0 · Cited by 201
Design and Analysis of Reliability Studies: The Statistical Evaluation of Measurement Errors.
Ackerman, T. A., & Dunn, G. (1991) · Journal of the American Statistical Association · doi:10.2307/2290615 · Cited by 182
A Patient Satisfaction Instrument: Precision by Replication
HINSHAW, A. S., & Atwood, J. R. (1982) · Nursing Research · doi:10.1097/00006199-198205000-00011 · Cited by 180
Litt, I. F., & Cuskey, W. R. (1984) · Journal of Adolescent Health Care · doi:10.1016/s0197-0070(84)80042-x · Cited by 161
A randomized trial of early hospital discharge and home follow-up of women having cesarean birth.
Brooten, D. A., Roncoli, M., Finkler, S. A., Arnold, L. D., Cohen, A., & Mennuti, M. T. (1994) · PubMed · Cited by 159
Patients' satisfaction with stroke services
Pound, P., Gompertz, P., & Ebrahim, S. B. J. (1994) · Clinical Rehabilitation · doi:10.1177/026921559400800102 · Cited by 150
Instruments for clinical nursing research
Frank‐Stromborg, M. (1988) · Medical Entomology and Zoology · Cited by 147
How was your hospital stay? Patients' reports about their care in Canadian hospitals.
Charles, C. A., Gauld, M., Chambers, L. W., O’Brien, B. J., Haynes, R. B., & Labelle, R. J. (1994) · PubMed · Cited by 124
Methodological Studies: Instrument Development
Mishel, M. H. (1998) · Sage eBooks · doi:10.4135/9781452204840.n10 · Cited by 109
Community Psychiatric Nurse Teams: Intensive Support Versus Generic Care
Muijen, M., Cooney, M., Strathdee, G., Bell, R., & Hudson, A. (1994) · British Journal of Psychiatry · doi:10.1192/bjp.165.2.211 · Cited by 106
A Controlled Trial of Outpatient Geriatric Evaluation and Management
Boult, C., Boult, L., Murphy, C., Ebbitt, B., Luptak, M., & Kane, R. L. (1994) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.1994.tb04965.x · Cited by 97
Further assessment of a method to estimate reliability and validity of qualitative research findings
Hinds, P. S., Scandrett-Hibdon, S. L., & McAulay, L. S. (1990) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.1990.tb01836.x · Cited by 77
Factors influencing inpatient satisfaction with care.
Er, D. (1983) · PubMed · Cited by 76
Research Methods for Nurses and the Caring Professions
Abbott, P., & Sapsford, R. J. (1992) · Medical Entomology and Zoology · Cited by 62
Gelkopf, M., Sigal, M., & Krämer, R. (1994) · The Journal of Social Psychology · doi:10.1080/00224545.1994.9711380 · Cited by 55
Compendium of Instruments Measuring Patient Satisfaction with Nursing Care
McDaniel, C., & Nash, J. G. (1990) · QRB - Quality Review Bulletin · doi:10.1016/s0097-5990(16)30361-x · Cited by 54
Research Methods for Nurses and the Caring Professions
Draper, P. (1993) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.1993.180508472.x · Cited by 46
Do Therapist Experience, Diagnosis and Functional Level Predict Outcome in Short Term Psychotherapy?
Propst, A., Paris, J., & Rosberger, Z. (1994) · The Canadian Journal of Psychiatry · doi:10.1177/070674379403900309 · Cited by 27
Relations between satisfaction with therapists and psychotherapy outcome
Conte, H. R., Buckley, P. J., Picard, S., & Karasu, T. B. (1994) · Journal of psychotherapy practice and research/The Journal of psychotherapy practice and research · Cited by 22
Knox, J. (1989) · Social Science & Medicine · doi:10.1016/0277-9536(89)90259-1 · Cited by 14
McGrory, A., & Assmann, S. (1994) · PubMed · Cited by 13
Cited by
Good practice in the conduct and reporting of survey research
KELLEY, K. (2003) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzg031 · Cited by 2,157
Crow, R., Gage, H., Hampson, S., Hart, J., Kimber, A., Storey, L., & Thomas, H. (2002) · Health Technology Assessment · doi:10.3310/hta6320 · Cited by 1,270
A critical review of patient satisfaction
Gill, L., & White, L. (2009) · Leadership in Health Services · doi:10.1108/17511870910927994 · Cited by 400
Predictors of patient satisfaction with hospital health care
Quintana, J. M., González, N., Bilbao, A., Aizpuru, F., Escobar, A., Esteban, C., San-Sebastián, J. A., de-la-Sierra, E., & Thompson, A. (2006) · BMC Health Services Research · doi:10.1186/1472-6963-6-102 · Cited by 385
Patient satisfaction revisited: A multilevel approach
Hekkert, K. D., Cihangir, S., Kleefstra, S. M., van den Berg, B., & Kool, R. B. (2009) · Social Science & Medicine · doi:10.1016/j.socscimed.2009.04.016 · Cited by 311
Metadata
- Authors
- Sitzia, J.
- Venue
- International Journal for Quality in Health Care
- Publisher
- Oxford University Press (OUP)
- Published
- 1999-08-01
- Volume
- 11
- Issue
- 4
- Pages
- 319-328
- ISSN
- 1353-4505 (ISSN-L)
- 1464-3677
- OpenAlex
- W2115375689
- Cited by
- 515
- References
- 44
- FWCI
- 17.07
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.