How Robust Are Hospital Ranks Based on Composite Performance Measures?
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Composite indices of healthcare performance are an aggregation of underlying individual performance measures. They are increasingly being used to rank healthcare organizations. Although composite indicators present the "big picture" in a way that is easy to interpret, misleading conclusions may be drawn if attention is not paid to key methodological issues in their construction. OBJECTIVES: We examine variability in performance measures in the context of the construction and use of composite measures. We illustrate how variability in the underlying data and the resulting composite may undermine the robustness of performance measures in health care. We also illustrate how variation in the methodological rules applied to aggregate the individual indicators can have an important impact on composite scores. METHODS: We use data for 117 English acute hospitals to illustrate the generic methodological issues. The variance in performance measures is partitioned into "controllable" and "uncontrollable" elements. We create a composite index from the underlying performance indicators and use Monte Carlo simulations to examine the robustness of the composite. RESULTS: Random variation beyond the control of organizations gives rise to considerable uncertainty in hospital scores. Composites are also sensitive to changes made to the weighting system and to the aggregation rules. Some hospitals can jump almost half of the league table as a result of subtle changes. CONCLUSIONS: Great care is warranted in interpreting the results of composite performance measures. Suggestions for their future development are made.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Suggestion for Using Powerful and Informative Tests of Normality
D'agostino, R. B., Belanger, A., & D'agostino, R. B. (1990) · The American Statistician · doi:10.2307/2684359 · Cited by 1,809
Composite Indicators of Country Performance: A Critical Assessment
(2003) · OECD Science, Technology and Industry Working Papers · doi:10.1787/405566708255 · Cited by 523
Cited by
Bradley, E. H., Herrin, J., Elbel, B., McNamara, R. L., Magid, D. J., Nallamothu, B. K., Wang, Y., Normand, S.-L. T., Spertus, J. A., & Krumholz, H. M. (2006) · JAMA · doi:10.1001/jama.296.1.72 · Cited by 375
Separate And Unequal: Racial Segregation And Disparities In Quality Across U.S. Nursing Homes
Smith, D. B., Feng, Z., Fennell, M. L., Zinn, J. S., & Mor, V. (2007) · Health Affairs · doi:10.1377/hlthaff.26.5.1448 · Cited by 258
Review and new evidence on composite innovation indicators for evaluating national performance
Grupp, H., & Schubert, T. (2010) · Research Policy · doi:10.1016/j.respol.2009.10.002 · Cited by 256
Brown, C., Hofer, T., Johal, A., Thomson, R., Nicholl, J., Franklin, B. D., & Lilford, R. J. (2008) · Quality and Safety in Health Care · doi:10.1136/qshc.2007.023655 · Cited by 100
Data Quality in Clinical Research
Nahm, M. (2012) · Health Informatics · doi:10.1007/978-1-84882-448-5_10 · Cited by 32
Metadata
- Authors
- Jacobs, R., Goddard, M., & Smith, P. C.
- Venue
- Medical Care
- Published
- 2005-12
- Volume
- 43
- Issue
- 12
- Pages
- 1177-1184
- ISSN
- 0025-7079 (ISSN-L)
- OpenAlex
- W1982670407
- Cited by
- 107
- References
- 2
- FWCI
- 10.05
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How Robust Are Hospital Ranks Based on Composite Performance Measures? https://app.cochat.ai/papers/10.1097/01.mlr.0000185692.72905.4a
- The paper
- Jacobs, R., Goddard, M., & Smith, P. C. (2005). How Robust Are Hospital Ranks Based on Composite Performance Measures? Medical Care. https://doi.org/10.1097/01.mlr.0000185692.72905.4a
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.