How do aggregated patient-reported outcome measures data stimulate health care improvement? A realist synthesis
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Objectives: Internationally, there has been considerable debate about the role of data in supporting quality improvement in health care. Our objective was to understand how, why and in what circumstances the feedback of aggregated patient-reported outcome measures data improved patient care. Methods: We conducted a realist synthesis. We identified three main programme theories underlying the use of patient-reported outcome measures as a quality improvement strategy and expressed them as nine ‘if then’ propositions. We identified international evidence to test these propositions through searches of electronic databases and citation tracking, and supplemented our synthesis with evidence from similar forms of performance data. We synthesized this evidence through comparing the mechanisms and impact of patient-reported outcome measures and other performance data on quality improvement in different contexts. Results: Three programme theories were identified: supporting patient choice, improving accountability and enabling providers to compare their performance with others. Relevant contextual factors were extent of public disclosure, use of financial incentives, perceived credibility of the data and the practicality of the results. Available evidence suggests that patients or their agents rarely use any published performance data when selecting a provider. The perceived motivation behind public reporting is an important determinant of how providers respond. When clinicians perceived that performance indicators were not credible but were incentivized to collect them, gaming or manipulation of data occurred. Outcome data do not provide information on the cause of poor care: providers needed to integrate and interpret patient-reported outcome measures and other outcome data in the context of other data. Lack of timeliness of performance data constrains their impact. Conclusions: Although there is only limited research evidence to support some widely held theories of how aggregated patient-reported outcome measures data stimulate quality improvement, several lessons emerge from interventions sharing the same programme theories to help guide the increasing use of these measures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Foucault, M. (1982) · Critical Inquiry · doi:10.1086/448181 · Cited by 8,496
Kluger, A. N., & DeNisi, A. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.254 · Cited by 5,882
Foucault, M. (1998) · Ethische und politische Freiheit · doi:10.1515/9783110815764.387 · Cited by 3,248
Pawson, R. (2006) · doi:10.4135/9781849209120 · Cited by 1,791
RAMESES publication standards: realist syntheses
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-21 · Cited by 1,374
Fung, C. H., Lim, Y.-W., Mattke, S., Damberg, C., & Shekelle, P. G. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-2-200801150-00006 · Cited by 980 · cited 2×
“Previous reviews of quantitative evidence suggest there is little support for the ‘selection’ pathway but there is evidence to support the ‘change’ pathway.(11,12) These reviews show that changes made in response to performance data vary in their impact on provider behaviour and health outcomes, while other work has n…”
KAPLAN, H. C., BRADY, P. W., DRITZ, M. C., HOOPER, D. K., LINAM, W. M., FROEHLE, C. M., & MARGOLIS, P. (2010) · Milbank Quarterly · doi:10.1111/j.1468-0009.2010.00611.x · Cited by 714 · cited 1×
“Although testing these theories led us to consider how some characteristics of organizations supported or limited the success of the implementation of quality improvement activities, we did not comprehensively review this literature.(52) Similarly, there are other explanations for why patients may not utilize quality…”
BackgroundConnections Between Quality Measurement and Improvement
Berwick, D. M., James, B., & Coye, M. J. (2003) · Medical Care · doi:10.1097/00005650-200301001-00004 · Cited by 575 · cited 1×
“It has been hypothesized that such data can improve the quality and outcomes of patient care either through a ‘selection’ pathway (whereby patients choose a high-quality hospital) or through a ‘change’ pathway (whereby providers initiate quality improvement).(8) Provider initiatives include those seeking to preserve t…”
BackgroundHospital Performance Reports: Impact On Quality, Market Share, And Reputation
Hibbard, J. H., Stockard, J., & Tusler, M. (2005) · Health Affairs · doi:10.1377/hlthaff.24.4.1150 · Cited by 413 · cited 2×
“There is evidence to indicate that public reporting places additional pressure on providers to respond, particularly poor performers (Proposition 2).(9,32) When data were fed back privately, with no public reporting or financial incentives, providers either ignored them or focused on altering their data collection pra…”
BackgroundCollecting data on patient experience is not enough: they must be used to improve care
Coulter, A., Locock, L., Ziebland, S., & Calabrese, J. (2014) · BMJ · doi:10.1136/bmj.g2225 · Cited by 409 · cited 1×
“We also acknowledge that patient experience data enable providers to understand what matters to patients more generally which may inform training, rather than, or in addition to, quality improvement activities.(54) Literature on the use of aggregated PROMs data is limited as the use of these measures for quality assur…”
BackgroundHibbard, J. H., & Peters, E. (2003) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.24.100901.141005 · Cited by 405 · cited 1×
“Similarly, there are other explanations for why patients may not utilize quality data that we did not explore.(53) We also acknowledge that patient experience data enable providers to understand what matters to patients more generally which may inform training, rather than, or in addition to, quality improvement activ…”
BackgroundDoes Publicizing Hospital Performance Stimulate Quality Improvement Efforts?
Hibbard, J. H., Stockard, J., & Tusler, M. (2003) · Health Affairs · doi:10.1377/hlthaff.22.2.84 · Cited by 402 · cited 3×
“There is evidence to indicate that public reporting places additional pressure on providers to respond, particularly poor performers (Proposition 2).(9,32) When data were fed back privately, with no public reporting or financial incentives, providers either ignored them or focused on altering their data collection pra…”
BackgroundRAMESES publication standards: realist syntheses
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013) · Journal of Advanced Nursing · doi:10.1111/jan.12095 · Cited by 385
How Organisations Measure Success: The Use of Performance Indicators in Government
Carter, N., Κlein, R., & Day, P. (1992) · Cited by 324
Digging for Nuggets: How ‘Bad’ Research Can Yield ‘Good’ Evidence
Pawson, R. (2006) · International Journal of Social Research Methodology · doi:10.1080/13645570600595314 · Cited by 317 · cited 1×
“Each fragment of evidence was appraised, as it was extracted, for its relevance to theory testing and the rigour with which it has been produced.(24) In many instances, only a subset of findings from each study that related specifically to the theory being tested was included in the synthesis.”
MethodAchieving And Sustaining Improved Quality: Lessons From New York State And Cardiac Surgery
Chassin, M. R. (2002) · Health Affairs · doi:10.1377/hlthaff.21.4.40 · Cited by 306 · cited 1×
“In general, providers did not see a change in their market share following the public reporting of performance data (Proposition 3).(11,27) Nonetheless, one US study suggested they were more likely to experience a change in market share if performance data provided ‘new information’ and revealed that previously ‘good’…”
BackgroundValderas, J. M., & Alonso, J. (2008) · Quality of Life Research · doi:10.1007/s11136-008-9396-4 · Cited by 298 · cited 2×
“More recently, the focus has shifted on the use of PROMs data in managing the performance of providers (organizations and individuals) and supporting patients’ choice of hospital.(2) The use of aggregated PROMs data as a tool for quality assurance and improvement is relatively recent, and there is inevitably only limi…”
BackgroundSelf-citationDoran, T., Kontopantelis, E., Valderas, J. M., Campbell, S., Roland, M., Salisbury, C., & Reeves, D. (2011) · BMJ · doi:10.1136/bmj.d3590 · Cited by 258
Self-citationVAN DER WEES, P. J., NIJHUIS‐VAN DER SANDEN, M. W. G., AYANIAN, J. Z., BLACK, N., WESTERT, G. P., & SCHNEIDER, E. C. (2014) · The Milbank Quarterly · doi:10.1111/1468-0009.12091 · Cited by 242
Self-citationDowrick, C., Leydon, G. M., McBride, A., Howe, A., Burgess, H., Clarke, P., Maisey, S., & Kendrick, T. (2009) · BMJ · doi:10.1136/bmj.b663 · Cited by 213 · cited 3×
“Under these conditions, clinicians managed the uncertainty of diagnosis by raising their thresholds for diagnosing depression in order to ensure they were not financially penalized.(40,41) While mandatory public reporting may help focus the attention of hospital leaders and clinicians on quality,(20,39) providers were…”
BackgroundMedical Autonomy and the UK State 1975 to 2025
Harrison, S., & Ahmad, W. I. U. (2000) · Sociology · doi:10.1017/s0038038500000092 · Cited by 197
Beyond metrics? Utilizing ‘soft intelligence’ for healthcare quality and safety
Martin, G. P., McKee, L., & Dixon‐Woods, M. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.07.027 · Cited by 172 · cited 2×
“Dissemination should involve top-down and bottom-up approaches so that clinical staff are aware of and know how to interpret performance data but also so that staff can bring their insights about the possible causes of problems to hospital leaders.(1)”
BackgroundDavies, E., Shaller, D., Edgman‐Levitan, S., Safran, D. G., Oftedahl, G., Sakowski, J., & Cleary, P. D. (2008) · Health Expectations · doi:10.1111/j.1369-7625.2007.00483.x · Cited by 164 · cited 5×
“Seventh, even when providers were given support to carry out additional investigations, implementing changes remained a challenge.(44) This suggests that bringing about significant, sustainable quality improvement requires a system-wide approach to change.”
BackgroundInfluentialChoice of hospital: Which type of quality matters?
Gutacker, N., Siciliani, L., Moscelli, G., & Gravelle, H. (2016) · Journal of Health Economics · doi:10.1016/j.jhealeco.2016.08.001 · Cited by 164 · cited 1×
“One UK study indicated that demand for hospitals was more responsive to changes in PROMs scores than to readmission or mortality rates, but the overall change in market share was very small.(29) Patients were more likely to choose a non-local hospital if they had a bad previous experience with their local hospital.”
BackgroundStart spreading the news: A structural estimate of the effects of New York hospital report cards
Dranove, D. S., & Sfekas, A. (2008) · Journal of Health Economics · doi:10.1016/j.jhealeco.2008.03.001 · Cited by 161 · cited 1×
“if performance data damaged their reputation (Proposition 4).(28) One UK study indicated that demand for hospitals was more responsive to changes in PROMs scores than to readmission or mortality rates, but the overall change in market share was very small.”
BackgroundEvidence and Policy and Naming and Shaming
Pawson, R. (2002) · Policy Studies · doi:10.1080/0144287022000045993 · Cited by 135 · cited 1×
“However, we can draw on evidence about the use of other forms of performance data which shares similar ideas and assumptions about how they are intended to work.(7) Learning from these programmes across different countries can inform the implementation of PROMs feedback.”
MethodHospital Survey Shows Improvements In Patient Experience
Elliott, M. N., Lehrman, W. G., Goldstein, E. H., Giordano, L. A., Beckett, M. K., Cohea, C. W., & Cleary, P. D. (2010) · Health Affairs · doi:10.1377/hlthaff.2009.0876 · Cited by 121 · cited 2×
“Where changes were made, these tended to focus on the organizational or functional aspects of care (sometimes referred to as ‘easy stuff’), rather than the more challenging relational aspects such as the behaviour and communication skills of clinicians.(42,43) Interventions were more likely to succeed if they did not…”
BackgroundGreenhalgh, J., Pawson, R., Wright, J. M., Black, N. A., Valderas, J. M., Meads, D. M., Gibbons, E. J., Wood, L., Wood, C., Mills, C., et al. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-005601 · Cited by 119
Self-citationExworthy, M., Wilkinson, E. K., McColl, A., Moore, M., Roderick, P., Smith, H., & Gabbay, J. (2003) · Social Science & Medicine · doi:10.1016/s0277-9536(02)00151-x · Cited by 90
The perceived impact of public reporting hospital performance data: interviews with hospital staff
Hafner, J. M., Williams, S., Koss, R. G., Tschurtz, B. A., Schmaltz, S. P., & Loeb, J. M. (2011) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzr056 · Cited by 90
Mitchell, C., Dwyer, R., Hagan, T., & Mathers, N. (2011) · British Journal of General Practice · doi:10.3399/bjgp11x572472 · Cited by 79 · cited 3×
“Under these conditions, clinicians managed the uncertainty of diagnosis by raising their thresholds for diagnosing depression in order to ensure they were not financially penalized.(40,41) While mandatory public reporting may help focus the attention of hospital leaders and clinicians on quality,(20,39) providers were…”
BackgroundBoiko, O. V., Campbell, J. L., Elmore, N., Davey, A. F., Roland, M. O., & Burt, J. (2014) · Health Expectations · doi:10.1111/hex.12298 · Cited by 66 · cited 1×
“Where changes were made, these tended to focus on the organizational or functional aspects of care (sometimes referred to as ‘easy stuff’), rather than the more challenging relational aspects such as the behaviour and communication skills of clinicians.(42,43) Interventions were more likely to succeed if they did not…”
BackgroundUsing Patient-reported Outcome Measures to Improve Health Care: Time for a New Approach
Browne, J. P., Cano, S., & Smith, S. C. (2017) · Medical Care · doi:10.1097/mlr.0000000000000792 · Cited by 57 · cited 1×
“The English National PROMs programme has recently been reviewed and some have called for a shift in purpose.(56) Our review synthesized evidence from interventions which share the same programme theories to understand how context shapes the mechanisms through which PROMs feedback may support quality improvement activi…”
BackgroundDo reviews of healthcare interventions teach us how to improve healthcare systems?
Pawson, R., Greenhalgh, J., Brennan, C. A., & Glidewell, L. (2014) · Social Science & Medicine · doi:10.1016/j.socscimed.2014.05.032 · Cited by 56 · cited 1×
“The investigation of key factors involved in this process has been explored in other reviews.(48) Eighth, PROMs data are only one of many pieces of performance data that providers have to respond to and when providers are not classed as an outlier, PROMs data run the risk of being ignored as providers tend to focus on…”
BackgroundSelf-citationMartin, G. P., Leslie, M., Minion, J. T., Willars, J., & Dixon‐Woods, M. (2013) · Social Science & Medicine · doi:10.1016/j.socscimed.2013.10.018 · Cited by 56 · cited 1×
“Using Harrison’s and Ahmad’s(49) work on scientific bureaucratic medicine and Foucault’s(50) ideas of disciplinary power and governmentality would have provided a useful alternative lens to test of our accountability theory and enabled us to explore how and when performance data function as curb to clinical autonomy(2…”
BackgroundBoyce, M., Browne, J. P., & Greenhalgh, J. (2014) · Implementation Science · doi:10.1186/1748-5908-9-84 · Cited by 54 · cited 2×
“There is some evidence that clinicians remain sceptical about the validity of using patients’ subjective assessments of outcome as an indicator of the quality of patient care.(21,33) Third, the perceived motivation behind public reporting is an important determinant of how providers respond.”
BackgroundSelf-citationDavies, H. T. O. (2001) · BMJ Quality & Safety · doi:10.1136/qhc.10.2.104 · Cited by 51
Smith, M. A., Wright, A., Queram, C., & Lamb, G. C. (2012) · Health Affairs · doi:10.1377/hlthaff.2011.0853 · Cited by 48 · cited 4×
“The evidence suggested that mechanisms such as clinicians’ intrinsic motivation to improve, aspirations to be as good as, or better than, their peers, or a desire to learn from the best practices of others were more likely to be triggered when the public reporting of data was clinically led (Proposition 5 and 8).(35,4…”
BackgroundInfluentialEmployers’ Efforts To Measure And Improve Hospital Quality: Determinants Of Success
Mehrotra, A., Bodenheimer, T. S., & Dudley, R. A. (2003) · Health Affairs · doi:10.1377/hlthaff.22.2.60 · Cited by 39 · cited 3×
“Providers held more favourable attitudes towards process of care and patient experience measures than outcome and cost data.(34) Although patient experience data identified potential problems, these were often already known to staff(42) and this did not always lead to improvements.”
BackgroundDissemination of performance information and continuous improvement: A narrative systematic review
Lemire, M., Demers‐Payette, O., & Jefferson-Falardeau, J. (2013) · Journal of Health Organization and Management · doi:10.1108/jhom-08-2011-0082 · Cited by 26 · cited 1×
“These reviews show that changes made in response to performance data vary in their impact on provider behaviour and health outcomes, while other work has noted the importance of contextual features in shaping the impact of such interventions.(13) However, there has been no systematic attempt to synthesize how context…”
BackgroundAlexander, J. A., Maeng, D. D., Casalino, L. P., & Rittenhouse, D. R. (2012) · Health Services Research · doi:10.1111/j.1475-6773.2012.01454.x · Cited by 17 · cited 1×
“There is some evidence to suggest that greater improvements in the quality of patient care occurred when providers are subjected to both financial incentives and public reporting than when they are subjected to either initiative alone (Proposition 7).(37) However, financial incentives only led to a short-term improvem…”
Background
Metadata
- Authors
- Greenhalgh, J., Dalkin, S., Gibbons, E., Wright, J., Valderas, J. M., Meads, D., & Black, N.
- Venue
- Journal of Health Services Research & Policy
- Publisher
- SAGE Publications
- Published
- 2017-12-20
- Volume
- 23
- Issue
- 1
- Pages
- 57-65
- ISSN
- 1355-8196 (ISSN-L)
- 1758-1060
- OpenAlex
- W2779490288
- Cited by
- 178
- References
- 41
- FWCI
- 40.10
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.