Predictive Validity of a Quality Measure for Intensive Substance use Disorder Treatment
Abstract
Background: Measures of substance use disorder (SUD) treatment quality are essential tools for performance improvement. The Veterans Health Administration (VHA) developed a measure of access to and engagement in intensive outpatient programs (IOPs) for SUD. However, predictive validity, or associations between this measure and treatment outcomes, has not been examined. Methods: Data on veterans with SUD came from 3 samples: the Outcomes Monitoring Project ( N = 5436), a national evaluation of VHA mental health services ( N = 339,887), and patients receiving detoxification services ( N = 23,572). Propensity score–weighted mixed-effects regressions modeled associations between receiving at least 1 week of IOP treatment and patient outcomes, controlling for facility-level performance and a random effect for facility. Results: Propensity score weighting reduced or eliminated observable baseline differences between patient groups. Patients who accessed IOPs versus those who did not reported significantly reduced alcohol- and drug-related symptom severity, with significantly fewer past-month days drinking alcohol ( b = 1.83, P < .001) and fewer past-month days intoxicated ( b = 1.55, P < .001). Patients who received IOP after detoxification services had higher 6-month utilization of SUD outpatient visits ( b = 2.09, P < .001), more subsequent detoxification episodes ( b = 0.25, P < .001), and lower odds of 2-year mortality (odds ratio [OR] = 0.68, 95% confidence interval [CI]: 0.61–0.75; P < .001). Conclusions: Receiving at least 1 week of SUD treatment in an IOP was associated with higher follow-up utilization, improved health outcomes, and reduced mortality. These associations lend support to the predictive validity of VHA's IOP quality measure. Future research should focus on measure feasibility and validity outside of VHA, and whether predictive validity is maintained once this quality measure is tied to performance incentives.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some recently reported measures for improved processes of care, such as appropriate pharmacotherapy, continuity of care, and psychotherapy use, are associated with reduced mortality and reduced symptom severity.
“Some notable exceptions that have been reported recently show that measures for improved processes of care (e.g., appropriate pharmacotherapy, continuity of care, and psychotherapy use) are associated with reduced mortality [28][29][30][31] and reduced symptom severity 32 .”
From Measuring and improving the quality of mental health care: a global perspective (2018) · cites this paper as [32]
Cites
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638 · see citing works
Analysis of Incomplete Multivariate Data
Schafer, J. L. (1997) · doi:10.1201/9781439821862 · Cited by 6,012 · see citing works
Donabedian, A. (1988) · JAMA · doi:10.1001/jama.260.12.1743 · Cited by 5,895 · see citing works
Analysis of Incomplete Multivariate Data
Booth, D. E., & Schafer, J. L. (2000) · Technometrics · doi:10.2307/1271466 · Cited by 5,627 · see citing works
The National Committee for Quality Assurance
Iglehart, J. K. (1996) · New England Journal of Medicine · doi:10.1056/nejm199609263351322 · Cited by 1,826 · see citing works
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.403 · Cited by 1,317 · see citing works
Hirschfeld, R. M. A. (1997) · JAMA · doi:10.1001/jama.277.4.333 · Cited by 896 · see citing works
The De Facto US Mental Health Services System
Regier, D. A. (1978) · Archives of General Psychiatry · doi:10.1001/archpsyc.1978.01770300027002 · Cited by 879 · see citing works
Ethnic Disparities in Unmet Need for Alcoholism, Drug Abuse, and Mental Health Care
Wells, K., Klap, R., Koike, A., & Sherbourne, C. (2001) · American Journal of Psychiatry · doi:10.1176/appi.ajp.158.12.2027 · Cited by 773 · see citing works
Relationship Between Medicare’s Hospital Compare Performance Measures and Mortality Rates
Werner, R. M., & Bradlow, E. T. (2006) · JAMA · doi:10.1001/jama.296.22.2694 · Cited by 495 · see citing works
National Survey of Substance Abuse Treatment Services (N-SSATS), 2000
United States Department Of Health And Human Services. Substance Abuse And Mental Health Services Administration. Office Of Applied Studies (2002) · ICPSR Data Holdings · doi:10.3886/icpsr03436.v4 · Cited by 211 · see citing works
National Survey of Substance Abuse Treatment Services (N-SSATS), 2000
United States Department Of Health And Human Services. Substance Abuse And Mental Health Services Administration. Office Of Applied Studies (2002) · ICPSR Data Holdings · doi:10.3886/icpsr03436 · Cited by 211 · see citing works
Pitfalls of Converting Practice Guidelines Into Quality Measures
Walter, L. C., Davidowitz, N. P., Heineken, P. A., & Covinsky, K. E. (2004) · JAMA · doi:10.1001/jama.291.20.2466 · Cited by 197 · see citing works
Consistency of self‐administered and interview‐based Addiction Severity Index composite scores
Rosen, C. S., Henson, B. R., Finney, J. W., & Moos, R. H. (2000) · Addiction · doi:10.1046/j.1360-0443.2000.95341912.x · Cited by 182 · see citing works
Self-citationWachter, R. M., Flanders, S. A., Fee, C. R., & Pronovost, P. J. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-149-1-200807010-00007 · Cited by 180 · see citing works
Patient Satisfaction and Administrative Measures as Indicators of the Quality of Mental Health Care
Druss, B. G., Rosenheck, R. A., & Stolar, M. J. (1999) · Psychiatric Services · doi:10.1176/ps.50.8.1053 · Cited by 155 · see citing works
Adapting Washington Circle performance measures for public sector substance abuse treatment systems
Garnick, D. W., Lee, M. T., Horgan, C. M., & Acevedo, A. (2008) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2008.06.008 · Cited by 139 · see citing works
Establishing the feasibility of performance measures for alcohol and other drugs
Garnick, D. W., Lee, M. T., Chalk, M., Gastfriend, D. R., Horgan, C. M., McCorry, F., McLellan, A. T., & Merrick, E. L. (2002) · Journal of Substance Abuse Treatment · doi:10.1016/s0740-5472(02)00303-3 · Cited by 128 · see citing works
Substance Abuse Intensive Outpatient Programs: Assessing the Evidence
McCarty, D. W., Braude, L., Lyman, D. R., Dougherty, R. H., Daniels, A. S., Ghose, S. S., & Delphin-Rittmon, M. E. (2014) · Psychiatric Services · doi:10.1176/appi.ps.201300249 · Cited by 94 · see citing works
Does Meeting the HEDIS Substance Abuse Treatment Engagement Criterion Predict Patient Outcomes?
Harris, A. H. S., Humphreys, K. N., Bowe, T. R., Tiêt, Q. Q., & Finney, J. W. (2008) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-008-9142-2 · Cited by 77 · see citing works
Self-citationHarris, A. H. S., Humphreys, K. N., Bowe, T. R., Kivlahan, D. R., & Finney, J. W. (2008) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2008.05.011 · Cited by 70 · see citing works
Self-citationA practical system for monitoring the outcomes of substance use disorder patients
Tiet, Q. Q., Byrnes, H. F., Barnett, P., & Finney, J. W. (2006) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.03.002 · Cited by 60 · see citing works
Self-citationEngagement in Outpatient Substance Abuse Treatment and Employment Outcomes
Dunigan, R., Acevedo, A., Campbell, K. M., Garnick, D. W., Horgan, C. M., Huber, A., Lee, M. T., Panas, L., & Ritter, G. A. (2013) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-013-9334-2 · Cited by 54 · see citing works
Performance Measures and Clinical Outcomes
Horn, S. D. (2006) · JAMA · doi:10.1001/jama.296.22.2731 · Cited by 49 · see citing works
A Comparison of Regression Approaches for Analyzing Clustered Data
Desai, M., & Begg, M. D. (2008) · American Journal of Public Health · doi:10.2105/ajph.2006.108233 · Cited by 48 · see citing works
Garnick, D. W., Lee, M. T., O’Brien, P. L., Panas, L., Ritter, G. A., Acevedo, A., Garner, B. R., Funk, R. R., & Godley, M. D. (2012) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2012.01.011 · Cited by 48 · see citing works
Managed care and outpatient substance abuse treatment intensity
Lemak, C. H., & Alexander, J. A. (2001) · The Journal of Behavioral Health Services & Research · doi:10.1007/bf02287231 · Cited by 46 · see citing works
Veterans Health Administration Mental Health Program Evaluation: Capstone Report
Watkins, K. E., RAND, Pincus, H. A., Smith, B., Paddock, S. M., Mannle, T. E., Woodroffe, A., Solomon, J., Sorbero, M. E. S., Farmer, C. M., et al. (2011) · PsycEXTRA Dataset · doi:10.1037/e525432012-001 · Cited by 45 · see citing works
Harris, A. H. S., Humphreys, K. N., & Finney, J. W. (2007) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2006.12.015 · Cited by 43 · see citing works
Self-citationVHA Mental Health Information System
Trafton, J. A., Greenberg, G. A., Harris, A. H. S., Tavakoli, S., Kearney, L. K., McCarthy, J., Blow, F., Hoff, R. A., & Schohn, M. (2013) · Medical Care · doi:10.1097/mlr.0b013e31827da836 · Cited by 39 · see citing works
Self-citationHarris, A. H. S., Gupta, S., Bowe, T. R., Ellerbe, L. S., Phelps, T. E., Rubinsky, A. D., Finney, J. W., Asch, S. M., Humphreys, K. N., & Trafton, J. A. (2015) · Addiction Science & Clinical Practice · doi:10.1186/s13722-015-0042-5 · Cited by 36 · see citing works
Self-citationGreenberg, G. A., Rosenheck, R., & Seibyl, C. (2002) · Medical Care · doi:10.1097/00005650-200203000-00008 · Cited by 32 · see citing works
What Makes a Good Quality Measure?
McGlynn, E. A., & Adams, J. (2014) · JAMA · doi:10.1001/jama.2014.12819 · Cited by 32 · see citing works
Harris, A. H. S., Bowe, T. R., Finney, J. W., & Humphreys, K. N. (2009) · Population Health Management · doi:10.1089/pop.2008.0028 · Cited by 29 · see citing works
Self-citationBrucker, D. L., & Stewart, M. T. (2011) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-010-9228-5 · Cited by 29 · see citing works
Greenberg, G. A., Rosenheck, R., & Fontana, A. (2003) · Journal of Behavioral Health Services & Research · doi:10.1007/bf02289808 · Cited by 24 · see citing works
Unintended consequences of a quality measure for acute bronchitis.
Roth, S., Gonzales, R., Anderer, T., Bloom, F. J., Graf, T. R., Stahl, M. S., Maselli, J. H., & Metlay, J. P. (2012) · PubMed · Cited by 24 · see citing works
Improving Mental Health Care: A Guide to Measurement-Based Quality Improvement
Sederer, L. I. (2006) · Psychiatric Services · doi:10.1176/appi.ps.57.10.1531 · Cited by 24 · see citing works
Selective Contracting in Managed Care: The Case of Substance Abuse Treatment
Lemak, C. H., Alexander, J. A., & D’Aunno, T. A. (2001) · Medical Care Research and Review · doi:10.1177/107755870105800405 · Cited by 19 · see citing works
Public Managed Care and Service Access in Outpatient Substance Abuse Treatment Units
Chuang, E., Wells, R. S., & Alexander, J. A. (2010) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-010-9230-y · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Measuring and improving the quality of mental health care: a global perspective
Kilbourne, A. M., Beck, K., Spaeth‐Rublee, B., Ramanuj, P., O'Brien, R. W., Tomoyasu, N., & Pincus, H. A. (2018) · World Psychiatry · doi:10.1002/wps.20482 · Cited by 457 · see citing works
Harris, M., Bharat, C., Glantz, M. D., Sampson, N. A., Al-Hamzawi, A. A., Alonso, J., Bruffaerts, R., Caldas‐de‐Almeida, J. M., Cía, A. H., de Girolamo, G., et al. (2019) · Addiction · doi:10.1111/add.14599 · Cited by 55 · see citing works
At the intersection of sleep deficiency and opioid use: mechanisms and therapeutic opportunities
Greenwald, M. K., Moses, T. E. H., & Roehrs, T. (2021) · Translational research · doi:10.1016/j.trsl.2021.03.006 · Cited by 29 · see citing works
Paddock, S. M., Hepner, K. A., Hudson, T. J., Ounpraseuth, S., Schrader, A. M., Sullivan, G., & Watkins, K. E. (2017) · Journal of Studies on Alcohol and Drugs · doi:10.15288/jsad.2017.78.588 · Cited by 25 · see citing works
Johnson, K., Hills, H. A., Ma, J., Brown, C. H., & McGovern, M. P. (2020) · The American Journal of Drug and Alcohol Abuse · doi:10.1080/00952990.2020.1824236 · Cited by 23 · see citing works
Follow-up after ED visits for opioid use disorder: Do they reduce future overdoses?
Cunningham, P. J., Barnes, A. J., Mohamoud, S., Allen, L., Talbert, J., Jarlenski, M., Kim, J. Y., Gordon, A. J., Tang, L., Chang, C.-C. H., et al. (2022) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2022.108807 · Cited by 14 · see citing works
Szymanski, B. R., Hein, T. C., Schoenbaum, M., McCarthy, J. F., & Katz, I. R. (2021) · Psychiatric Services · doi:10.1176/appi.ps.202000282 · Cited by 9 · see citing works
Lothes, J. E., II, & Hall, M. (2024) · Counseling Outcome Research and Evaluation · doi:10.1080/21501378.2024.2437990 · Cited by 3 · see citing works
MacKinnon, L., Choi, J., Kennedy, M. C., Brar, R., Milloy, M.-J., Hayashi, K., & Socías, M. E. (2022) · Journal of Addiction Medicine · doi:10.1097/adm.0000000000000984 · Cited by 2 · see citing works
Riblet, N. B. V., Gottlieb, D. J., Shiner, B. R., Zubkoff, L., Rice, K., & Rusch, B. (2024) · Journal of Substance Use and Addiction Treatment · doi:10.1016/j.josat.2024.209514 · Cited by 2 · see citing works
Payer-Level Care Coordination and Re-admission to Acute Mental Health Care for Uninsured Individuals
Proctor, S. L., Gursky-Landa, B., Kannarkat, J. T., Guimaraes, J., & Newcomer, J. W. (2022) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-022-09789-1
Brown, J. A., Akgul, T., Lewis, T. F., & Senn, P. (2026) · Journal of Addictions & Offender Counseling · doi:10.1002/jaoc.70020
- Authors
- Schmidt, E. M., Gupta, S., Bowe, T., Ellerbe, L. S., Phelps, T. E., Finney, J. W., Asch, S. M., Humphreys, K., Trafton, J., Vanneman, M., & Harris, A. H. S.
- Affiliations
- VA Palo Alto Health Care System (US)
- Corresponding author
- Harris, Alex H. S.
- ORCID
- Venue
- Substance Abuse
- Publisher
- SAGE Publications
- Published
- 2017-07
- Volume
- 38
- Issue
- 3
- Pages
- 317-323
- ISSN
- 0889-7077 (ISSN-L)
- 1547-0164
- OpenAlex
- W2499123317
- Cited by
- 12
- References
- 41
- Citations by year
- 2026: 1 · 2024: 2 · 2022: 3 · 2021: 2 · 2020: 1 · 2019: 1 · 2018: 1 · 2017: 1
- FWCI
- 0.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Substance Abuse (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. Department of Veterans Affairs (IK6 HX002840, I01 HX000517)
- Topics
- Health Sciences › Medicine › Epidemiology › Substance Abuse Treatment and Outcomes
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Opioid Use Disorder Treatment
- Health Sciences › Health Professions › Health Information Management › Healthcare Quality and Management
- Keywords
- intensive outpatient treatment, addiction treatment, predictive validity, quality metrics, veterans, alcohol consumption, drug use, detoxification, treatment utilization, mortality, performance improvement, Veterans Health Administration
- MeSH terms
- Ambulatory Care, Female, Humans, Male, Mental Health Services, Middle Aged, Patient Acceptance of Health Care, Quality Assurance, Health Care, United States, United States Department of Veterans Affairs, Veterans, Reproducibility of Results, Treatment Outcome, Substance-Related Disorders
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
