CoChat

Machine learning methods for developing precision treatment rules with observational data

Open access
Kessler, Ronald C., Bossarte, Robert M., Luedtke, Alex, Zaslavsky, Alan M., Zubizarreta, Jose R. · 2019 · Behaviour Research and Therapy
Cited by 81

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.

Abstract

Clinical trials have identified a variety of predictor variables for use in precision treatment protocols, ranging from clinical biomarkers and symptom profiles to self-report measures of various sorts. Although such variables are informative collectively, none has proven sufficiently powerful to guide optimal treatment selection individually. This has prompted growing interest in the development of composite precision treatment rules (PTRs) that are constructed by combining information across a range of predictors. But this work has been hampered by the generally small samples in randomized clinical trials and the use of suboptimal analysis methods to analyze the resulting data. In this paper, we propose to address the sample size problem by: working with large observational electronic medical record databases rather than controlled clinical trials to develop preliminary PTRs; validating these preliminary PTRs in subsequent pragmatic trials; and using ensemble machine learning methods rather than individual algorithms to carry out statistical analyses to develop the PTRs. The major challenges in this proposed approach are that treatment are not randomly assigned in observational databases and that these databases often lack measures of key prescriptive predictors and mental disorder treatment outcomes. We proposed a tiered case-cohort design approach that uses innovative methods for measuring and balancing baseline covariates and estimating PTRs to address these challenges.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.brat.2019.103412

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 38 references

Research works printed but not linked: 84

  1. Abadie, 2018, Endogenous stratification in randomized experiments, The Review of Economics and Statistics, C, 567, 10.1162/rest_a_00732 · DOI
  2. Baiocchi, 2014, Instrumental variable methods for causal inference, Statistics in Medicine, 33, 2297, 10.1002/sim.6128 · DOI · Paper page
  3. Barbui, 2009, Selective serotonin reuptake inhibitors and risk of suicide: A systematic review of observational studies, Canadian Medical Associaiton Journal, 180, 291, 10.1503/cmaj.081514 · DOI · Paper page
  4. Berger, 2017, Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE special task force on real-world evidence in health care decision making, Pharmacoepidemiology and Drug Safety, 26, 1033, 10.1002/pds.4297 · DOI · Paper page
  5. Bernert, 2014, A review of multidisciplinary clinical practice guidelines in suicide prevention: Toward an emerging standard in suicide risk assessment and management, training and practice, Academic Psychiatry, 38, 585, 10.1007/s40596-014-0180-1 · DOI · Paper page
  6. Bousman, 2019, Pharmacogenetic tests and depressive symptom remission: A meta-analysis of randomized controlled trials, Pharmacogenomics, 20, 37, 10.2217/pgs-2018-0142 · DOI · Paper page
  7. Bryan, 2018, Differential effects of brief CBT versus treatment as usual on posttreatment suicide attempts among groups of suicidal patients, Psychiatric Services, 69, 703, 10.1176/appi.ps.201700452 · DOI
  8. Bzdok, 2018, Machine learning for precision psychiatry: Opportunities and challenges. Biological Psychiatry, Cognitive Neuroscience and Neuroimaging, 3, 223
  9. Cattaneo, 2015, Randomization inference in the regression discontinuity design: An application to party advantages in the US Senate, Journal of Causal Inference, 3, 1, 10.1515/jci-2013-0010 · DOI · Paper page
  10. Chan, 2016, Globally efficient non-parametric inference of average treatment effects by empirical balancing calibration weighting, Journal of the Royal Statistical Society: Series B, 78, 673, 10.1111/rssb.12129 · DOI · Paper page
  11. Chesin, 2018, Perceptions of adjunctive mindfulness-based cognitive therapy to prevent suicidal behavior among high suicide-risk outpatient participants, Crisis, 39, 451, 10.1027/0227-5910/a000519 · DOI · Paper page
  12. Cooper, 2013, Are hospital services for self-harm getting better? An observational study examining management, service provision and temporal trends in england, BMJ Open, 3, 10.1136/bmjopen-2013-003444 · DOI · Paper page
  13. Diamond, 2013, Genetic matching for estimating causal effects: A general multivariate matching method for achieving balance in observational studies, The Review of Economics and Statistics, 95, 932, 10.1162/REST_a_00318 · DOI · Paper page
  14. Durand, 2013, Does raising type 1 error rate improve power to detect interactions in linear regression models? A simulation study, PLoS One, 8, 10.1371/journal.pone.0071079 · DOI · Paper page
  15. Fabbri, 2016, Progress and prospects in pharmacogenetics of antidepressant drugs, Expert Opinion on Drug Metabolism and Toxicology, 12, 1157, 10.1080/17425255.2016.1202237 · DOI · Paper page
  16. Forkmann, 2016, The effects of mindfulness-based cognitive therapy and cognitive behavioral analysis system of psychotherapy added to treatment as usual on suicidal ideation in chronic depression: Results of a randomized-clinical trial, Journal of Affective Disorders, 200, 51, 10.1016/j.jad.2016.01.047 · DOI
  17. Greist, 2014, Predictive value of baseline electronic Columbia-Suicide Severity Rating Scale (eC-SSRS) assessments for identifying risk of prospective reports of suicidal behavior during research participation, Innovations in Clinical Neuroscience, 11, 23
  18. Gruber, 2015, Ensemble learning of inverse probability weights for marginal structural modeling in large observational datasets, Statistics in Medicine, 34, 106, 10.1002/sim.6322 · DOI · Paper page
  19. Hahn, 2001, Identification and estimation of treatment effects with a regression-discontinuity design, Econometrica, 69, 201, 10.1111/1468-0262.00183 · DOI · Paper page
  20. Hansen, 2004, Full matching in an observational study of coaching for the SAT, Journal of the American Statistical Association, 99, 609, 10.1198/016214504000000647 · DOI · Paper page
  21. Hjorthoj, 2014, Risk of suicide according to level of psychiatric treatment: A nationwide nested case-control study, Social Psychiatry and Psychiatric Epidemiology, 49, 1357, 10.1007/s00127-014-0860-x · DOI
  22. Imai, 2014, Covariate balancing propensity score, Journal of the Royal Statistical Society: Series B, 76, 243, 10.1111/rssb.12027 · DOI · Paper page
  23. Jobes, 2017, A randomized controlled trial of the collaborative assessment and management of suicidality versus enhanced care as usual with suicidal soldiers, Psychiatry, 80, 339, 10.1080/00332747.2017.1354607 · DOI · Paper page
  24. Kang, 2007, Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data, Statistical Science, 22, 523
  25. Keele, 2015, Enhancing a geographic regression discontinuity design through matching to estimate the effect of ballot initiatives on voter turnout, Journal of the Royal Statistical Society: Series A, 178, 223, 10.1111/rssa.12056 · DOI
  26. Kessler, 2017, Using patient self-reports to study heterogeneity of treatment effects in major depressive disorder, Epidemiology and Psychiatric Sciences, 26, 22, 10.1017/S2045796016000020 · DOI · Paper page
  27. Kinard, 1980, Changes in life style following mental hospitalization, The Journal of Nervous and Mental Disease, 168, 666, 10.1097/00005053-198011000-00005 · doi:10.1097/00005053-198011000-00005
  28. Kovalchik, 2013, Assessing heterogeneity of treatment effect in a clinical trial with the proportional interactions model, Statistics in Medicine, 32, 4906, 10.1002/sim.5881 · DOI
  29. van der Laan, 2015, Targeted learning of the mean outcome under an optimal dynamic treatment rule, Journal of Causal Inference, 3, 61, 10.1515/jci-2013-0022 · DOI · Paper page
  30. van der Laan, 2007, Super learner, Statistical Applications in Genetics and Molecular Biology, 6
  31. Large, 2018, Psychiatric hospitalisation and the risk of suicide, The British Journal of Psychiatry: Journal of Mental Science, 212, 269, 10.1192/bjp.2018.22 · DOI · Paper page
  32. LeDell, 2016, AUC-maximizing ensembles through metalearning, International Journal of Biostatistics, 12, 203, 10.1515/ijb-2015-0035 · DOI
  33. Linehan, 2015, Dialectical behavior therapy for high suicide risk in individuals with borderline personality disorder: A randomized clinical trial and component analysis, JAMA Psychiatry, 72, 475, 10.1001/jamapsychiatry.2014.3039 · DOI · Paper page
  34. Lin, 2011, Trends in solids/liquids poisoning suicide rates in Taiwan: A test of the substitution hypothesis, BMC Public Health, 11, 712, 10.1186/1471-2458-11-712 · DOI
  35. Lipschitz, 2017, Understanding collaborative care implementation in the Department of Veterans Affairs: Core functions and implementation challenges, BMC Health Services Research, 17, 691, 10.1186/s12913-017-2601-9 · DOI
  36. Lubin, 2010, Decrease in suicide rates after a change of policy reducing access to firearms in adolescents: A naturalistic epidemiological study, Suicide and Life-Threatening Behavior, 40, 10.1521/suli.2010.40.5.421 · DOI · Paper page
  37. Luedtke, 2017, Evaluating the impact of treating the optimal subgroup, Statistical Methods in Medical Research, 26, 1630, 10.1177/0962280217708664 · DOI · Paper page
  38. McClellan, 1994, Does more intensive treatment of acute myocardial infarction in the elderly reduce mortality? Analysis using instrumental variables, Journal of the American Medical Association, 272, 859, 10.1001/jama.1994.03520110039026 · DOI · Paper page
  39. McCoy, 2016, Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing, JAMA Psychiatry, 73, 1064, 10.1001/jamapsychiatry.2016.2172 · DOI · Paper page
  40. McCoy, 2019, Research Domain Criteria scores estimated through natural language processing are associated with risk for suicide and accidental death, Depression and Anxiety, 36, 392, 10.1002/da.22882 · DOI
  41. McIntosh, 2016, Data science for mental health: A UK perspective on a global challenge, Lancet Psychiatry, 3, 993, 10.1016/S2215-0366(16)30089-X · DOI · Paper page
  42. McMain, 2018, Outcome trajectories and prognostic factors for suicide and self-harm behaviors in patients with borderline personality disorder following one year of outpatient psychotherapy, Journal of Personality Disorders, 32, 497, 10.1521/pedi_2017_31_309 · DOI
  43. Moscoe, 2015, Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: A review of current and best practice, Journal of Clinical Epidemiology, 68, 122, 10.1016/j.jclinepi.2014.06.021 · DOI · Paper page
  44. Mulder, 2016, The futility of risk prediction in psychiatry, The British Journal of Psychiatry: Journal of Mental Science, 209, 271, 10.1192/bjp.bp.116.184960 · DOI · Paper page
  45. Murphy, 2003, Optimal dynamic treatment regimes, Journal of the Royal Statistical Society - Series B: Statistical Methodology, 65, 331, 10.1111/1467-9868.00389 · DOI · Paper page
  46. Ning, 2017, Estimating treatment effects in observational studies with both prevalent and incident cohorts, Canadian Journal of Statistics, 45, 202, 10.1002/cjs.11317 · DOI
  47. Noma, 2017, Analysis of case-cohort designs with binary outcomes: Improving efficiency using whole-cohort auxiliary information, Statistical Methods in Medical Research, 26, 691, 10.1177/0962280214556175 · DOI
  48. Olbrich, 2015, Personalized medicine: Review and perspectives of promising baseline EEG biomarkers in major depressive disorder and attention deficit hyperactivity disorder, Neuropsychobiology, 72, 229, 10.1159/000437435 · DOI · Paper page
  49. Ondra, 2016, Methods for identification and confirmation of targeted subgroups in clinical trials: A systematic review, Journal of Biopharmaceutical Statistics, 26, 99, 10.1080/10543406.2015.1092034 · DOI · Paper page
  50. Penkower, 1988, Husbands' layoff and wives' mental health. A prospective analysis, Archives of General Psychiatry, 45, 994, 10.1001/archpsyc.1988.01800350028004 · DOI
  51. Perel, 2006, Systematic review of prognostic models in traumatic brain injury, BMC Medical Informatics and Decision Making, 6, 38, 10.1186/1472-6947-6-38 · DOI · Paper page
  52. Pestian, 2017, A machine learning approach to identifying the thought markers of suicidal subjects: A prospective multicenter trial, Suicide and Life-Threatening Behavior, 47, 112, 10.1111/sltb.12312 · DOI · Paper page
  53. Petkova, 2017, Statistical analysis plan for stage 1 EMBARC (establishing moderators and biosignatures of antidepressant response for clinical care) study, Contemporary Clinical Trials Communications, 6, 22, 10.1016/j.conctc.2017.02.007 · DOI · Paper page
  54. Prendes-Alvarez, 2018, Personalized medicine: Prediction of disease vulnerability in mood disorders, Neuroscience Letters, 669, 10, 10.1016/j.neulet.2016.09.049 · DOI
  55. Prieto-Merino, 2012, The science of risk models, European Journal of Preventive Cardiology, 19, 7, 10.1177/2047487312448995 · DOI
  56. Quinlivan, 2016, Which are the most useful scales for predicting repeat self-harm? A systematic review evaluating risk scales using measures of diagnostic accuracy, BMJ Open, 6, 10.1136/bmjopen-2015-009297 · DOI · Paper page
  57. Robins, 1995, Semiparametric efficiency in multivariate regression models with missing data, Journal of the American Statistical Association, 90, 122, 10.1080/01621459.1995.10476494 · DOI · Paper page
  58. Rosellini, 2017, Sexual assault victimization and mental health treatment, suicide attempts, and career outcomes among women in the us army, American Journal of Public Health, 107, 732, 10.2105/AJPH.2017.303693 · DOI
  59. Rosenbaum, 1983, The central role of the propensity score in observational studies for causal effects, Biometrika, 70, 41, 10.1093/biomet/70.1.41 · DOI · Paper page
  60. Rubin, 2012, Statistical issues and limitations in personalized medicine research with clinical trials, The International Journal of Biostatics, 8, 18
  61. Rudd, 2012, Brief cognitive behavioral therapy (BCBT) for suicidality in military populations, Military Psychology, 24, 592, 10.1080/08995605.2012.736325 · DOI · Paper page
  62. Rumshisky, 2016, Predicting early psychiatric readmission with natural language processing of narrative discharge summaries, Translational Psychiatry, 6, e921, 10.1038/tp.2015.182 · DOI · Paper page
  63. Rupasingha, 2006, The production of social capital in US counties, The Journal of Socio-Economics, 35, 83, 10.1016/j.socec.2005.11.001 · DOI · Paper page
  64. Saltiel, 2015, Major depressive disorder: Mechanism-based prescribing for personalized medicine, Neuropsychiatric Disease and Treatment, 2015, 875
  65. Silverman, 2015, The American Psychiatric Association practice guidelines for the psychiatric evaluation of adults, American Journal of Psychiatry, 172, 798, 10.1176/appi.ajp.2015.1720501 · DOI · Paper page
  66. Smith, 2014, Correcting for optimistic prediction in small data sets, American Journal of Epidemiology, 180, 318, 10.1093/aje/kwu140 · DOI · Paper page
  67. Svindseth, 2007, Patients' experience of humiliation in the admission process to acute psychiatric wards, Nordic Journal of Psychiatry, 61, 47, 10.1080/08039480601129382 · DOI · Paper page
  68. Swanson, 2017, Instrumental variable analyses in pharmacoepidemiology: What target trials do we emulate?, Current Epidemiology Reports, 4, 281, 10.1007/s40471-017-0120-1 · DOI
  69. Trivedi, 2016, Establishing moderators and biosignatures of antidepressant response in clinical care (EMBARC): Rationale and design, Journal of Psychiatric Research, 78, 11, 10.1016/j.jpsychires.2016.03.001 · DOI · Paper page
  70. VanderWeele, 2019, Selecting optimal subgroups for treatment using many covariates, Epidemiology, 30, 334, 10.1097/EDE.0000000000000991 · DOI · Paper page
  71. Venkataramani, 2016, Regression discontinuity designs in healthcare research, BMJ, 352, i1216, 10.1136/bmj.i1216 · DOI
  72. Visconti, 2018, Handling limited overlap in observational studies with cardinality matching, Observational Studies, 4, 217, 10.1353/obs.2018.0012 · DOI
  73. Waldrop, 2017, Measurement-based care in psychiatry, Journal of Psychosocial Nursing and Mental Health Services, 55, 30, 10.3928/02793695-20170818-01 · DOI · Paper page
  74. Wang, 2019, Minimal dispersion approximately balancing weights: Asymptotic properties and practical considerations, Biometrika, 10.1093/biomet/asz050 · DOI
  75. Wigmore, 2019, Genome-wide association study of antidepressant treatment resistance in a population-based cohort using health service prescription data and meta-analysis with GENDEP, The Pharmacogenomics Journal
  76. Yiu, 2018, Covariate association eliminating weights: A unified weighting framework for causal effect estimation, Biometrika, 105, 709, 10.1093/biomet/asy015 · DOI
  77. Zalsman, 2016, Suicide prevention strategies revisited: 10-year systematic review, Lancet Psychiatry, 3, 646, 10.1016/S2215-0366(16)30030-X · DOI · Paper page
  78. Zhang, 2012, Estimating optimal treatment regimes from a classification perspective, Stat, 1, 103, 10.1002/sta.411 · DOI · Paper page
  79. Zhang, 2018, Psychiatric stressor recognition from clinical notes to reveal association with suicide, Health Informatics Journal, 1460458218796598
  80. Zhao, 2012, Estimating individualized treatment rules using outcome weighted learning, Journal of the American Statistical Association, 107, 1106, 10.1080/01621459.2012.695674 · DOI · Paper page
  81. Zhou, 2017, Residual weighted learning for estimating individualized treatment rules, Journal of the American Statistical Association, 112, 169, 10.1080/01621459.2015.1093947 · DOI · Paper page
  82. Zhu, 2017, Greedy outcome weighted tree learning of optimal personalized treatment rules, Biometrics, 73, 391, 10.1111/biom.12593 · DOI
  83. Zubizarreta, 2012, Using mixed integer programming for matching in an observational study of kidney failure after surgery, Journal of the American Statistical Association, 107, 1360, 10.1080/01621459.2012.703874 · DOI · Paper page
  84. Zubizarreta, 2013, Stronger instruments via integer programming in an observational study of late preterm birth outcomes, Annals of Applied Statistics, 7, 25, 10.1214/12-AOAS582 · DOI

Cited by

See all 81

Metadata

Authors
Kessler, R. C., Bossarte, R. M., Luedtke, A., Zaslavsky, A. M., & Zubizarreta, J. R.
Venue
Behaviour Research and Therapy
Publisher
Elsevier BV
Published
2019-09
Volume
120
Pages
103412
ISSN
0005-7967 (ISSN-L)
OpenAlex
W2947984607
Cited by
81
References
38
FWCI
10.46
Type
journal article
Language
en
Open access
green
Licence
  • www.elsevier.com/tdm/userlicense/1.0/ (tdm)
  • www.elsevier.com/legal/tdmrep-license (tdm)
Funders
Alfred P. Sloan Foundation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Machine learning methods for developing precision treatment rules with observational data. https://app.cochat.ai/papers/10.1016/j.brat.2019.103412
The paper
Kessler, R. C., Bossarte, R. M., Luedtke, A., Zaslavsky, A. M., & Zubizarreta, J. R. (2019). Machine learning methods for developing precision treatment rules with observational data. Behaviour Research and Therapy. https://doi.org/10.1016/j.brat.2019.103412

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.