CoChat
Sign Up
PreprintCited by 12Advanced Causal Inference TechniquesMachine Learning in HealthcareAdvanced Bandit Algorithms Research

Learning Cost-Effective Treatment Regimes using Markov Decision Processes

Open access
Lakkaraju, Himabindu, Rudin, Cynthia·2016·arXiv
DOI 10.48550/arxiv.1610.06972arXiv 1610.06972
Open this paper in CoChat — free accountOpen PDF

Abstract

Decision makers, such as doctors and judges, make crucial decisions such as recommending treatments to patients, and granting bails to defendants on a daily basis. Such decisions typically involve weighting the potential benefits of taking an action against the costs involved. In this work, we aim to automate this task of learning \emph{cost-effective, interpretable and actionable treatment regimes}. We formulate this as a problem of learning a decision list -- a sequence of if-then-else rules -- which maps characteristics of subjects (eg., diagnostic test results of patients) to treatments. We propose a novel objective to construct a decision list which maximizes outcomes for the population, and minimizes overall costs. We model the problem of learning such a list as a Markov Decision Process (MDP) and employ a variant of the Upper Confidence Bound for Trees (UCT) strategy which leverages customized checks for pruning the search space effectively. Experimental results on real world observational data capturing judicial bail decisions and treatment recommendations for asthma patients demonstrate the effectiveness of our approach.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1610.06972

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior research often overlooks the selective labels problem, evaluating models by computing traditional metrics such as AUC or accuracy only on data points for which ground truth labels are available.

    “While prior research has explored applications of machine learning to domains where human judgments are part of the data labeling process (crime [7, 21, 43], medical diagnosis [6, 22]), the selective labels problem is often overlooked, with evaluation of the models typically carried out by computing traditional metrics such as AUC or accuracy only on the data points for which ground truth labels”

    From The Selective Labels Problem (2017) · cites this paper as [22]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. [1][1] Eva-Maria Abulesz and Gerasimos Lyberatos. Novel approach for determining optimal treatment regimen for cancer cemotherapy. International journal of systems science , 19(8):1483-1497, 1988. -
  2. [3][3] James O Berger, Xiaojing Wang, and Lei Shen. A bayesian approach to subgroup identification. Journal of biopharmaceutical statistics , 24(1):110-129, 2014. -
  3. [4][4] Jacob Bien and Robert Tibshirani. Classification by set cover: The prototype vector machine. arXiv preprint arXiv:0908.2284 , 2009. - · arXiv
  4. [5][5] Louis-Philippe Boulet, Marie- ` Eve Boulay, Guylaine Gauthier, Livia Battisti, Val´ erie Chabot, MarieFrance Beauchesne, Denis Villeneuve, and Patricia Cˆ ot´ e. Benefits of an asthma education program provided at primary care sites on asthma outcomes. Respiratory medicine , 109(8):991-1000, 2015. -
  5. [6][6] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games , 4(1):1-43, 2012. -
  6. [8][8] Ralph B DAgostino. Estimating treatment effects using observational data. Jama , 297(3):314-316, 2007. -
  7. [9][9] Ailin Fan, Wenbin Lu, Rui Song, et al. Sequential advantage selection for optimal treatment regime. The Annals of Applied Statistics , 10(1):32-53, 2016. -
  8. [10][10] Jared C Foster, Jeremy MG Taylor, and Stephen J Ruberg. Subgroup identification from randomized clinical trial data. Statistics in medicine , 30(24):28672880, 2011. -
  9. [11][11] Kosuke Imai, Marc Ratkovic, et al. Estimating treatment effect heterogeneity in randomized program evaluation. The Annals of Applied Statistics , 7(1):443-470, 2013. -
  10. [12][12] Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for casebased reasoning and prototype classification. In Advances in Neural Information Processing Systems , pages 1952-1960, 2014. -
  11. [13][13] Levente Kocsis and Csaba Szepesv´ ari. Bandit based monte-carlo planning. In European conference on machine learning , pages 282-293. Springer, 2006. -
  12. [14][14] EB Laber and YQ Zhao. Tree-based methods for individualized treatment regimes. Biometrika , 102(3):501-514, 2015. -
  13. [15][15] Eric B Laber, Daniel J Lizotte, Min Qian, William E Pelham, and Susan A Murphy. Dynamic treatment regimes: Technical challenges and applications. Electronic journal of statistics , 8(1):1225, 2014. -
  14. [17][17] Himabindu Lakkaraju and Jure Leskovec. Confusions over time: An interpretable bayesian model to characterize trends in decision making. In Advances in Neural Information Processing Systems (NIPS) , 2016. -
  15. [20][20] Wei-Yin Loh, Xu He, and Michael Man. A regression tree approach to identifying subgroups with differential treatment effects. Statistics in medicine , 34(11):1818-1833, 2015. -
  16. [22][22] Jared K Lunceford and Marie Davidian. Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Statistics in medicine , 23(19):2937-2960, 2004. -
  17. [23][23] Douglas B Marlowe, David S Festinger, Karen L Dugosh, Kathleen M Benasutti, Gloria Fox, and Jason R Croft. Adaptive programming improves outcomes in drug court an experimental trial. Criminal justice and behavior , 39(4):514-532, 2012. -
  18. [24][24] James J McGough and Stephen V Faraone. Estimating the size of treatment effects: moving beyond p values. Psychiatry (1550-5952) , 6(10), 2009. -
  19. [25][25] Erica EM Moodie, Bibhas Chakraborty, and Michael S Kramer. Q-learning for estimating optimal dynamic treatment rules from observational data. Canadian Journal of Statistics , 40(4):629-645, 2012. -
  20. [26][26] Erica EM Moodie, Nema Dean, and Yue Ru Sun. Q-learning: Flexible learning about useful utilities. Statistics in Biosciences , 6(2):223-243, 2014. -
  21. [27][27] Jorge Pereira, Priscilla Porto-Figueira, Carina Cavaco, Khushman Taunk, Srikanth Rapole, Rahul Dhakne, Hampapathalu Nagarajaram, and Jos´ e S Cˆ amara. Breath analysis as a potential and noninvasive frontier in disease diagnosis: an overview. Metabolites , 5(1):3-55, 2015. -
  22. [28][28] Min Qian and Susan A Murphy. Performance guarantees for individualized treatment rules. Annals of statistics , 39(2):1180, 2011. -
  23. [29][29] James M Robins. Correcting for non-compliance in randomized trials using structural nested mean models. Communications in Statistics-Theory and methods , 23(8):2379-2412, 1994. -
  24. [30][30] Richard N Shiffman. Representation of clinical practice guidelines in conventional and augmented decision tables. Journal of the American Medical Informatics Association , 4(5):382-393, 1997. -
  25. [31][31] Xiaogang Su, Chih-Ling Tsai, Hansheng Wang, David M Nickerson, and Bogong Li. Subgroup analysis via recursive partitioning. Journal of Machine Learning Research , 10(Feb):141-158, 2009. -
  26. [32][32] Lu Tian, Ash A Alizadeh, Andrew J Gentles, and Robert Tibshirani. A simple method for estimating interactions between a treatment and a large number of covariates. Journal of the American Statistical Association , 109(508):1517-1532, 2014. -
  27. [33][33] Stijn Vansteelandt, Marshall Joffe, et al. Structural nested models and g-estimation: The partially realized promise. Statistical Science , 29(4):707-731, 2014. -
  28. [34][34] Michael P Wallace and Erica EM Moodie. Personalizing medicine: a review of adaptive treatment strategies. Pharmacoepidemiology and drug safety , 23(6):580-585, 2014. -
  29. [35][35] Baqun Zhang, Anastasios A Tsiatis, Eric B Laber, and Marie Davidian. A robust method for estimating optimal treatment regimes. Biometrics , 68(4):1010-1018, 2012. -
  30. [36][36] Yichi Zhang, Eric B. Laber, Anastasios Tsiatis, and Marie Davidian. Using decision lists to construct interpretable and parsimonious treatment regimes. Biometrics , 71(4):895-904, 2015. -
  31. [37][37] Yichi Zhang, Eric B Laber, Anastasios Tsiatis, and Marie Davidian. Interpretable dynamic treatment regimes. arXiv preprint arXiv:1606.01472 , 2016. - · arXiv
  32. [38][38] Ying-Qi Zhao, Donglin Zeng, Eric B Laber, Rui Song, Ming Yuan, and Michael Rene Kosorok. Doubly robust learning for estimating individualized treatment with censored data. Biometrika , 102(1):151168, 2015. -
  33. [39][39] Yingqi Zhao, Donglin Zeng, A John Rush, and Michael R Kosorok. Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association , 107(499):1106-1118, 2012. -
  34. [40][40] Yufan Zhao, Michael R Kosorok, and Donglin Zeng. Reinforcement learning design for cancer clinical trials. Statistics in medicine , 28(26):3294-3315, 2009.

Cited by

Sorted by citations

Authors
Lakkaraju, H., & Rudin, C.
Affiliations
Duke University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2016
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1610.06972
arXiv
1610.06972
OpenAlex
W2545158281
Cited by
12
References
3
Citations by year
2025: 1 · 2024: 2 · 2022: 1 · 2019: 3 · 2018: 2 · 2017: 1 · 2016: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
Keywords
decision lists, Markov decision process, automated decision-making, interpretable decision rules, cost minimization, search space pruning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review