The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Reinforcement learning design for cancer clinical trials
Open accessAbstract
We develop reinforcement learning trials for discovering individualized treatment regimens for life‐threatening diseases such as cancer. A temporal‐difference learning method called Q‐learning is utilized that involves learning an optimal policy from a single training set of finite longitudinal patient trajectories. Approximating the Q‐function with time‐indexed parameters can be achieved by using support vector regression or extremely randomized trees. Within this framework, we demonstrate that the procedure can extract optimal strategies directly from clinical data without relying on the identification of any accurate mathematical models, unlike approaches based on adaptive design. We show that reinforcement learning has tremendous potential in clinical research because it can select actions that improve outcomes by taking into account delayed effects even when the relationship between actions and outcomes is not fully known. To support our claims, the methodology's practical utility is illustrated in a simulation analysis. In the immediate future, we will apply this general strategy to studying and identifying new treatments for advanced metastatic stage IIIB/IV non‐small cell lung cancer, which usually includes multiple lines of chemotherapy treatment. Moreover, there is significant potential of the proposed methodology for developing personalized treatment strategies in other cancers, in cystic fibrosis, and in other life‐threatening diseases. Copyright © 2009 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
A tutorial on support vector regression
Smola, A. J., & Schölkopf, B. (2004) · Statistics and Computing · doi:10.1023/b:stco.0000035301.49549.88 · Cited by 13,387
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Solutions of Ill-Posed Problems.
Bell, J. B., Tikhonov, A. N., & Arsenin, V. Y. (1978) · Mathematics of Computation · doi:10.2307/2006360 · Cited by 8,101
Paclitaxel–Carboplatin Alone or with Bevacizumab for Non–Small-Cell Lung Cancer
Sandler, A., Gray, R., Perry, M. C., Brahmer, J., Schiller, J. H., Dowlati, A., Lilenbaum, R., & Johnson, D. H. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa061884 · Cited by 5,761
Erlotinib in Previously Treated Non–Small-Cell Lung Cancer
Shepherd, F. A., Rodrigues Pereira, J., Ciuleanu, T., Tan, E. H., Hirsh, V., Thongprasert, S., Campos, D., Maoleekoonpiroj, S., Smylie, M., Martins, R., et al. (2005) · New England Journal of Medicine · doi:10.1056/nejmoa050753 · Cited by 5,419
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
The Mathematical Theory of Optimal Processes
Bellman, R., Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V., Mishchenko, E. F., Trirogoff, K. N., & Neustadt, L. W. (1965) · Econometrica · doi:10.2307/1911909 · Cited by 4,211
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1023/a:1022676722315 · Cited by 3,907
COMPARISON OF FOUR CHEMOTHERAPY REGIMENS FOR ADVANCED NON-SMALL-CELL LUNG CANCER
Chiller, J. O. H. S., Arrington, D. A. H., Elani, C. H. P. B., Anger, C., Andler, A. L. S., Rook, J. A. K., Hu, J., & Ohnson, D. A. H. J. (2002) · Cited by 3,730
Dynamic Programming and Markov Processes.
Freimer, M., & Howard, R. A. (1961) · Journal of the American Statistical Association · doi:10.2307/2282268 · Cited by 3,378
Scagliotti, G. V., Parikh, P., von Pawel, J., Biesma, B., Vansteenkiste, J., Manegold, C., Serwatowski, P., Gatzemeier, U., Digumarti, R., Zukin, M., et al. (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2007.15.0375 · Cited by 3,203
Support Vector Method for Function Approximation, Regression Estimation, and Signal Processing
Vapnik, V. N., Golowich, S. E., & Smola, A. J. (1997) · Cited by 2,757
Deterministic and Stochastic Optimal Control.
Bather, J. A., Fleming, W. H., & Rishel, R. (1976) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344363 · Cited by 2,690
Hanna, N., Shepherd, F. A., Fossella, F. V., Pereira, J. R., De Marinis, F., von Pawel, J., Gatzemeier, U., Tsao, T. C. Y., Pless, M., Muller, T., et al. (2004) · Journal of Clinical Oncology · doi:10.1200/jco.2004.08.163 · Cited by 2,426
Shepherd, F. A., Dancey, J., Ramlau, R., Mattson, K., Gralla, R., O’Rourke, M., Levitan, N., Gressot, L., Vincent, M., Burkes, R., et al. (2000) · Journal of Clinical Oncology · doi:10.1200/jco.2000.18.10.2095 · Cited by 2,181
Cover, T. M. (1965) · IEEE Transactions on Electronic Computers · doi:10.1109/pgec.1965.264137 · Cited by 2,044
XVI. Functions of positive and negative type, and their connection the theory of integral equations
Mercer, J. (1909) · Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character · doi:10.1098/rsta.1909.0016 · Cited by 2,001
A Technique for the Numerical Solution of Certain Integral Equations of the First Kind
Phillips, D. L. (1962) · Journal of the ACM · doi:10.1145/321105.321114 · Cited by 1,985
On the algorithmic implementation of multiclass kernel-based vector machines
Crammer, K., & Singer, Y. (2002) · Cited by 1,887
Steps toward Artificial Intelligence
Minsky, M. (1961) · Proceedings of the IRE · doi:10.1109/jrproc.1961.287775 · Cited by 1,675
Dynamic Programming: Deterministic and Stochastic Models
Bertsekas, D. P. (1987) · Medical Entomology and Zoology · Cited by 1,325
Optimal Dynamic Treatment Regimes
Murphy, S. A. (2003) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00389 · Cited by 1,007
A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines
Kimeldorf, G. S., & Wahba, G. (1970) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177697089 · Cited by 939
Modeling immunotherapy of the tumor - immune interaction
Kirschner, D., & Panetta, J. C. (1998) · Journal of Mathematical Biology · doi:10.1007/s002850050127 · Cited by 915
Tree-Based Batch Mode Reinforcement Learning
Ernst, D., Geurts, P., & Wehenkel, L. A. (2005) · doi:10.5555/1046920.1088690 · Cited by 854
Optimal Control Theory with Economic Applications
Becker, R. A., Sierstad, A., Sydsæter, K., & Sydsaeter, K. (1989) · The Scandinavian Journal of Economics · doi:10.2307/3440172 · Cited by 843
Load Forecasting Using Support Vector Machines: A Study on EUNITE Competition 2001
Chen, B.-J., Chang, M.-W., & Lin, C.-J. (2004) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2004.835679 · Cited by 841
TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play
Tesauro, G. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.215 · Cited by 787
On the Convergence of Stochastic Iterative Dynamic Programming Algorithms
Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994) · Neural Computation · doi:10.1162/neco.1994.6.6.1185 · Cited by 770
Skipper, H. E., Schabel, F. M., & Wilcox, W. S. (1964) · PubMed · Cited by 767
An experimental design for the development of adaptive treatment strategies
Murphy, S. A. (2005) · Statistics in Medicine · doi:10.1002/sim.2022 · Cited by 746
Lee, Y., Lin, Y., & Wahba, G. (2004) · Journal of the American Statistical Association · doi:10.1198/016214504000000098 · Cited by 741
Stroup, T. S., McEvoy, J. P., Swartz, M. S., Byerly, M. J., Glick, I. D., Canive, J. M., McGee, M. F., Simpson, G. M., Stevens, M. C., & Lieberman, J. A. (2003) · Schizophrenia Bulletin · doi:10.1093/oxfordjournals.schbul.a006986 · Cited by 636
de Pillis, L. G., Gu, W., & Radunskaya, A. E. (2006) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2005.06.037 · Cited by 575
Optimal Structural Nested Models for Optimal Sequential Decisions
Robins, J. M. (2004) · Lecture Notes in Statistics · doi:10.1007/978-1-4419-9076-1_11 · Cited by 569
Autonomous Inverted Helicopter Flight via Reinforcement Learning
Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., & Liang, E. (2006) · Springer Tracts in Advanced Robotics · doi:10.1007/11552246_35 · Cited by 553
Treatment of Non-small Cell Lung Cancer-Stage IIIA
Robinson, L. A., Ruckdeschel, J. C., Wagner, H., & Stevens, C. W. (2007) · Chest · doi:10.1378/chest.07-1379 · Cited by 457
Asynchronous stochastic approximation and Q-learning
Tsitsiklis, J. N. (1994) · Machine Learning · doi:10.1007/bf00993306 · Cited by 453
Tumor size, sensitivity to therapy, and design of treatment schedules.
Norton, L., & Simon, R. M. (1977) · PubMed · Cited by 447
Feature-based methods for large scale dynamic programming
Tsitsiklis, J. N., & van Roy, B. (1996) · Machine Learning · doi:10.1007/bf00114724 · Cited by 432
A Mathematical Tumor Model with Immune Resistance and Drug Therapy: An Optimal Control Approach
De Pillis, L. G., & Radunskaya, A. (2001) · Computational and Mathematical Methods in Medicine · doi:10.1080/10273660108833067 · Cited by 425
Bunn, P. A., & Kelly, K. L. (1998) · PubMed · Cited by 400
Kernel-Based Reinforcement Learning
(2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_100235 · Cited by 367
Role of optimal control theory in cancer chemotherapy
Swan, G. W. (1990) · Mathematical Biosciences · doi:10.1016/0025-5564(90)90021-p · Cited by 305
Elevator Group Control Using Multiple Reinforcement Learning Agents
Crites, R. H., & Barto, A. G. (1998) · Machine Learning · doi:10.1023/a:1007518724497 · Cited by 275
de Pillis, L. G., Gu, W., Fister, K. R., Head, T., Maples, K., Murugan, A., Neal, T., & Yoshida, K. (2007) · Mathematical Biosciences · doi:10.1016/j.mbs.2006.05.003 · Cited by 245
Modified Policy Iteration Algorithms for Discounted Markov Decision Problems
Puterman, M. L., & Shin, M. C. (1978) · Management Science · doi:10.1287/mnsc.24.11.1127 · Cited by 239
Dynamic Multidrug Therapies for HIV: Optimal and STI Control Approaches
M. Adams, B., 1. Center for Research in Scientific Computation, N. C. S. U., T. Banks, H., Kwon, H.-D., T. Tran, H., & 2. Center for Research in Scientific Computation, N. C. S. U. (2004) · Mathematical Biosciences and Engineering · doi:10.3934/mbe.2004.1.223 · Cited by 228
Demystifying Optimal Dynamic Treatment Regimes
Moodie, E. E. M., Richardson, T. S., & Stephens, D. A. (2007) · Biometrics · doi:10.1111/j.1541-0420.2006.00686.x · Cited by 224
A Generalization Error for Q-Learning.
Murphy, S. A. (2005) · PubMed · Cited by 216
Developing adaptive treatment strategies in substance abuse research
Murphy, S. A., Lynch, K. G., Oslin, D., McKay, J. R., & TenHave, T. (2007) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2006.09.008 · Cited by 211
O'Byrne, K. J., Dalgleish, A. G., Browning, M. J., Steward, W. P., & Harris, A. L. (2000) · European Journal of Cancer · doi:10.1016/s0959-8049(99)00241-5 · Cited by 206
Pirker, R., Szczesna, A., von Pawel, J., Krzakowski, M., Ramlau, R., Park, K., Gatzemeier, U., Bajeta, E., Emig, M., & Pereira, J. R. (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2008.26.15_suppl.3 · Cited by 172
A Support Vector Approach to Censored Targets
Shivaswamy, P. K., Chu, W., & Jansche, M. (2007) · Proceedings · doi:10.1109/icdm.2007.93 · Cited by 156
Programming backgammon using self-teaching neural nets
Tesauro, G. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(01)00110-2 · Cited by 153
Pharmacokinetic and pharmacodynamic modeling in vivo.
Holford, N. H. G., & Sheiner, L. B. (1981) · PubMed · Cited by 151
Farrar, J. D., Katz, K. H., Windsor, J. A., Thrush, G. R., Scheuermann, R. H., Uhr, J. W., & Street, N. E. (1999) · PubMed · Cited by 147
Ivanov, V. V., & Gupta, G. D. (1978) · Journal of Applied Mechanics · doi:10.1115/1.3424255 · Cited by 142
Evaluating multiple treatment courses in clinical trials
Thall, P. F., Millikan, R. E., & Sung, H.-G. (2000) · Statistics in Medicine · Cited by 141
Statistical Consideration of Adaptive Methods in Clinical Development
Chow, S., Chang, M., & Pong, A. (2005) · Journal of Biopharmaceutical Statistics · doi:10.1081/bip-200062277 · Cited by 136
GOSAVII, A., Bandla, N., & Das, T. K. (2002) · IIE Transactions · doi:10.1080/07408170208928908 · Cited by 135
MCATS, F., Murphy, S. A., Oslin, D. W., Rush, A. J., & Zhu, J. (2006) · Neuropsychopharmacology · doi:10.1038/sj.npp.1301241 · Cited by 123
Selecting Therapeutic Strategies Based on Efficacy and Death in Multicourse Clinical Trials
Thall, P. F., Sung, H., & Estey, E. H. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502753479202 · Cited by 119
Clinical data based optimal STI strategies for HIV: a reinforcement learning approach
Ernst, D., Stan, G.-B. V., Gonçalves, J. M., & Wehenkel, L. A. (2006) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2006.377527 · Cited by 111
Texas Medication Algorithm Project, Phase 3 (TMAP-3)
Rush, A. J., Crismon, M. L., Kashner, T. M., Toprac, M. G., Carmody, T. J., Trivedi, M. H., Suppes, T. M., Miller, A. L., Biggs, M. M., Shores‐Wilson, K., et al. (2003) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.v64n0402 · Cited by 108
Considerations for Second-Line Therapy of Non-Small Cell Lung Cancer
Stinchcombe, T. E., & Socinski, M. A. (2008) · The Oncologist · doi:10.1634/theoncologist.13-s1-28 · Cited by 102
Maintenance pemetrexed plus best supportive care (BSC) versus placebo plus BSC: A phase III study
Ciuleanu, T., Brodowicz, T., Belani, C. P., Kim, J., Krzakowski, M. J., Laack, E., Wu, Y.-L. L. C., Peterson, P. M., Adachi, S., & Zielinski, C. C. (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2008.26.15_suppl.8011 · Cited by 98
Müller, M., Gounari, F., Prifti, S., Hacker, H. J., Schirrmacher, V., & Khazaie, K. (1998) · PubMed · Cited by 97
Evaluating multiple treatment courses in clinical trials
Thall, P. F., Millikan, R. E., & Sung, H. (2000) · Statistics in Medicine · Cited by 87
Adaptive treatment of epilepsy via batch-mode reinforcement learning
Guez, A., Vincent, R. D., Avoli, M., & Pineau, J. (2008) · Cited by 76
Thall, P. F., Wooten, L. H., Logothetis, C. J., Millikan, R. E., & Tannir, N. M. (2007) · Statistics in Medicine · doi:10.1002/sim.2894 · Cited by 75
Constructing evidence-based treatment strategies using methods from computer science
Pineau, J., Bellemare, M. G., Rush, A. J., Ghizaru, A., & Murphy, S. A. (2007) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2007.01.005 · Cited by 49
Application of reinforcement learning for segmentation of transrectal ultrasound images
Sahba, F., Tizhoosh, H. R., & Salama, M. M. A. (2008) · BMC Medical Imaging · doi:10.1186/1471-2342-8-8 · Cited by 47
Fidias, P. M., Dakhil, S. R., Lyss, A. P., Loesch, D., Waterhouse, D. M., Cunneen, J. L., Chen, R., Treat, J., Obasaju, C. K., & Schiller, J. H. (2007) · Journal of Clinical Oncology · doi:10.1200/jco.2007.25.18_suppl.lba7516 · Cited by 44
Adaptive Decision Making in a Lymphocyte Infusion Trial
Thall, P. F., Inoue, L. Y. T., & Martin, T. G. (2002) · Biometrics · doi:10.1111/j.0006-341x.2002.00560.x · Cited by 30
Dudley, M. E., Sundberg, J. P., & Roopenian, D. C. (1996) · International Journal of Cancer · doi:10.1002/(sici)1097-0215(19960117)65:2<204::aid-ijc13>3.0.co;2-d · Cited by 18
Strategies for Combining Chemotherapy and Biotherapy in Melanoma
Buzaid, A. C. (2000) · Cancer Control · doi:10.1177/107327480000700214 · Cited by 16
Widening bottlenecks in drug discovery Glimpses from Drug Discovery Technology Europe 2005
Högberg, T. (2005) · Drug Discovery Today · doi:10.1016/s1359-6446(05)03490-2 · Cited by 3
Cited by
Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906
Reinforcement Learning in Healthcare: A Survey
Yu, C., Liu, J., Nemati, S., & Yin, G. (2023) · ACM Computing Surveys · doi:10.1145/3477600 · Cited by 628
Q-Learning: Theory and Applications
Clifton, J., & Laber, E. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041220 · Cited by 492
A Robust Method for Estimating Optimal Treatment Regimes
Zhang, B., Tsiatis, A. A., Laber, E. B., & Davidian, M. (2012) · Biometrics · doi:10.1111/j.1541-0420.2012.01763.x · Cited by 422
Kosorok, M. R., & Laber, E. B. (2019) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-030718-105251 · Cited by 370
Self-citation
Metadata
- Authors
- Zhao, Y., Kosorok, M. R., & Zeng, D.
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2009-09-11
- Volume
- 28
- Issue
- 26
- Pages
- 3294-3315
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2156682402
- Cited by
- 271
- References
- 86
- FWCI
- 7.11
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.