Towards an early warning system for monitoring of cancer patients using hybrid interactive machine learning
Open accessAbstract
Background: The use of smartphone apps in cancer patients undergoing systemic treatment can promote the early detection of symptoms and therapy side effects and may be supported by machine learning (ML) for timely adaptation of therapies and reduction of adverse events and unplanned admissions. Objective: We aimed to create an Early Warning System (EWS) to predict situations where supportive interventions become necessary to prevent unplanned visits. For this, dynamically collected standardized electronic patient reported outcome (ePRO) data were analyzed in context with the patient's individual journey. Information on well-being, vital parameters, medication, and free text were also considered for establishing a hybrid ML model. The goal was to integrate both the strengths of ML in sifting through large amounts of data and the long-standing experience of human experts. Given the limitations of highly imbalanced datasets (where only very few adverse events are present) and the limitations of humans in overseeing all possible cause of such events, we hypothesize that it should be possible to combine both in order to partially overcome these limitations. Methods: The prediction of unplanned visits was achieved by employing a white-box ML algorithm (i.e., rule learner), which learned rules from patient data (i.e., ePROs, vital parameters, free text) that were captured via a medical device smartphone app. Those rules indicated situations where patients experienced unplanned visits and, hence, were captured as alert triggers in the EWS. Each rule was evaluated based on a cost matrix, where false negatives (FNs) have higher costs than false positives (FPs, i.e., false alarms). Rules were then ranked according to the costs and priority was given to the least expensive ones. Finally, the rules with higher priority were reviewed by two oncological experts for plausibility check and for extending them with additional conditions. This hybrid approach comprised the application of a sensitive ML algorithm producing several potentially unreliable, but fully human-interpretable and -modifiable rules, which could then be adjusted by human experts. Results: From a cohort of 214 patients and more than 16’000 available data entries, the machine-learned rule set achieved a recall of 19% on the entire dataset and a precision of 5%. We compared this performance to a set of conditions that a human expert had defined to predict adverse events. This “human baseline” did not discover any of the adverse events recorded in our dataset, i.e., it came with a recall and precision of 0%. Despite more plentiful results were expected by our machine learning approach, the involved medical experts a) had understood and were able to make sense of the rules and b) felt capable to suggest modification to the rules, some of which could potentially increase their precision. Suggested modifications of rules included e.g., adding or tightening certain conditions to make them less sensitive or changing the rule consequences: sometimes further monitoring the situation, applying certain test (such as a CRP test) or applying some simple pain-relieving measures was deemed sufficient, making a costly consultation with the physician unnecessary. We can thus conclude that it is possible to apply machine learning as an inspirational tool that can help human experts to formulate rules for an EWS. While humans seem to lack the ability to define such rules without such support, they are capable of modifying the rules to increase their precision and generalizability. Conclusions: Learning rules from dynamic ePRO datasets may be used to assist human experts in establishing an early warning system for cancer patients in outpatient settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI-enabled models may detect early signals of treatment-related toxicity by integrating wearable-derived data with patient-reported symptoms and clinical variables.
“AI-enabled models may detect early signals of treatment-related toxicity by integrating wearable-derived data with patient-reported symptoms and clinical variables.31”
From Artificial Intelligence in Oncology: Practical Applications Across Clinical Care, Scholarship, and Translation. (2026) · cites this paper as [31]
The Introduction and Methods describe an ML early-warning approach using patient-reported symptoms, vital parameters, and other clinical information. The Discussion and conclusions (page unavailable) describe wearable-derived data only as a possible future improvement; the study did not use it to detect early treatment-related toxicity and explicitly says it did not aim to discover serious adverse events as early as possible. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
Cohen, W. W. (1995) · Elsevier eBooks · doi:10.1016/b978-1-55860-377-6.50023-2 · Cited by 3,890 · see citing works
Basch, E., Deal, A. M., Dueck, A. C., Scher, H. I., Kris, M. G., Hudis, C., & Schrag, D. (2017) · JAMA · doi:10.1001/jama.2017.7156 · Cited by 2,302 · see citing works
Data mining and knowledge discovery handbook
(2011) · Choice Reviews Online · doi:10.5860/choice.48-5729 · Cited by 1,174 · see citing works
Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906 · see citing works
Machine learning in medical applications: a review of state-of-the-art methods
Shehab, M., Abualigah, L., Shambour, Q., Abu-Hashem, M. A., Shambour, M. K. Y., Alsalibi, A. I., & Gandomi, A. H. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105458 · Cited by 542 · see citing works
Handelman, G. S., Kok, H. K., Chandra, R. V., Razavi, A. H., Huang, S., Brooks, M., Lee, M. J., & Asadi, H. (2018) · American Journal of Roentgenology · doi:10.2214/ajr.18.20224 · Cited by 410 · see citing works
Adoption of telemedicine: from pilot stage to routine delivery
Zanaboni, P., & Wootton, R. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-1 · Cited by 317 · see citing works
Role of machine learning in medical research: A survey
Garg, A., & Mago, V. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100370 · Cited by 255 · see citing works
Patient-reported outcomes in cancer care — hearing the patient voice at greater volume
LeBlanc, T. W., & Abernethy, A. P. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.153 · Cited by 246 · see citing works
Interactive machine learning: experimental evidence for the human in the algorithmic loop
Holzinger, A., Plass, M., Kickmeier-Rust, M., Holzinger, K., Crişan, G. C., Pintea, C.-M., & Palade, V. (2018) · Applied Intelligence · doi:10.1007/s10489-018-1361-5 · Cited by 231 · see citing works
Ginestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · see citing works
Expert-augmented machine learning
Gennatas, E. D., Friedman, J. H., Ungar, L., Pirracchio, R., Eaton, E., Reichmann, L. G., Interian, Y., Luna, J. M., Simone, C. B., Auerbach, A., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1906831117 · Cited by 123 · see citing works
Luna, J. M., Chao, H., Diffenderfer, E. S., Valdés, G., Chinniah, C., Grace, X., Cengel, K. A., Solberg, T. D., Berman, A. T., & Simone, C. B. (2019) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2019.01.003 · Cited by 99 · see citing works
Bull, L. M., Lunt, M., Martin, G. P., Hyrich, K. L., & Sergeant, J. C. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00078-z · Cited by 69 · see citing works
The use of machine learning and artificial intelligence within pediatric critical care
Shah, N. A., Arshad, A., Mazer, M. B., Carroll, C. L., Shein, S. L., & Remy, K. E. (2022) · Pediatric Research · doi:10.1038/s41390-022-02380-6 · Cited by 64 · see citing works
Klement, W., & Emam, K. E. (2023) · Journal of Medical Internet Research · doi:10.2196/48763 · Cited by 61 · see citing works
Iivanainen, S., Ekström, J., Virtanen, H., Kataja, V. V., & Koivunen, J. P. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01564-0 · Cited by 52 · see citing works
Mining geriatric assessment data for in-patient fall prediction models and high-risk subgroups
Marschollek, M., Gövercin, M., Rust, S., et al. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-19 · Cited by 51 · see citing works
Schmalz, O., Jacob, C., Ammann, J., Liss, B., Iivanainen, S., Kammermann, M., Koivunen, J., Klein, A., & Popescu, R. A. (2020) · Journal of Medical Internet Research · doi:10.2196/18655 · Cited by 50 · see citing works
Nyrop, K. A., Deal, A. M., Reeve, B. B., Basch, E., Chen, Y. T., Park, J., Shachar, S. S., Carey, L., Reeder-Hayes, K. E., Dees, E. C., et al. (2020) · Cancer · doi:10.1002/cncr.32898 · Cited by 39 · see citing works
Amer, A. Y. A., Wouters, F., Vranken, J., et al. (2020) · Sensors · doi:10.3390/s20226593 · Cited by 35 · see citing works
Credit scoring by feature-weighted support vector machines
Shi, J., Zhang, S., & Qiu, L. (2013) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/jzus.c1200205 · Cited by 31 · see citing works
Machine Learning-Based Cardiac Arrest Prediction for Early Warning System
Chae, M., Gil, H., Cho, N., & Lee�, H. (2022) · Mathematics · doi:10.3390/math10122049 · Cited by 26 · see citing works
An alternative text representation to TF-IDF and Bag-of-WordsPreprint
Xu, Z., Xu, Chen, M., Weinberger, K. Q., & Sha, F. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.6770 · Cited by 24 · see citing works
Gong, L., Gong, J., Sun, X., Yu, L., Liao, B. Y., Chen, X., & Li, Y. (2023) · Frontiers in Immunology · doi:10.3389/fimmu.2023.1138489 · Cited by 24 · see citing works
Predicting Hospital Re-Admissions from Nursing Care Data of Hospitalized Patients
Lodhi, M. K., Ansari, R., Yao, Y., Keenan, G. M., Wilkie, D. J., & Khokhar, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-62701-4_14 · Cited by 23 · see citing works
Machine Learning Based Early Detection System of Cardiac Arrest
Liu, J.-H., Chang, H.-K., Wu, C.-T., et al. (2019) · doi:10.1109/taai48200.2019.8959922 · Cited by 14 · see citing works
Collado‐Borrell, R., Escudero‐Vilaplana, V., Narrillos-Moraza, Á., et al. (2022) · PubMed · Cited by 13 · see citing works
Trojan, A., Leuthold, N., Thomssen, C., Rody, A., Winder, T., Jakob, A., Egger, C., Held, U., & Jackisch, C. (2021) · Journal of Medical Internet Research · doi:10.2196/29271 · Cited by 12 · see citing works
Self-citationTrojan, A., Bättig, B., Mannhart, M., Seifert, B., Brauchbar, M., & Egbring, M. (2021) · JMIR Cancer · doi:10.2196/26950 · Cited by 11 · see citing works
Self-citationProtocol for a systematic review of prognostic models for recurrent events in chronic conditions
Watson, V., Smith, C. T., & Bonnett, L. J. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-0070-9 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [26]Domingos P. Metacost: a general method for making classifiers cost-sensitive. Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (1999). p. 155– 64. 10.1145/312129.312220 · DOI
Cited by
Sorted by citations
Hundal, J., Veettil, A. A. V., Chung, C., & Hosseini, M. (2026) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk-26-517280 · Cited by 1 · see citing works
- Authors
- Trojan, A., Laurenzi, E., Jüngling, S., Roth, S., Kiessling, M., Atassi, Z., Kadvany, Y., Mannhart, M., Jackisch, C., Kullak-Ublick, G., & Witschel, H. F.
- Affiliations
- University Hospital Zurich (CH)
- FHNW University of Applied Sciences and Arts Northwestern Switzerland (CH)
- Breast Center (CH)
- Sana Klinikum Offenbach (DE)
- Corresponding author
- Trojan, Andreas
- ORCID
- Venue
- Frontiers in Digital Health
- Publisher
- Frontiers Media SA
- Published
- 2024-08-14
- Volume
- 6
- Pages
- 1443987-1443987
- ISSN
- 2673-253X (ISSN-L)
- PMC
- PMC11349615
- OpenAlex
- W4401592572
- Cited by
- 10
- References
- 32
- Citations by year
- 2026: 6 · 2025: 4
- FWCI
- 3.06
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Digital Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
- Institutional Repository (IHS Vienna) (cc-by)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Health Sciences › Medicine › Oncology › Cancer survivorship and care
- Keywords
- early warning systems, cancer patients, electronic patient-reported outcomes, unplanned hospital visits, adverse event prediction, hybrid machine learning, rule-based learning, mobile application, patient monitoring, class imbalance, human-in-the-loop, expert validation, treatment side effects, vital parameters, free-text data, clinical decision support, precision oncology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
