Application of convex hull analysis for the evaluation of data heterogeneity between patient populations of different origin and implications of hospital bias in downstream machine-learning-based data processing: A comparison of 4 critical-care patient datasets
Open accessAbstract
Machine learning (ML) models are developed on a learning dataset covering only a small part of the data of interest. If model predictions are accurate for the learning dataset but fail for unseen data then generalization error is considered high. This problem manifests itself within all major sub-fields of ML but is especially relevant in medical applications. Clinical data structures, patient cohorts, and clinical protocols may be highly biased among hospitals such that sampling of representative learning datasets to learn ML models remains a challenge. As ML models exhibit poor predictive performance over data ranges sparsely or not covered by the learning dataset, in this study, we propose a novel method to assess their generalization capability among different hospitals based on the convex hull (CH) overlap between multivariate datasets. To reduce dimensionality effects, we used a two-step approach. First, CH analysis was applied to find mean CH coverage between each of the two datasets, resulting in an upper bound of the prediction range. Second, 4 types of ML models were trained to classify the origin of a dataset (i.e., from which hospital) and to estimate differences in datasets with respect to underlying distributions. To demonstrate the applicability of our method, we used 4 critical-care patient datasets from different hospitals in Germany and USA. We estimated the similarity of these populations and investigated whether ML models developed on one dataset can be reliably applied to another one. We show that the strongest drop in performance was associated with the poor intersection of convex hulls in the corresponding hospitals' datasets and with a high performance of ML methods for dataset discrimination. Hence, we suggest the application of our pipeline as a first tool to assess the transferability of trained models. We emphasize that datasets from different hospitals represent heterogeneous data sources, and the transfer from one database to another should be performed with utmost care to avoid implications during real-world applications of the developed models. Further research is needed to develop methods for the adaptation of ML models to new hospitals. In addition, more work should be aimed at the creation of gold-standard datasets that are large and diverse with data from varied application sites.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Substantial overlap in feature ranges between datasets generally suggests strong potential for cross-hospital generalizability.
“a pattern that generally suggests strong potential for cross-hospital generalizability [72]”
From Toward generalizable and interpretable machine learning models in healthcare: Insights from ICU outcome predictions (2026) · cites this paper as [72]
The Introduction (page unavailable) says convex-hull overlap can provide a first estimate of whether populations are similar enough for model generalization. But it also says that even extensive overlap may conceal distribution differences that impair generalization; overlap alone does not establish the claimed strong potential. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430 · see citing works
Chen, Y., Wang, L., Kang, Q., Zhang, X., Yu, G., Wan, X., Wang, J., & Zhu, K. (2017) · Cellular Physiology and Biochemistry · doi:10.1159/000477308 · Cited by 10,730 · see citing works
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2019) · Apollo - University of Cambridge Repository · doi:10.17863/cam.64134 · Cited by 10,460 · see citing works
Acute Respiratory Distress Syndrome
Ards Definition Task Force (2012) · JAMA · doi:10.1001/jama.2012.5669 · Cited by 8,806 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
Deep learning for healthcare: review, opportunities and challenges
Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bbx044 · Cited by 3,076 · see citing works
Schubert, E., Sander, J., Ester, M., Kriegel, H., & Xu, X. (2017) · ACM Transactions on Database Systems · doi:10.1145/3068335 · Cited by 2,789 · see citing works
Torralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · see citing works
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · see citing works
An efficient algorith for determining the convex hull of a finite planar set
Graham, R. (1972) · Information Processing Letters · doi:10.1016/0020-0190(72)90045-2 · Cited by 1,670 · see citing works
An interpretable mortality prediction model for COVID-19 patients
Li, Y., Zhang, H., Gonçalves, J. M., Xiao, Y., Wang, M., Guo, Y., Sun, C., Tang, X., Jing, L., Zhang, M., et al. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0180-7 · Cited by 1,304 · see citing works
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0213-5 · Cited by 1,290 · see citing works
Assessing the Generalizability of Prognostic Information
Justice, A. C., Covinsky, K. E., & Berlin, J. A. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-6-199903160-00016 · Cited by 1,160 · see citing works
A clinically applicable approach to continuous prediction of future acute kidney injury
Tomašev, N., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Mottram, A., Meyer, C., Ravuri, S., Protsyuk, I., et al. (2019) · Nature · doi:10.1038/s41586-019-1390-1 · Cited by 1,086 · see citing works
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · see citing works
Understanding deep convolutional networks
Mallat, S. (2016) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2015.0203 · Cited by 698 · see citing works
Debray, T. P. A., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E. W., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.06.018 · Cited by 597 · see citing works
From hype to reality: data science enabling personalized medicine
Fröhlich, H., Balling, R., Beerenwinkel, N., Kohlbacher, O., Kumar, S., Lengauer, T., Maathuis, M. H., Moreau, Y., Murphy, S. A., Przytycka, T. M., et al. (2018) · BMC Medicine · doi:10.1186/s12916-018-1122-7 · Cited by 413 · see citing works
Self-citationA Review of Challenges and Opportunities in Machine Learning for Health
Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L., Chen, I. Y., & Ranganath, R. (2018) · PubMed Central · doi:10.48550/arxiv.1806.00388 · Cited by 347 · see citing works
Deep learning algorithm predicts diabetic retinopathy progression in individual patients
Arcadu, F., Benmansour, F., Maunz, A., Willis, J., Haskova, Z., & Prunotto, M. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0172-3 · Cited by 305 · see citing works
Use of machine learning to analyse routinely collected intensive care unit data: a systematic review
Shillan, D., Sterne, J. A. C., Champneys, A., & Gibbison, B. (2019) · Critical Care · doi:10.1186/s13054-019-2564-9 · Cited by 251 · see citing works
Deep learning and alternative learning strategies for retrospective real-world clinical data
Chen, D. J., Liu, S., Kingsbury, P., Sohn, S., Storlie, C. B., Habermann, E. B., Naessens, J. M., Larson, D. W., & Liu, H. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0122-0 · Cited by 223 · see citing works
Deep learning for segmentation of brain tumors: Impact of cross‐institutional training and testing
AlBadawy, E. A., Saha, A., & Mazurowski, M. A. (2018) · Medical Physics · doi:10.1002/mp.12752 · Cited by 208 · see citing works
Mårtensson, G., Ferreira, D., Granberg, T., Cavallin, L., Oppedal, K., Padovani, A., Rektorova, I., Bonanni, L., Pardini, M., Kramberger, M. G., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101714 · Cited by 176 · see citing works
Mortality Trends of Acute Respiratory Distress Syndrome in the United States from 1999 to 2013
Cochi, S. E., Kempker, J. A., Annangi, S., Kramer, M. R., & Martin, G. S. (2016) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201512-841oc · Cited by 151 · cited 2× · see citing works
“Second, we investigated the applicability of the developed pipeline using the example of acute respiratory distress syndrome (ARDS)—a potentially life-threatening condition leading to respiratory insufficiency with possible multi-organ failure and fatal outcomes (Cochi et al., 2016; Raymondos et al., 2017).”
MethodPooch, E. H. P., Ballester, P., & Barros, R. C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-62469-9_7 · Cited by 134 · see citing works
A Convex Hull-Based Estimator of Home-Range Size
Worton, B. J. (1995) · Biometrics · doi:10.2307/2533254 · Cited by 115 · see citing works
Kelliny, C., William, J., Riesen, W., Paccaud, F. M., & Bovet, P. (2008) · Cardiovascular Diabetology · doi:10.1186/1475-2840-7-27 · Cited by 113 · see citing works
Learning in High Dimension Always Amounts to Extrapolation
Balestriero, R., Pesenti, J., & LeCun, Y. (2021) · arXiv · doi:10.48550/arxiv.2110.09485 · Cited by 93 · see citing works
Reynolds, H. N., McCunn, M., Borg, U., Habashi, N. M., Cottingham, C. A., & Bar‐Lavie, Y. (1998) · Critical Care · doi:10.1186/cc121 · Cited by 83 · see citing works
Barish, M., Bolourani, S., Lau, L. F., Shah, S., & Zanos, T. P. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00254-2 · Cited by 58 · see citing works
Three algorithms for estimating the domain of validity of feedforward neural networks
Courrieu, P. (1994) · Neural Networks · doi:10.1016/0893-6080(94)90065-5 · Cited by 57 · see citing works
Approximation and Estimation for High-Dimensional Deep Learning NetworksPreprint
Barron, A. R., & Klusowski, J. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03090 · Cited by 39 · see citing works
Outcome of acute respiratory distress syndrome in university and non-university hospitals in Germany
Raymondos, K., Dirks, T., Quintel, M., Molitoris, U., Ahrens, J., Dieck, T., Johanning, K., Henzler, D., Rossaint, R., Putensen, C., et al. (2017) · Critical Care · doi:10.1186/s13054-017-1687-0 · Cited by 38 · see citing works
Nearest Neighbor Convex Hull Classification Method for Face Recognition
Zhou, X., & Shi, Y. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-01973-9_64 · Cited by 26 · see citing works
Replication of a mortality prediction model in Dutch patients with COVID-19
Quanjel, M. J. R., van Holten, T. C., der Vliet, P. C. G.-V., Wielaard, J., Karakaya, B., Söhne, M., Moeniralam, H. S., & Grutters, J. C. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00253-3 · Cited by 25 · see citing works
On FastMap and the convex hull of multivariate data: toward fast and robust dimension reduction
George Ostrouchov, A., & Samatova, N. F. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.164 · Cited by 22 · see citing works
Limited applicability of a COVID-19 specific mortality prediction rule to the intensive care setting
Dupuis, C., de Montmollin, É., Neuville, M., Mourvillier, B., Ruckly, S., & Timsit, J. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00252-4 · Cited by 18 · see citing works
Schätzle, L.-K., Esfahani, A. H., & Schuppert, A. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007803 · Cited by 16 · see citing works
Self-citationMarx, G., Bickenbach, J., Fritsch, S. J., Kunze, J. B., Maaßen, O., Deffge, S., Kistermann, J., Haferkamp, S., Lutz, I., Voellm, N. K., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-045589 · Cited by 15 · see citing works
Self-citationGonçalves, J. M., Li, Y., Zhang, H., Xiao, Y., Wang, M., Guo, Y., Sun, C., Tang, X., Cao, Z., Li, S., et al. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00251-5 · Cited by 9 · see citing works
Kunze, J., Fritsch, S. J., Peine, A., Maaßen, O., Marx, G., & Bickenbach, J. (2020) · Trends in Anaesthesia and Critical Care · doi:10.1016/j.tacc.2020.05.005 · Cited by 7 · see citing works
Self-citationAn improved method for quality control of in situ data from Argo floats using α convex hulls
Shesu, R. V., Bhaskar, T. V. S. U., Rao, E. P. R., Ravichandran, M., & Rao, B. V. (2021) · MethodsX · doi:10.1016/j.mex.2021.101337 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Sharafutdinov, K., Fritsch, S. J., Iravani, M., Ghalati, P. F., Saffaran, S., Bates, D. G., Hardman, J. G., Polzin, R., Mayer, H., Marx, G., et al. (2023) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2023.3243190 · Cited by 15 · see citing works
Self-citationSamadi, M. E., Guzman-Maldonado, J., Nikulina, K., Mirzaieazar, H., Sharafutdinov, K., Fritsch, S. J., & Schuppert, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-55577-6 · Cited by 11 · see citing works
Self-citationCross-Domain Feature learning and data augmentation for few-shot proxy development in oil industry
Cirac, G., Farfan, J., Avansi, G. D., Schiozer, D. J., & Rocha, A. D. R. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110972 · Cited by 10 · see citing works
Barakat, C., Sharafutdinov, K., Busch, J., Saffaran, S., Bates, D. G., Hardman, J. G., Schuppert, A., Brynjólfsson, S., Fritsch, S. J., & Riedel, M. (2023) · Diagnostics · doi:10.3390/diagnostics13122098 · Cited by 7 · see citing works
Self-citationBarakat, C., Aach, M., Schuppert, A., Brynjólfsson, S., Fritsch, S. J., & Riedel, M. (2023) · Diagnostics · doi:10.3390/diagnostics13030391 · Cited by 4 · see citing works
Self-citationSharafutdinov, K., Fritsch, S. J., Iravani, M., Ghalati, P. F., Saffaran, S., Bates, D. G., Hardman, J. G., Polzin, R., Mayer, H., Marx, G., et al. (2022) · medRxiv · doi:10.1101/2022.12.02.22283033 · Cited by 4 · see citing works
Self-citationDistribution Analysis for Diagnostics and Therapeutics of Motor Actions
Srivatsa, A. V., Aghamohammadi, N. R., Ryali, P., Scarpellini, A., Kim, J. J. H., Aggarwal, A., Wright, Z. A., Huang, F. C., Reinkensmeyer, D. J., Esmailbeigi, H., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3636009 · Cited by 2 · see citing works
Privacy-preserving federated prediction of health outcomes using multi-center survey data
Das, S., Rafiei, M., Kammer, P., Skou, S. T., Grønne, D. T., Roos, E. M., Hajek, A., König, H. H., Ullah, M. S., Probul, N., et al. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02785-5 · Cited by 1 · see citing works
Bohlen, L., Rosenberger, J., Hambauer, N., Zähringer, D., Franz, V., Zschech, P., & Kraus, M. (2026) · Health Care Management Science · doi:10.1007/s10729-026-09760-y
“%) between the two datasets, a pattern that generally sug - gests strong potential for cross-hospital generalizability [72].”
Role of Vitamin D supplementation in rheumatoid arthritis patients: A review
Kumar, D. D. M., Senthil, V., & Khanna, M. (2024) · International Journal of Orthopaedics Sciences · doi:10.22271/ortho.2024.v10.i2a.3528
- Authors
- Sharafutdinov, K., Bhat, J. S., Fritsch, S. J., Nikulina, K., E. Samadi, M., Polzin, R., Mayer, H., Marx, G., Bickenbach, J., & Schuppert, A.
- Affiliations
- RWTH Aachen University (DE)
- Forschungszentrum Jülich (DE)
- Bayer (Germany) (DE)
- Universitätsklinikum Aachen (DE)
- Corresponding author
- Sharafutdinov, Konstantin
- ORCID
- Venue
- Frontiers in Big Data
- Publisher
- Frontiers Media SA
- Published
- 2022-10-31
- Volume
- 5
- Pages
- 603429-603429
- ISSN
- 2624-909X (ISSN-L)
- PMC
- PMC9659720
- OpenAlex
- W4307870422
- Cited by
- 10
- References
- 44
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 2 · 2023: 4 · 2022: 1
- FWCI
- 0.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Big Data (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
- RWTH Publications (RWTH Aachen)
- Funders
- Bundesministerium für Bildung und Forschung (01ZZ1803M, 01ZZ1803B)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- convex hull analysis, data heterogeneity, model transferability, generalization error, machine learning in healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
