Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
Open accessThe DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Abstract
Electrocardiography (ECG) is a very common, non-invasive diagnostic procedure and its interpretation is increasingly supported by algorithms. The progress in the field of automatic ECG analysis has up to now been hampered by a lack of appropriate datasets for training as well as a lack of well-defined evaluation procedures to ensure comparability of different algorithms. To alleviate these issues, we put forward first benchmarking results for the recently published, freely accessible clinical 12-lead ECG dataset PTB-XL, covering a variety of tasks from different ECG statement prediction tasks to age and sex prediction. Among the investigated deep-learning-based timeseries classification algorithms, we find that convolutional neural networks, in particular resnet- and inception-based architectures, show the strongest performance across all tasks. We find consistent results on the ICBEB2018 challenge ECG dataset and discuss prospects of transfer learning using classifiers pretrained on PTB-XL. These benchmarking results are complemented by deeper insights into the classification algorithm in terms of hidden stratification, model uncertainty and an exploratory interpretability analysis, which provide connecting points for future research on the dataset. Our results emphasize the prospects of deep-learning-based algorithms in the field of ECG analysis, not only in terms of quantitative accuracy but also in terms of clinically equally important further quality metrics such as uncertainty quantification and interpretability. With this resource, we aim to establish the PTB-XL dataset as a resource for structured benchmarking of ECG analysis algorithms and encourage other researchers in the field to join these efforts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 1×
“1 within the framework of layer-wise relevance propagation [52].”
BackgroundSelf-citationUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“During finetuning a pretrained classifier for transfer learning from PTB-XL to ICBEB2018, we use gradual unfreezing and discriminative learning rates [53], [54] to avoid catastrophic forgetting i.e. overwriting information captured during the initial training phase on PTB-XL.”
Deep learning with convolutional neural networks for EEG decoding and visualization
Schirrmeister, R. T., Springenberg, J. T., Fiederer, L. D. J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., & Ball, T. (2017) · Human Brain Mapping · doi:10.1002/hbm.23730 · Cited by 3,751
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666 · cited 3×
“Relation between model uncertainty (standard deviation of ensemble predictions as in [56]) and diagnosis likelihood as quantified by the annotating cardiologist, see Section IV-C for details.”
MethodInfluentialDeep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342
A Review on Multi-Label Learning Algorithms
Zhang, M.-L., & Zhou, Z.-H. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2013.39 · Cited by 3,321
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019) · Nature Medicine · doi:10.1038/s41591-018-0268-3 · Cited by 3,148
Time series classification from scratch with deep neural networks: A strong baseline
Wang, Z., Yan, W., & Oates, T. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966039 · Cited by 2,299 · cited 3×
“…neural networks : – standard : fully convolutional [25], Deep4Net [26] – resnet-based : one-dimensional adaptations of standard resnets [25], [27], wide resnets [28] and xresnets [29] – inception-based : InceptionTime [30] • recurrent neural networks : LSTM [31], GRU [32] • baseline…”
BackgroundResultInfluentialFixing Weight Decay Regularization in Adam
Loshchilov, I., & Hutter, F. (2018) · Cited by 1,986
Inceptiontime: finding alexnet for time series classification
Ismail Fawaz, H., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D. F., Weber, J., Webb, G. I., Idoumghar, L., Muller, P.-A., & Petitjean, F. (2020) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00710-y · Cited by 1,788
Bag of Tricks for Image Classification with Convolutional Neural Networks
He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., & Li, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00065 · Cited by 1,680 · cited 1×
“…fully convolutional [25], Deep4Net [26] – resnet-based : one-dimensional adaptations of standard resnets [25], [27], wide resnets [28] and xresnets [29] – inception-based : InceptionTime [30] • recurrent neural networks : LSTM [31], GRU [32] • baseline classifiers : – feature-based: Wavelet +…”
BackgroundAttia, Z. I., Noseworthy, P. A., Lopez-Jimenez, F., Asirvatham, S. J., Deshmukh, A. J., Gersh, B. J., Carter, R. E., Yao, X., Rabinstein, A. A., Erickson, B. J., et al. (2019) · The Lancet · doi:10.1016/s0140-6736(19)31721-0 · Cited by 1,613
Deep learning-based electroencephalography analysis: a systematic review
Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., & Faubert, J. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab260c · Cited by 1,532
Attia, Z. I., Kapa, S., Lopez-Jimenez, F., McKie, P. M., Ladewig, D. J., Satam, G., Pellikka, P. A., Enriquez-Sarano, M., Noseworthy, P. A., Munger, T. M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-018-0240-2 · Cited by 1,389
Bagnall, A., Lines, J., Bostrom, A., Large, J., & Keogh, E. (2017) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-016-0483-9 · Cited by 1,377
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6 · Cited by 1,377
PTB-XL, a large publicly available electrocardiography dataset
Wagner, P., Strodthoff, N., Bousseljot, R.-D., Kreiseler, D., Lunze, F. I., Samek, W., & Schaeffter, T. (2020) · Scientific Data · doi:10.1038/s41597-020-0495-6 · Cited by 1,315
Self-citationA survey of hierarchical classification across different application domains
Silla, C. N., & Freitas, A. A. (2011) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0175-9 · Cited by 1,077
Automatic diagnosis of the 12-lead ECG using a deep neural network
Ribeiro, A. H., Ribeiro, M. H., Paixão, G. M. M., Oliveira, D. M., Gomes, P. R., Canazart, J. A., Ferreira, M. P. S., Andersson, C. R., Macfarlane, P. W., Meira, W., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-15432-4 · Cited by 894 · cited 2×
“[25] (resnet1d_wang), and were successfully applied in different large-scale studies [10], [27].”
BackgroundMethodSmith, L. N. (2018) · arXiv · doi:10.48550/arxiv.1803.09820 · Cited by 790
PyWavelets: A Python package for wavelet analysis
Lee, G., Gommers, R., Waselewski, F., Wohlfahrt, K., & O'Leary, A. (2019) · Journal of Open Source Software · doi:10.21105/joss.01237 · Cited by 766
Ribeiro, A. H., Ribeiro, M. H., Paixao, G. M., Oliveira, D. M., Gomes, P. R., Canazart, J. A., Ferreira, M. P. S., Andersson, C. R., Macfarlane, P. W., Meira, W., et al. (2020) · RePEc: Research Papers in Economics · doi:10.5281/zenodo.3625006 · Cited by 715
Dagenais, G. R., Leong, D. P., Rangarajan, S., Lanas, F., Lopez-Jaramillo, P., Gupta, R., Diaz, R., Avezum, A., Oliveira, G. B. F., Wielgosz, A., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(19)32007-0 · Cited by 651
Nutzung der EKG-Signaldatenbank CARDIODAT der PTB über das Internet
Bousseljot, R., Kreiseler, D., & Schnabel, A. (2009) · Biomedizinische Technik/Biomedical Engineering · doi:10.1515/bmte.1995.40.s1.317 · Cited by 623
Liu, F., Liu, C., Zhao, L., Zhang, X., Wu, X., Xu, X., Liu, Y., Ma, C., Wei, S., He, Z., et al. (2018) · Journal of Medical Imaging and Health Informatics · doi:10.1166/jmihi.2018.2442 · Cited by 620
Hong, S., Zhou, Y., Shang, J., Xiao, C., & Sun, J. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103801 · Cited by 464
Multi-Scale Convolutional Neural Networks for Time Series Classification
Cui, Z., Chen, W., & Chen, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06995 · Cited by 455
Age and Sex Estimation Using Artificial Intelligence From Standard 12-Lead ECGs
Attia, Z. I., Friedman, P. A., Noseworthy, P. A., Lopez-Jimenez, F., Ladewig, D. J., Satam, G., Pellikka, P. A., Munger, T. M., Asirvatham, S. J., Scott, C. G., et al. (2019) · Circulation: Arrhythmia and Electrophysiology · doi:10.1161/circep.119.007284 · Cited by 455
Zhou, N., Jiang, Y., Bergquist, T. R., Lee, A. J., Kacsoh, B. Z., Crocker, A. W., Lewis, K. A., Georghiou, G., Nguyen, H. N., Hamid, M. N., et al. (2019) · Genome Biology · doi:10.1186/s13059-019-1835-8 · Cited by 452 · cited 1×
“For the selection of sample-centric metrics, we follow the evaluation procedures over the course of todate three CAFA classification challenges [37].”
MethodHidden stratification causes clinically meaningful failures in machine learning for medical imaging
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384468 · Cited by 411 · cited 3×
“Although identifying hidden stratification is straightforward to identify in hindsight given the hierarchical organization of the diagnostic labels, [55] also demonstrated how to identify groups of samples exhibiting hidden stratification for a given class label under consideration using an unsupervised clustering app…”
BackgroundInfluentialAn expanded evaluation of protein function prediction methods shows an improvement in accuracy
Jiang, Y., Oron, T. R., Clark, W. T., Bankapur, A. R., D’Andrea, D., Lepore, R., Funk, C. S., Kahanda, I., Verspoor, K. M., Ben-Hur, A., et al. (2016) · Genome Biology · doi:10.1186/s13059-016-1037-6 · Cited by 396
A brief note on overlapping confidence intervals
Austin, P. C., & Hux, J. E. (2002) · Journal of Vascular Surgery · doi:10.1067/mva.2002.125015 · Cited by 386
Computer-Interpreted Electrocardiograms
Schläpfer, J., & Wellens, H. J. (2017) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2017.07.723 · Cited by 377
Measuring psychological uncertainty: Verbal versus numeric methods.
Windschitl, P. D., & Wells, G. L. (1996) · Journal of Experimental Psychology: Applied · doi:10.1037/1076-898x.2.4.343 · Cited by 326 · cited 1×
“One possible explanation for this observation is an overconfidence of human annotators when it comes to seemingly very obvious statements that goes in with the human inability to precisely quantify uncertainties, which is a well-known phenomenon in cognitive psychology, see e.g. [47].”
BackgroundSalerno, S. M., Alguire, P. C., & Waxman, H. S. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-9-200305060-00013 · Cited by 261
Maglaveras, N., Stamkopoulos, T., Diamantaras, K., Pappas, C., & Strintzis, M. (1998) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(98)00138-5 · Cited by 235
A Unified View of Multi-Label Performance MeasuresPreprint
Wu, X.-Z., & Zhou, Z. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.00288 · Cited by 205
Hierarchical Multi-Label Classification Networks
Wehrmann, J., Barros, R. C., & Cerri, R. (2018) · International Conference on Machine Learning · Cited by 195
Detecting and interpreting myocardial infarction using fully convolutional neural networks
Strodthoff, N., & Strodthoff, C. (2019) · Physiological Measurement · doi:10.1088/1361-6579/aaf34d · Cited by 172
Self-citationDeep Learning for Time-Series Analysis
Gamboa, J. C. B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.01887 · Cited by 157
Sharma, L. D., & Sunkaria, R. K. (2017) · Signal Image and Video Processing · doi:10.1007/s11760-017-1146-z · Cited by 148
Sex differences in the mechanisms underlying long QT syndrome
Salama, G., & Bett, G. C. L. (2014) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00864.2013 · Cited by 131
ECG signals classification: a review
Houssein, E. H., Kilany, M., & Hassanien, A. E. (2017) · International Journal of Intelligent Engineering Informatics · doi:10.1504/ijiei.2017.087944 · Cited by 111
Teaching the interpretation of electrocardiograms: Which method is best?
Fent, G. J., Gosai, J. N., & Purva, M. (2014) · Journal of Electrocardiology · doi:10.1016/j.jelectrocard.2014.12.014 · Cited by 107
Smith, S. W., Walsh, B., Grauer, K., Wang, K., Rapin, J., Li, J., Fennell, W., & Taboulet, P. (2019) · Journal of Electrocardiology · doi:10.1016/j.jelectrocard.2018.11.013 · Cited by 106
Novel Bloodless Potassium Determination Using a Signal‐Processed Single‐Lead ECG
Attia, Z. I., DeSimone, C. V., Dillon, J. J., Sapir, Y., Somers, V. K., Dugan, J. L., Bruce, C. J., Ackerman, M. J., Asirvatham, S. J., Striemer, B. L., et al. (2016) · Journal of the American Heart Association · doi:10.1161/jaha.115.002746 · Cited by 86
QT/RR curvatures in healthy subjects: sex differences and covariates
Malík, M., Hnatkova, K., Kowalski, D. L., Keirns, J. J., & van Gelderen, E. M. (2013) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00577.2013 · Cited by 76
Interpreting Deep Neural Networks for Single-Lead ECG Arrhythmia Classification
Vijayarangan, S., Murugesan, B., Vignesh, R., Preejith, S. P., Joseph, J., & Sivaprakasam, M. (2020) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc44109.2020.9176396 · Cited by 65
Predicting “Heart Age” Using Electrocardiography
Ball, R. L., Feiveson, A. H., Schlegel, T. T., Starc, V., & Dabney, A. R. (2014) · Journal of Personalized Medicine · doi:10.3390/jpm4010065 · Cited by 55
Techniques for visualizing LSTMs applied to electrocardiogramsPreprint
van der Westhuizen, J., & Lasenby, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08153 · Cited by 18
Cited by
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
Deep Neural Network-Estimated Electrocardiographic Age as a Mortality Predictor
Lima, E. M., Ribeiro, A. H., Paixão, G. M. M., Ribeiro, M. H., Pinto-Filho, M. M., Gomes, P. R., Oliveira, D. M., Sabino, E. C., Duncan, B. B., Giatti, L., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-25351-7 · Cited by 249
Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discovery
Wagner, P., Mehari, T., Haverkamp, W., & Strodthoff, N. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108525 · Cited by 48
“We consider both a shallow model inspired by the LeNet architecture [56] and a ResNet-based [57] architecture, which was shown to lead to competitive results [5] for ECG classification tasks on PTBXL.”
BackgroundMethodInfluentialSelf-citationExplainable AI for time series via Virtual Inspection Layers
Vielhaben, J., Lapuschkin, S., Montavon, G., & Samek, W. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110309 · Cited by 38
Self-citationMehari, T., & Strodthoff, N. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3310989 · Cited by 29
“In the case of PTB-XL, we followed a previously established benchmarking methodology[6], where the first eight label-balanced folds of the dataset are used for training, the ninth for validation and the tenth for testing in a multi-label classification task.”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Strodthoff, N., Wagner, P., Schaeffter, T., & Samek, W.
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2021-05
- Volume
- 25
- Issue
- 5
- Pages
- 1519-1528
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W3021394814
- Cited by
- 491
- References
- 57
- FWCI
- 25.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.