On modelling label uncertainty in deep neural networks: Automatic estimation of intra-observer variability in 2d echocardiography quality assessment
Open accessThe DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Abstract
Uncertainty of labels in clinical data resulting from intra-observer variability can have direct impact on the reliability of assessments made by deep neural networks. In this paper, we propose a method for modelling such uncertainty in the context of 2D echocardiography (echo), which is a routine procedure for detecting cardiovascular disease at point-of-care. Echo imaging quality and acquisition time is highly dependent on the operator's experience level. Recent developments have shown the possibility of automating echo image quality quantification by mapping an expert's assessment of quality to the echo image via deep learning techniques. Nevertheless, the observer variability in the expert's assessment can impact the quality quantification accuracy. Here, we aim to model the intra-observer variability in echo quality assessment as an aleatoric uncertainty modelling regression problem with the introduction of a novel method that handles the regression problem with categorical labels. A key feature of our design is that only a single forward pass is sufficient to estimate the level of uncertainty for the network output. Compared to the 0.11 ± 0.09 absolute error (in a scale from 0 to 1) archived by the conventional regression method, the proposed method brings the error down to 0.09 ± 0.08, where the improvement is statistically significant and equivalents to 5.7% test accuracy improvement. The simplicity of the proposed approach means that it could be generalized to other applications of deep learning in medical imaging, where there is often uncertainty in clinical labels.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Raghu al. [688] | et | 2019 | N/A | Classification | DR | 1 | DUP (Direct Uncertainty Pre- diction )”
From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [688]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 6×
“In parallel to the Mean and STD modules, the LSTM features are also fed to compute an additional value lambda (softmaxed over the 10 steps) to be the weighting parameter for each component.”
MethodInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“The Adam [52] optimizer is 1The symbols defined in Fig.”
MethodInfluentialDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“, 3 × 3 filter size with stride 1, no padding, batchnormalized [46], and use ReLU [47] activation function.”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 2×
“The soft targets methods [41], [42] are used as regularization techniques for training a small model with the predictions inferred from a high complexity model or an ensemble of them (trained from the same dataset), for the purposes of capturing the knowledge of the high complexity models with fewer model parameters (…”
BackgroundMethod(2010) · International Conference on Machine Learning · Cited by 13,995
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 1×
“performance variations may be subtle when the deployed number of units/parameters are similar [49]–[51].”
BackgroundDropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · cited 2×
“is one of the commonly used Bayesian inference approximation method [36], which has been used in medical imaging to derive robust solutions for disease detection, lesion/organ segmentation [37]–[39].”
MethodBayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276 · cited 2×
“Several Bayesian inference approaches and more recent Bayesian neural networks (BNN) [25], [26], [27], [28] are designed to address the uncertainty in the induced classifier by imposing a prior distribution over model parameters.”
MethodOn calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435 · cited 1×
“7, statistically speaking the number of correct predictions should be close to 70% [53].”
BackgroundWhat Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088 · cited 2×
“, learning the variability distribution as an output of a model [23], [24].”
BackgroundA practical Bayesian framework for backpropagation networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.448 · Cited by 2,917 · cited 2×
“Several Bayesian inference approaches and more recent Bayesian neural networks (BNN) [25], [26], [27], [28] are designed to address the uncertainty in the induced classifier by imposing a prior distribution over model parameters.”
MethodZadeh, L. A. (1997) · Fuzzy Sets and Systems · doi:10.1016/s0165-0114(97)00077-8 · Cited by 2,897 · cited 1×
“The diversity in experts’ opinions follows the fuzziness in human reasoning [31].”
BackgroundFitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859 · cited 1×
“The soft targets methods [41], [42] are used as regularization techniques for training a small model with the predictions inferred from a high complexity model or an ensemble of them (trained from the same dataset), for the purposes of capturing the knowledge of the high complexity models with fewer model parameters (…”
MethodAleatory or epistemic? Does it matter?
Kiureghian, A. D., & Ditlevsen, O. (2009) · Structural Safety · doi:10.1016/j.strusafe.2008.06.020 · Cited by 2,454 · cited 1×
“, namely the aleatoric uncertainty, where the aim is to capture the noise inherent in the observations through a Bayesian approach, i.e., learning the variability distribution as an output of a model [23], [24]. In a clinical labelling context, this assumes the observer’s gold standard opinion, regarding a single samp…”
BackgroundClassification in the Presence of Label Noise: A Survey
Frenay, B., & Verleysen, M. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2292894 · Cited by 1,777 · cited 1×
“In the machine learning field, one way to treat observer variability is to consider it as label noise problem [13].”
BackgroundA sequential algorithm for training text classifiers
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,731
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342 · cited 1×
“Since the marginal distribution p(Y|X) is intractable, various methods approximate p(W|D) with a simple distribution qθ(W), parameterized by θ [33], [34], [35], [36].”
MethodWeight Uncertainty in Neural Network
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · International Conference on Machine Learning · Cited by 1,335
A Sequential Algorithm for Training Text Classifiers
Lewis, D. D., & Gale, W. A. (1994) · SIGIR ’94 · doi:10.1007/978-1-4471-2099-5_1 · Cited by 1,208
Heterogeneous Uncertainty Sampling for Supervised Learning
Lewis, D. D., & Catlett, J. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50026-x · Cited by 1,150 · cited 2×
“An uncertainty sampling learner model [21] treating low confidence of the prediction, i.”
BackgroundPractical Variational Inference for Neural Networks
Graves, A. (2011) · Cited by 1,078
Convexity, Classification, and Risk Bounds
Bartlett, P. L., Jordan, M. I., & McAuliffe, J. D. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000000907 · Cited by 1,037 · cited 1×
“The noise-robust and noise-tolerant loss functions and models [16], [17] commonly work well in low”
BackgroundStatistical methods for assessing observer variability in clinical measures.
Brennan, P., & Silman, A. (1992) · BMJ · doi:10.1136/bmj.304.6840.1491 · Cited by 985 · cited 1×
“In clinical studies [11], [12], the lack of consistency in diagnostic judgment and decision making has been long observed.”
BackgroundComparative Study of CNN and RNN for Natural Language Processing
Yin, W., Kann, K., Yu, M., & Schütze, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01923 · Cited by 861 · cited 1×
“Nevertheless, it should be noted that the performance variations may be subtle when the deployed number of units/parameters are similar [49], [50], [51].”
BackgroundIdentifying Mislabeled Training Data
Brodley, C. E., & Friedl, M. A. (1999) · Cited by 718
Estimating the mean and variance of the target probability distribution
Nix, D. A., & Weigend, A. S. (1994) · Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) · doi:10.1109/icnn.1994.374138 · Cited by 700 · cited 7×
“(1)), where the training relies on the regression label y [29].”
BackgroundMethodInfluentialGeng, X. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2545658 · Cited by 691
Identifying Mislabeled Training Data
Brodley, C. E., & Friedl, M. A. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.606 · Cited by 612 · cited 1×
“noise samples before training a classifier [14], [15], but hard (informative but difficult for a model to predict the correct”
BackgroundBayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
Gal, Y., & Ghahramani, Z. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02158 · Cited by 563 · cited 1×
“Since the marginal distribution p(Y|X) is intractable, various methods approximate p(W|D) with a simple distribution qθ(W), parameterized by θ [33], [34], [35], [36].”
MethodLeveraging uncertainty information from deep neural networks for disease detection
Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017) · Scientific Reports · doi:10.1038/s41598-017-17876-z · Cited by 510 · cited 1×
“is one of the commonly used Bayesian inference approximation method [36], which has been used in medical imaging to derive robust solutions for disease detection, lesion/organ segmentation [37]–[39].”
MethodDeep Label Distribution Learning With Label Ambiguity
Gao, B.-B., Xing, C., Xie, C.-W., Wu, J., & Geng, X. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2689998 · Cited by 508 · cited 1×
“It should be pointed out that the soft targets method resembles the label distribution learning (LDL) methods [43], [44], where the essence of LDL methods is to to match a likelihood distribution estimation (by a model) to the ground truth label distribution by using KL divergence.”
MethodFUIQA: Fetal Ultrasound Image Quality Assessment with Deep Convolutional Networks
Wu, L., Cheng, J.-Z., Li, S., Lei, B., Wang, T., & Ni, D. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2671898 · Cited by 242 · cited 1×
“[8] proposed a convolution neural networks (CNN) for fetal US quality assessment scheme in order to control US image quality.”
BackgroundTransforming Neural-Net Output Levels to Probability Distributions
Denker, J. S., & LeCun, Y. (1990) · Cited by 241
Semi-supervised SVM batch mode active learning for image retrieval
Hoi, S. C. H., Jin, R., Zhu, J., & Lyu, M. R. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587350 · Cited by 160 · cited 1×
“Active learning methods [18]–[20] coupled with uncertainty”
Roy, A. G., Conjeti, S., Navab, N., & Wachinger, C. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.03.042 · Cited by 137
Abdi, A. H., Luong, C., Tsang, T., Allan, G., Nouranian, S., Jue, J., Hawley, D., Fleming, S., Gin, K., Swift, J., et al. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2690836 · Cited by 137 · cited 1×
“showed that deep models can leverage CNN and recurrent neural networks (RNN) to automate the echo quality assessment of images and cine series by learning from experts’ labelled samples [9], [10].”
BackgroundSelf-citationAutomated segmentation of pulmonary structures in thoracic computed tomography scans: a review
van Rikxoort, E. M., & van Ginneken, B. (2013) · Physics in Medicine and Biology · doi:10.1088/0031-9155/58/17/r187 · Cited by 127 · cited 1×
“In clinical studies [11], [12], the lack of consistency in diagnostic judgment and decision making has been long observed.”
BackgroundLoizou, C. P., Pattichis, C. S., Pantziaris, M., Tyllis, T., & Nicolaides, A. (2006) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-006-0045-1 · Cited by 111 · cited 1×
“Beyond conventional methods in estimating ultrasound image quality [3], [4], [5], [6], [7], most recent methods are based on deep learning to assess image quality.”
MethodEhler, D., CARNEY, D., Dempsey, A. L., Rigling, R., Kraft, C. D., Witt, S. A., Kimball, T. R., Sisk, E. J., Geiser, E. A., & Gresser, C. D. (2001) · Journal of the American Society of Echocardiography · doi:10.1067/mje.2001.109922 · Cited by 103 · cited 1×
“Accurate acquisition of echo images requires many years of practice to gain adequate anatomical knowledge and hand-eye coordination to adjust imaging to individual patients [1].”
Use of Classification Algorithms in Noise Detection and Elimination
Miranda, A., Garcia, L. P. F., de Carvalho, A. C. P. L. F., & Lorena, A. C. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02319-4_50 · Cited by 99 · cited 1×
“noise samples before training a classifier [14], [15], but hard (informative but difficult for a model to predict the correct”
BackgroundDezaki, F. T., Liao, Z., Luong, C. L., Girgis, H., Dhungel, N., Abdi, A. H., Behnami, D., Gin, K., Rohling, R. N., Abolmaesumi, P., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2888807 · Cited by 95 · cited 1×
“performance variations may be subtle when the deployed number of units/parameters are similar [49]–[51].”
BackgroundSelf-citationNair, T., Precup, D., Arnold, D. L., & Arbel, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00928-1_74 · Cited by 92
Learning with annotation noise
Beigman, E., & Klebanov, B. B. (2009) · Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - ACL-IJCNLP '09 · doi:10.3115/1687878.1687919 · Cited by 86 · cited 1×
“The noise-robust and noise-tolerant loss functions and models [16], [17] commonly work well in low”
BackgroundA machine-learning framework for automatic reference-free quality assessment in MRI
Küstner, T., Gatidis, S., Liebgott, A., Schwartz, M., Mauch, L., Martirosian, P., Schmidt, H. R., Schwenzer, N., Nikolaou, K., Bamberg, F., et al. (2018) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2018.07.003 · Cited by 64 · cited 1×
“Active learning methods [18]–[20] coupled with uncertainty”
Abdi, A. H., Luong, C. L., Tsang, T. S. M., Jue, J., Gin, K., Yeung, D. F., Hawley, D., Rohling, R. N., & Abolmaesumi, P. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_35 · Cited by 50 · cited 1×
“showed that deep models can leverage CNN and recurrent neural networks (RNN) to automate the echo quality assessment of images and cine series by learning from experts’ labelled samples [9], [10].”
BackgroundSelf-citationGaudet, J. E., Waechter, J. E., McLaughlin, K., Ferland, A., Godinez-Luna, T., Bands, C. J., Boucher, P. E., & Lockyer, J. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000001620 · Cited by 37 · cited 2×
“A) [30], quantified with an interval of 5%.”
BackgroundMethodConfidence-driven control of an ultrasound probe: Target-specific acoustic window optimization
Chatelain, P., Krupa, A., & Navab, N. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487522 · Cited by 35 · cited 1×
“image quality [3]–[7], most recent methods are based on deep learning to assess image quality.”
BackgroundMisclassified multinomial data: a Bayesian approach
Pérez, C. J., González-Torre, F. J. G., Martín, J., Ruiz, M. V., & Rojano, C. (2007) · Revista de la Real Academia de Ciencias Exactas Físicas y Naturales Serie A Matemáticas · Cited by 31
Tsantis, S., Spiliοpoulos, S., Skouroliakou, A., Karnabatidis, D., Hazle, J. D., & Kagadis, G. C. (2014) · Medical Physics · doi:10.1118/1.4883815 · Cited by 16 · cited 1×
“image quality [3]–[7], most recent methods are based on deep learning to assess image quality.”
BackgroundImage Quality Influences the Assessment of Left Ventricular Function
Großgasteiger, M., Hien, M. D., Graser, B., Rauch, H., Motsch, J., Gondan, M., & Rosendal, C. (2014) · Journal of Ultrasound in Medicine · doi:10.7863/ultra.33.2.297 · Cited by 16 · cited 1×
“This means poor quality echo images acquired by less experienced ultrasound operators can negatively affect diagnostic accuracy [2].”
BackgroundLearning the Manifold of Quality Ultrasound Acquisition
El-Zehiry, N. Y., Yan, M., Good, S., Fang, T., Zhou, S. K., & Grady, L. J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40811-3_16 · Cited by 12 · cited 1×
“Beyond conventional methods in estimating ultrasound image quality [3], [4], [5], [6], [7], most recent methods are based on deep learning to assess image quality.”
MethodActive learning for image quality assessment by model observer
Lorente, I., & Brankov, J. G. (2014) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2014.6868128 · Cited by 9 · cited 1×
“Active learning methods [18], [19], [20] coupled with uncertainty sampling strategy [21], [22] can detect hard informative samples and promote human observer to label only those samples.”
BackgroundDetection and display of acoustic window for guiding and training cardiac ultrasound users
Huang, S.-W., Radulescu, E. G., Wang, S., Thiele, K. E., Prater, D. M., Maxwell, D., Rafter, P. G., Dupuy, C., Drysdale, J., & Erkamp, R. (2014) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2044051 · Cited by 5 · cited 1×
“Beyond conventional methods in estimating ultrasound image quality [3], [4], [5], [6], [7], most recent methods are based on deep learning to assess image quality.”
Method
Cited by
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
Seoni, S., Jahmunah, V., Salvi, M., Barua, P. D., Molinari, F., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107441 · Cited by 239
Huang, L., Ruan, S., Xing, Y., & Feng, M. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103223 · Cited by 121
Towards Clinical Application of Artificial Intelligence in Ultrasound Imaging
Komatsu, M., Sakai, A., Dozen, A., Shozu, K., Yasutomi, S., Machino, H., Asada, K., Kaneko, S., & Hamamoto, R. (2021) · Biomedicines · doi:10.3390/biomedicines9070720 · Cited by 114
“proposed a quality assessment method for cardiac US images through modeling the label uncertainty in CNNs resulting from intra-observer variability in the labeling [100].”
MethodUlloa, A., Jing, L., Good, C. W., vanMaanen, D. P., Raghunath, S. M., Suever, J. D., Nevius, C. D., Wehner, G. J., Hartzel, D. N., Leader, J. B., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-020-00667-9 · Cited by 84
Metadata
- Authors
- Liao, Z., Girgis, H., Abdi, A., Vaseli, H., Hetherington, J., Rohling, R., Gin, K., Tsang, T., & Abolmaesumi, P.
- Affiliations
- University of British Columbia (CA)
- ORCID
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2020-06
- Volume
- 39
- Issue
- 6
- Pages
- 1868-1883
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2996089546
- Cited by
- 56
- References
- 56
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 6 · 2023: 11 · 2022: 6 · 2021: 10 · 2020: 4
- FWCI
- 1.59
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institutes of Health Research
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Keywords
- label uncertainty, intraobserver variability, two-dimensional echocardiography, aleatoric uncertainty, uncertainty estimation, categorical regression, single forward pass
- MeSH terms
- Echocardiography, Humans, Reproducibility of Results, Observer Variation, Neural Networks, Computer, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.