Recovering True Classifier Performance in Positive-Unlabeled Learning
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
A common approach in positive-unlabeled learning is to train a classification model between labeled and unlabeled data. This strategy is in fact known to give an optimal classifier under mild conditions; however, it results in biased empirical estimates of the classifier performance. In this work, we show that the typically used performance measures such as the receiver operating characteristic curve, or the precision recall curve obtained on such data can be corrected with the knowledge of class priors; i.e., the proportions of the positive and negative examples in the unlabeled data. We extend the results to a noisy setting where some of the examples labeled positive are in fact negative and show that the correction also requires the knowledge of the proportion of noisy examples in the labeled positives. Using state-of-the-art algorithms to estimate the positive class prior and the proportion of noise, we experimentally evaluate two correction approaches and demonstrate their efficacy on real-life data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · cited 1×
“Furthermore, the area under the ROC curve (AUC) has a meaningful probabilistic interpretation that correlates with the ability of the classifier to separate classes and is often used to rank classifiers (Hanley and McNeil 1982).”
BackgroundA direct adaptive method for faster backpropagation learning: the RPROP algorithm
Riedmiller, M., & Braun, H. (2002) · IEEE International Conference on Neural Networks · doi:10.1109/icnn.1993.298623 · Cited by 3,917 · cited 2×
“Each network had five hidden neurons and was trained using resilient propagation (Riedmiller and Braun 1993).”
MethodAdjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure
Saerens, M., Latinne, P., & Decaestecker, C. (2002) · Neural Computation · doi:10.1162/089976602753284446 · Cited by 361 · cited 1×
“…Mitchell 1998) or unbias the labeled data (Cortes et al. 2008); e.g., to estimate class proportions that are necessary to calibrate the model and accurately estimate precision when class balances (but not class-conditional distributions) in labeled data are not representative (Saerens et al. 2002).”
Background
Cited by
Inferring the molecular and phenotypic impact of amino acid variants with MutPred2
Pejaver, V., Urresti, J., Lugo-Martinez, J., Pagel, K. A., Lin, G. N., Nam, H.-J., Mort, M., Cooper, D. N., Sebat, J., Iakoucheva, L. M., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-19669-x · Cited by 923
Learning from positive and unlabeled data: a survey
Bekker, J., & Davis, J. (2020) · Machine Learning · doi:10.1007/s10994-020-05877-5 · Cited by 651
“, [15,41]) and it will be important to design metrics that work for other labeling mechanisms.”
BackgroundPredicting the clinical impact of human mutation with deep neural networks
Sundaram, L., Gao, H., Padigepati, S. R., McRae, J. F., Li, Y., Kosmicki, J. A., Fritzilas, N., Hakenberg, J., Dutta, A., Shon, J., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0167-z · Cited by 493
Fast nonparametric estimation of class proportions in the positive-unlabeled classification setting
Zeiberg, D., Jain, S., & Radivojac, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6151 · Cited by 29
“It has been recently shown that many true performance measures can be recovered if one has well-estimated class priors (Menon et al. 2015; Jain, White, and Radivojac 2017; Ramola, Jain, and Radivojac 2019).”
BackgroundSelf-citation
Metadata
- Authors
- Jain, S., White, M., & Radivojac, P.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2017-02-13
- Volume
- 31
- Issue
- 1
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2587621812
- Cited by
- 47
- References
- 3
- FWCI
- 2.79
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Recovering True Classifier Performance in Positive-Unlabeled Learning. https://app.cochat.ai/papers/10.1609/aaai.v31i1.10937
- The paper
- Jain, S., White, M., & Radivojac, P. (2017). Recovering True Classifier Performance in Positive-Unlabeled Learning. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v31i1.10937
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.