The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Confidence interval for micro-averaged F1 and macro-averaged F1 scores
Open accessAbstract
A binary classification problem is common in medical field, and we often use sensitivity, specificity, accuracy, negative and positive predictive values as measures of performance of a binary predictor. In computer science, a classifier is usually evaluated with precision (positive predictive value) and recall (sensitivity). As a single summary measure of a classifier’s performance, F1 score, defined as the harmonic mean of precision and recall, is widely used in the context of information retrieval and information extraction evaluation since it possesses favorable characteristics, especially when the prevalence is low. Some statistical methods for inference have been developed for the F1 score in binary classification problems; however, they have not been extended to the problem of multi-class classification. There are three types of F1 scores, and statistical properties of these F1 scores have hardly ever been discussed. We propose methods based on the large sample multivariate central limit theorem for estimating F1 scores with confidence intervals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515
Mixed Neural Network Approach for Temporal Sleep Stage Classification
Dong, H., Supratak, A., Pan, W., Wu, C., Matthews, P. M., & Guo, Y. (2018) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2017.2733220 · Cited by 295
Wang, S.-H., Nayak, D. R., Guttery, D. S., Zhang, X., & Zhang, Y.-D. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.11.005 · Cited by 239
Cyberattacks Detection in IoT-Based Smart City Applications Using Machine Learning Techniques
Rashid, M. M., Kamruzzaman, J., Hassan, M. M., Imam, T., & Gordon, S. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17249347 · Cited by 183
Estimating the Uncertainty of Average F1 Scores
Zhang, D., Wang, Y.-X., & Zhao, X. (2015) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/2808194.2809488 · Cited by 97
Information Retrieval: Advanced Topics and Techniques
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 88
Socoró, J. C., Álías, F., & Alsina‐Pagès, R. M. (2017) · Sensors · doi:10.3390/s17102323 · Cited by 83
Hong, N., Wen, A., Stone, D. J., Tsuji, S., Kingsbury, P., Rasmussen, L. V., Pacheco, J. A., Adekkanattu, P. M., Wang, F., Luo, Y., et al. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103310 · Cited by 79
Detection of driver manual distraction via image-based hand and ear recognition
Li, L., Zhong, B., Hutmacher, C. M., Liang, Y., Horrey, W. J., & Xu, X. (2020) · Accident Analysis & Prevention · doi:10.1016/j.aap.2020.105432 · Cited by 75
Chowdhury, S., Dong, X., Qian, L., Li, X., Guan, Y., Yang, J., & Yu, Q. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2467-9 · Cited by 66
Biomedical event trigger detection by dependency-based word embedding
Wang, J., Zhang, J., An, Y., Lin, H., Yang, Z., Zhang, Y., & Sun, Y. (2016) · BMC Medical Genomics · doi:10.1186/s12920-016-0203-8 · Cited by 29
Wang, Y., Li, J., Li, Y., Wang, R., & Yang, X. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2359667 · Cited by 26
Analyze Informant-Based Questionnaire for The Early Diagnosis of Senile Dementia Using Deep Learning
Zhu, F., Li, X., McGonigle, D., Tang, H., He, Z., Zhang, C., Hung, G., Chiu, P., & Zhou, W. (2019) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2019.2959331 · Cited by 23
An assessment of the portability of ancestry informative markers between human populations
Myles, S., Stoneking, M., & Timpson, N. J. (2009) · BMC Medical Genomics · doi:10.1186/1755-8794-2-45 · Cited by 8
Cited by
Evaluating the accuracy and reliability of AI content detectors in academic contexts
Hadra, M., Cambridge, K., & Mesbah, M. (2026) · International Journal for Educational Integrity · doi:10.1007/s40979-026-00213-1 · Cited by 27
Qorich, M., & El Ouazzani, R. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128405 · Cited by 6
Metadata
- Authors
- Takahashi, K., Yamamoto, K., Kuchiba, A., & Koyama, T.
- Venue
- Applied Intelligence
- Published
- 2021-07-31
- Volume
- 52
- Issue
- 5
- Pages
- 4961-4972
- ISSN
- 0924-669X (ISSN-L)
- 1573-7497
- OpenAlex
- W3187743275
- Cited by
- 297
- References
- 16
- FWCI
- 12.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.