Evaluating Learning Algorithms: A Classification Perspective
Abstract
The field of machine learning has matured to the point where many sophisticated learning approaches can be applied to practical applications. Thus it is of critical importance that researchers have the proper tools to evaluate learning approaches and understand the underlying issues. This book examines various aspects of the evaluation process with an emphasis on classification algorithms. The authors describe several techniques for classifier performance assessment, error estimation and resampling, obtaining statistical significance as well as selecting appropriate domains for evaluation. They also present a unified evaluation framework and highlight how different components of evaluation are both significantly interrelated and interdependent. The techniques presented in the book are illustrated using R and WEKA, facilitating better practical insight as well as implementation. Aimed at researchers in the theory and applications of machine learning, this book offers a solid basis for conducting performance evaluations of algorithms in practical settings.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many epistemic aspects of algorithmic models are open to question in various contexts.
“First, many epistemic aspects of algorithmic models are open to question in various contexts.”
From Algorithmic Accountability and Public Reason (2018) · cites this paper as Japkowicz & Shah, 2011
Cites
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · see citing works
A Survey of Predictive Modeling on Imbalanced Domains
Branco, P., Torgo, L., & Ribeiro, R. P. (2017) · ACM Computing Surveys · doi:10.1145/2907070 · Cited by 1,129 · see citing works
Learning under Concept Drift: A Review
Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., & Zhang, G. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2876857 · Cited by 1,603 · see citing works
The use of machine learning algorithms in recommender systems: a systematic review
Portugal, I., Alencar, P., & Cowan, D. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.12.020 · Cited by 743 · see citing works
Veale, M., & Binns, R. (2017) · Big Data & Society · doi:10.1177/2053951717743530 · Cited by 512 · see citing works
Algorithmic Accountability and Public Reason
Binns, R. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0263-5 · Cited by 457 · see citing works
Artificial Intelligence and Patient-Centered Decision-Making
Bjerring, J. C., & Busch, J. (2021) · Philosophy & Technology · doi:10.1007/s13347-019-00391-6 · Cited by 274 · see citing works
Discrete Bayesian Network Classifiers: A Survey
Bielza, C., & Larrañaga, P. (2014) · ACM Computing Surveys · doi:10.1145/2576868 · Cited by 248 · see citing works
Clinical prediction from structural brain MRI scans: A large-scale empirical study
Initiative, F. T. A. D. N., Sabuncu, M. R., & Konukoglu, E. (2015) · Neuroinformatics · doi:10.1007/s12021-014-9238-1 · Cited by 161 · see citing works
A review of machine learning methods used for educational data
Ersozlu, Z., Taheri, S., & Koch, I. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12704-0 · Cited by 96 · see citing works
Tahmasian, M., Shao, J., Meng, C., Grimmer, T., Diehl-Schmid, J., Yousefi, B. H., Förster, S., Riedl, V., Drzezga, A., & Sorg, C. (2016) · Journal of Nuclear Medicine · doi:10.2967/jnumed.115.165464 · Cited by 66 · see citing works
de Andrade, F. H. C., Pereira, F. G., Resende, C. Z., & Cavalieri, D. C. (2019) · IFMBE Proceedings · doi:10.1007/978-981-13-2517-5_42 · Cited by 8 · see citing works
- Authors
- Japkowicz, N., & Shah, M.
- Affiliations
- American University (US)
- McGill University (CA)
- Venue
- Cambridge University Press eBooks
- Publisher
- Cambridge University Press
- Published
- 2011-01-17
- OpenAlex
- W4245055982
- Cited by
- 767
- References
- 1
- Citations by year
- 2026: 37 · 2025: 62 · 2024: 58 · 2023: 69 · 2022: 75 · 2021: 68 · 2020: 71 · 2019: 55 · 2018: 53 · 2017: 43 · 2016: 48 · 2015: 42 · 2014: 46 · 2013: 29 · 2012: 8
- FWCI
- 11.11
- Type
- book
- Language
- en
- Open access
- closed
- Locations
- Cambridge University Press eBooks (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Keywords
- error estimation, resampling, statistical significance testing, evaluation framework, machine learning classifiers, WEKA, R
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
