Classifier Technology and the Illusion of Progress
Open accessAbstract
A great many tools have been developed for supervised classification, ranging from early methods such as linear discriminant analysis through to modern developments such as neural networks and support vector machines. A large number of comparative studies have been conducted in attempts to establish the relative superiority of these methods. This paper argues that these comparisons often fail to take into account important aspects of real problems, so that the apparent superiority of more sophisticated methods may be something of an illusion. In particular, simple methods typically yield performance almost as good as more sophisticated methods, to the extent that the difference in performance may be swamped by other sources of uncertainty that generally are not considered in the classical supervised classification paradigm.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The study performs an independent evaluation of machine learning methods, without proposing a new method that may bias an evaluation.
“In doing so we extend previous ML studies in financial time series forecasting (de Oliveira et al., 2013; Kara et al., 2011) by performing an independent evaluation of ML methods (i.e., we do not propose a new method, which may bias an evaluation (Hand, 2006)).”
From Bridging the divide in financial market forecasting: machine learners vs. financial economists (2016) · cites this paper as Hand, 2006
Simpler machine learning algorithms are less likely than more complex algorithms to be brittle in the face of changing distributions, so simpler methods can be preferred when data drift is expected.
“There are several responses to this: simpler machine learning algorithms are less likely than more complex algorithms to be 'brittle' in the face of changing distributions, so that simpler methods can be preferred when data drift is expected. 25”
From Machine learning for large‐scale wearable sensor data in Parkinson's disease: Concepts, promises, pitfalls, and futures (2016) · cites this paper as [25]
Almost all classification methods implicitly assume that the training set is a random sample from the population of documents to be coded.
“In fact, almost all classification methods implicitly assume that the training set is a random sample from the population of documents to be coded (Hand 2006).”
From Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts (2013) · cites this paper as Hand 2006
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS
Fisher, R. (1936) · Annals of Eugenics · doi:10.1111/j.1469-1809.1936.tb02137.x · Cited by 16,063 · see citing works
UCI Repository of machine learning databases
Blake, C. L. (1998) · Medical Entomology and Zoology · Cited by 10,491 · see citing works
Pattern Recognition and Neural Networks
Ripley, B. D. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812651 · Cited by 6,446 · see citing works
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · see citing works
Discriminant Analysis and Statistical Pattern Recognition.
Hawkins, D. M., & McLachlan, G. J. (1993) · Journal of the American Statistical Association · doi:10.2307/2290355 · Cited by 2,662 · see citing works
Learning in the Presence of Concept Drift and Hidden Contexts
Widmer, G., & Kubat, M. (1996) · Machine Learning · doi:10.1023/a:1018046501280 · Cited by 1,857 · see citing works
Very Simple Classification Rules Perform Well on Most Commonly Used Datasets
Holte, R. C. (1993) · Machine Learning · doi:10.1023/a:1022631118932 · Cited by 1,800 · see citing works
Statistical Pattern Recognition
Webb, A. R. (2002) · Wiley eBooks · doi:10.1002/0470854774 · Cited by 1,468 · see citing works
Statistical modeling: The two cultures
Breiman, L. (2001) · Cited by 1,314 · see citing works
Discriminant Analysis and Statistical Pattern Recognition
McLachlan, G. J. (1992) · Wiley series in probability and statistics · doi:10.1002/0471725293 · Cited by 1,272 · see citing works
Robust classification for imprecise environments
Provost, F., & Fawcett, T. (2001) · Machine Learning · doi:10.1023/a:1007601015854 · Cited by 1,250 · see citing works
Statistical Classification Methods in Consumer Credit Scoring: A Review
Hand, D. J., & Henley, W. E. (1997) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.1997.00078.x · Cited by 1,042 · see citing works
Self-citationOn Bias, Variance, 0/1—Loss, and the Curse-of-Dimensionality
Friedman, J. H. (1997) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009778005914 · Cited by 1,038 · see citing works
Discrimination and Classification
Loader, C. R. (1999) · Statisctics and computing/Statistics and computing · doi:10.1007/0-387-22732-6_8 · Cited by 929 · see citing works
Fawcett, T. E., & Provost, F. J. (1997) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009700419189 · Cited by 889 · see citing works
On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach
Salzberg, S. L. (1997) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009752403260 · Cited by 833 · see citing works
A survey of credit and behavioural scoring: forecasting financial risk of lending to consumers
Thomas, L. C. (2000) · International Journal of Forecasting · doi:10.1016/s0169-2070(00)00034-0 · Cited by 745 · see citing works
An Empirical Comparison of Pruning Methods for Decision Tree Induction
Mingers, J. (1989) · Machine Learning · doi:10.1023/a:1022604100933 · Cited by 632 · see citing works
Construction and Assessment of Classification Rules
Wade, M. R., & Hand, D. J. (1999) · Technometrics · doi:10.2307/1270579 · Cited by 621 · see citing works
Self-citationAn Empirical Comparison of Selection Measures for Decision-Tree Induction
Mingers, J. (1989) · Machine Learning · doi:10.1023/a:1022645801436 · Cited by 561 · see citing works
Evaluation Methods in Medical Informatics
Friedman, C. P., & Wyatt, J. C. (1997) · Computers and medicine · doi:10.1007/978-1-4757-2685-5 · Cited by 450 · see citing works
Detecting Concept Drift with Support Vector Machines
Klinkenberg, R., & Joachims, T. (2000) · Cited by 412 · see citing works
Symbolic and Neural Learning Algorithms: An Experimental Comparison
SHAVLIK, J. W., Mooney, R. J., & Towell, G. G. (1991) · Machine Learning · doi:10.1023/a:1022602303196 · Cited by 323 · see citing works
Quantitative Methods in Credit Management: A Survey
Rosenberg, E., & Gleit, A. (1994) · Operations Research · doi:10.1287/opre.42.4.589 · Cited by 240 · see citing works
The impact of changing populations on classifier performance
Kelly, M. G., Hand, D. J., & Adams, N. M. (1999) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/312129.312285 · Cited by 226 · see citing works
Self-citationNowak, G. J., & Phelps, J. E. (1997) · Journal of Direct Marketing · doi:10.1002/(sici)1522-7138(199723)11:4 · Cited by 189 · see citing works
Comparing classifiers when the misallocation costs are uncertain
Adams, N. M., & Hand, D. J. (1999) · Pattern Recognition · doi:10.1016/s0031-3203(98)00154-x · Cited by 171 · see citing works
Self-citationVertebral morphometry: normative data.
Gallagher, J. C., Hedlund, L. R., Stoner, S., et al. (1988) · PubMed · Cited by 130 · see citing works
Approaches to online learning and concept drift for user identification in computer security
Lane, T. D. R., & Brodley, C. E. (1998) · Cited by 120 · see citing works
Learning hard concepts through constructive induction: framework and rationale
Rendell, L. A., & Seshu, R. (1990) · Computational Intelligence · doi:10.1111/j.1467-8640.1990.tb00298.x · Cited by 97 · see citing works
Costs and payoffs in perceptual research.
von Winterfeldt, D., & Edwards, W. (1982) · Psychological Bulletin · doi:10.1037/0033-2909.91.3.609 · Cited by 95 · see citing works
A note on comparing classifiers
Duin, R. P. W. (1996) · Pattern Recognition Letters · doi:10.1016/0167-8655(95)00113-1 · Cited by 88 · see citing works
Modelling consumer credit risk
Hand, D. J. (2001) · IMA Journal of Management Mathematics · doi:10.1093/imaman/12.2.139 · Cited by 75 · see citing works
Self-citationMaximizing the predictive value of production rules
Weiss, S. M., Galen, R. S., & Tadepalli, P. V. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90037-z · Cited by 69 · see citing works
Mining Supervised Classification Performance Studies: A Meta-Analytic Investigation
Jamain, A., & Hand, D. J. (2008) · Journal of Classification · doi:10.1007/s00357-008-9003-y · Cited by 42 · see citing works
Self-citationLearning hard concepts through constructive induction: framework and rationale
Rendell, L. A., & Seshu, R. (1994) · Conference on Learning Theory · Cited by 40 · see citing works
Direct versus indirect credit scoring classifications
Li, H. G., & Hand, D. J. (2002) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601346 · Cited by 19 · see citing works
Self-citationA Conversaton with Leo Breiman
Olshen, R. A. (2001) · Statistical Science · doi:10.1214/ss/1009213290 · Cited by 17 · see citing works
Credit scoring with uncertain class definitions
Kelly, M. G. (1999) · IMA Journal of Management Mathematics · doi:10.1093/imaman/10.4.331 · Cited by 15 · see citing works
Classification and Computers: Shifting the Focus
Hand, D. J. (1996) · COMPSTAT · doi:10.1007/978-3-642-46992-3_7 · Cited by 12 · see citing works
Self-citationHand, D. J. (2004) · Springer eBooks · doi:10.1007/978-3-642-17103-1_21 · Cited by 11 · see citing works
Self-citationSupervised Classification Problems: How to Be Both Judge and Jury
Kelly, M. G., Hand, D. J., & Adams, N. M. (1999) · Lecture notes in computer science · doi:10.1007/3-540-48412-4_20 · Cited by 10 · see citing works
Self-citationSupervised classification and tunnel vision
Hand, D. J. (2005) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.540 · Cited by 9 · see citing works
Self-citationDefining the goals to optimise data mining performance
Kelly, M. G., Hand, D. J., & Adams, N. M. (1998) · Cited by 8 · see citing works
Self-citationMeta-analysis of classification methods
Jamain, A. (2005) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 6 · see citing works
Intelligent Data Analysis and Deep Understanding
Hand, D. J. (1999) · Springer eBooks · doi:10.1007/978-3-642-58648-4_5 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- BENTON, T. C. (2002). Theoretical and empirical models. Ph.D. dissertation, Dept. Mathematics, Imperial College Lon- don.
- HAND, D. J. (2001). Reject inference in credit operations. In Handbook of Credit Scoring (E. Mays, ed.) 225-240. Glen- lake, Chicago.
- Morgan Kaufmann, San Francisco.
Cited by
Sorted by citations
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · see citing works
Strobl, C., Malley, J., & Tutz, G. (2009) · Psychological Methods · doi:10.1037/a0016973 · Cited by 2,583 · see citing works
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119 · see citing works
Benchmarking state-of-the-art classification algorithms for credit scoring: an update of research
Lessmann, S., Baesens, B., Seow, H.-V., & Thomas, L. C. (2015) · European Journal of Operational Research · doi:10.1016/j.ejor.2015.05.030 · Cited by 1,284 · see citing works
Machine learning and its applications to biology
Tarca, A. L., Carey, V. J., Chen, X.-W., Romero, R., & Drăghici, S. (2007) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0030116 · Cited by 656 · see citing works
Predictive Analytics in Information Systems Research
Shmueli, G., & Koppius, O. (2010) · doi:10.2139/ssrn.1606674 · Cited by 676 · see citing works
Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008) · Political Analysis · doi:10.1093/pan/mpn018 · Cited by 542 · see citing works
The promise of machine learning in predicting treatment outcomes in psychiatry
Chekroud, A. M., Bondar, J., Delgadillo, J., Doherty, G., Wasil, A., Fokkema, M., Cohen, Z., Belgrave, D., DeRubeis, R., Iniesta, R., et al. (2021) · World Psychiatry · doi:10.1002/wps.20882 · Cited by 531 · see citing works
Data mining and machine learning in astronomy
BALL, N. M., & BRUNNER, R. J. (2010) · International Journal of Modern Physics D · doi:10.1142/s0218271810017160 · Cited by 383 · see citing works
Measuring the predictability of life outcomes with a scientific mass collaboration
Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915006117 · Cited by 322 · see citing works
Transaction aggregation as a strategy for credit card fraud detection
Whitrow, C., Hand, D. J., Juszczak, P., Weston, D., & Adams, N. M. (2009) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-008-0116-z · Cited by 337 · see citing works
Self-citationWILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · see citing works
Perkins, D. O., Jeffries, C. D., Addington, J., Bearden, C. E., Cadenhead, K. S., Cannon, T. D., Cornblatt, B. A., Mathalon, D. H., McGlashan, T. H., Seidman, L. J., et al. (2015) · Schizophrenia Bulletin · doi:10.1093/schbul/sbu099 · Cited by 238 · see citing works
A review of the F-measure: its history, properties, criticism, and alternatives
Christen, P., Hand, D. J., & Kirielle, N. (2024) · ACM Computing Surveys · doi:10.1145/3606367 · Cited by 236 · see citing works
Self-citationBig data in IBD: big progress for clinical practice
Seyed Tabib, N. S., Madgwick, M., Sudhakar, P., Verstockt, B., Korcsmaros, T., & Vermeire, S. (2020) · Gut · doi:10.1136/gutjnl-2019-320065 · Cited by 223 · see citing works
Kubota, K. J., Chen, J. A., & Little, M. A. (2016) · Movement Disorders · doi:10.1002/mds.26693 · Cited by 210 · see citing works
Conscientious Classification: A Data Scientist's Guide to Discrimination-Aware Classification
d'Alessandro, B., O'Neil, C., & LaGatta, T. (2017) · Big Data · doi:10.1089/big.2016.0048 · Cited by 195 · see citing works
Bridging the divide in financial market forecasting: machine learners vs. financial economists
Hsu, M.-W., Lessmann, S., Sung, M.-C., Ma, T., & Johnson, J. E. V. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.05.033 · Cited by 195 · see citing works
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211 · see citing works
Targeting Prospective Customers: Robustness of Machine-Learning Methods to Typical Data Challenges
Simester, D., Timoshenko, A., & Zoumpoulis, S. I. (2020) · Management Science · doi:10.1287/mnsc.2019.3308 · Cited by 133 · see citing works
Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)
Berrar, D., & Flach, P. (2012) · Briefings in Bioinformatics · doi:10.1093/bib/bbr008 · Cited by 136 · see citing works
Vale, D., El-Sharif, A., & Ali, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00142-y · Cited by 124 · see citing works
Bayesian networks for interpretable machine learning and optimization
Mihaljević, B., Bielza, C., & Larrañaga, P. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.138 · Cited by 123 · see citing works
Bayamlıoğlu, E. (2022) · Regulation & Governance · doi:10.1111/rego.12391 · Cited by 44 · see citing works
Constitutional Rights in the Machine Learning State
Huq, A. Z. (2020) · doi:10.2139/ssrn.3613282 · Cited by 21 · see citing works
A Paradigm for Democratizing Artificial Intelligence Research
Moreau, E., Vogel, C., & Barry, M. (2019) · Intelligent Systems Reference Library · doi:10.1007/978-3-030-15939-9_8 · Cited by 10 · see citing works
- Authors
- Hand, D. J.
- Corresponding author
- Hand, David J.
- ORCID
- Hand, David J.
- Venue
- Statistical Science
- Publisher
- Institute of Mathematical Statistics
- Published
- 2006-02-01
- Volume
- 21
- Issue
- 1
- Pages
- 1-14
- ISSN
- 0883-4237 (ISSN-L)
- 2168-8745
- OpenAlex
- W1973682096
- Cited by
- 723
- References
- 46
- Citations by year
- 2026: 29 · 2025: 21 · 2024: 29 · 2023: 25 · 2022: 39 · 2021: 55 · 2020: 43 · 2019: 59 · 2018: 32 · 2017: 32 · 2016: 38 · 2015: 37 · 2014: 49 · 2013: 45 · 2012: 37
- FWCI
- 31.53
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- public-domain
- Locations
- Statistical Science (publishedVersion, public-domain)
- arXiv (Cornell University) (submittedVersion)
- Project Euclid (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Keywords
- supervised classification, classifier comparison, linear discriminant analysis, artificial neural networks, support vector machine, classification uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
