The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
An extensive experimental survey of regression methods
Open accessAbstract
Regression is a very relevant problem in machine learning, with many different available approaches. The current work presents a comparison of a large collection composed by 77 popular regression models which belong to 19 families: linear and generalized linear models, generalized additive models, least squares, projection methods, LASSO and ridge regression, Bayesian models, Gaussian processes, quantile regression, nearest neighbors, regression trees and rules, random forests, bagging and boosting, neural networks, deep learning and support vector regression. These methods are evaluated using all the regression datasets of the UCI machine learning repository (83 datasets), with some exceptions due to technical reasons. The experimental work identifies several outstanding regression models: the M5 rule-based model with corrections based on nearest neighbors (cubist), the gradient boosted machine (gbm), the boosting ensemble of regression trees (bstTree) and the M5 regression tree. Cubist achieves the best squared correlation ( R2) in 15.7% of datasets being very near to it, with difference below 0.2 for 89.1% of datasets, and the median of these differences over the dataset collection is very low (0.0192), compared e.g. to the classical linear regression (0.150). However, cubist is slow and fails in several large datasets, while other similar regression models as M5 never fail and its difference to the best R2 is below 0.2 for 92.8% of datasets. Other well-performing regression models are the committee of neural networks (avNNet), extremely randomized regression trees (extraTrees, which achieves the best R2 in 33.7% of datasets), random forest (rf) and ε-support vector regression (svr), but they are slower and fail in several datasets. The fastest regression model is least angle regression lars, which is 70 and 2,115 times faster than M5 and cubist, respectively. The model which requires least memory is non-negative least squares (nnls), about 2 GB, similarly to cubist, while M5 requires about 8 GB. For 97.6% of datasets there is a regression model among the 10 bests which is very near (difference below 0.1) to the best R2, which increases to 100% allowing differences of 0.2. Therefore, provided that our dataset and model collection are representative enough, the main conclusion of this study is that, for a new regression problem, some model in our top-10 should achieve R2 near to the best attainable for that problem.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statistics and Computing · doi:10.1007/978-0-387-21706-2 · Cited by 24,305
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
An Analysis of Transformations
Box, G. E. P., & Cox, D. R. (1964) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1964.tb00553.x · Cited by 15,052
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300
Independent component analysis: algorithms and applications
Hyvärinen, A., & Oja, E. (2000) · Neural Networks · doi:10.1016/s0893-6080(00)00026-5 · Cited by 8,705
Multivariate Adaptive Regression Splines
Friedman, J. H. (1991) · The Annals of Statistics · doi:10.1214/aos/1176347963 · Cited by 8,000
Wood, S. N. (2011) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2010.00749.x · Cited by 7,537
Solving Least Squares Problems
Lawson, C. L., & Hanson, R. J. (1995) · doi:10.1137/1.9781611971217 · Cited by 7,103
Pattern Recognition and Neural Networks
Ripley, B. D. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812651 · Cited by 6,459
A Multiple Comparison Procedure for Comparing Several Treatments with a Control
Dunnett, C. W. (1955) · Journal of the American Statistical Association · doi:10.1080/01621459.1955.10501294 · Cited by 6,094
Nonparametric Statistical Methods
(1999) · Technometrics · doi:10.1080/00401706.1999.10485954 · Cited by 6,090
Feature Selection with the Boruta Package
Kursa, M. B., & Rudnicki, W. R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i11 · Cited by 5,659
Extreme Learning Machine for Regression and Multiclass Classification
Huang, G.-B., Zhou, H., Ding, X., & Zhang, R. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2011.2168604 · Cited by 5,550
A general regression neural network
Specht, D. F. (1991) · IEEE Transactions on Neural Networks · doi:10.1109/72.97934 · Cited by 4,492
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.415 · Cited by 4,388
Unbiased Recursive Partitioning: A Conditional Inference Framework
Hothorn, T., Hornik, K., & Zeileis, A. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x133933 · Cited by 4,021
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
Searle, S. R., & Searle, S. R. (1971) · Cited by 3,229
Regularization Paths for Cox's Proportional Hazards Model via Coordinate Descent
Simon, N., Friedman, J., Hastie, T., & Tibshirani, R. (2011) · Journal of Statistical Software · doi:10.18637/jss.v039.i05 · Cited by 2,499
Nonparametric Statistical Inference
(2012) · SpringerReference · doi:10.1007/springerreference_205502 · Cited by 2,386
Modern Applied Statistics with S,
Kemp, F. (2003) · Journal of the Royal Statistical Society: Series D (The Statistician) · doi:10.1046/j.1467-9884.2003.t01-19-00383_22.x · Cited by 2,320
Friedman, J. H., & Stuetzle, W. (1981) · Journal of the American Statistical Association · doi:10.1080/01621459.1981.10477729 · Cited by 2,216
SIMPLS: An alternative approach to partial least squares regression
de Jong, S. (1993) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(93)85002-x · Cited by 1,905
A weakly informative default prior distribution for logistic and other regression models
Gelman, A., Jakulin, A., Pittau, M. G., & Su, Y.-S. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas191 · Cited by 1,790
Graph-based approach for airborne light detection and ranging segmentation
Vilariño, D. L., Cabaleiro, J. C., Martínez, J., Rivera, F. F., & Pena, T. F. (2017) · Journal of Applied Remote Sensing · doi:10.1117/1.jrs.11.015020 · Cited by 1,721
The Lack of A Priori Distinctions Between Learning Algorithms
Wolpert, D. H. (1996) · Neural Computation · doi:10.1162/neco.1996.8.7.1341 · Cited by 1,647
Meinshausen, N. (2006) · Cited by 1,521
Pattern Recognition and Neural Networks
Fulkerson, B. (1997) · Technometrics · doi:10.1080/00401706.1997.10485099 · Cited by 1,371
A WEAKLY INFORMATIVE DEFAULT PRIOR DISTRIBUTION FOR LOGISTIC AND OTHER REGRESSION MODELS
Gelman, A., Jakulin, A., & Grazia, M. (2009) · Cited by 1,352
Statistical Comparisons of Classifiers over Multiple Data Sets
DemšarJanez (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248548 · Cited by 1,074
One-step sparse estimates in nonconcave penalized likelihood modelsPreprint
Zou, H., & Li, R. (2008) · Cited by 919
Sparse Partial Least Squares Regression for Simultaneous Dimension Reduction and Variable Selection
Chun, H., & Keleş, S. (2010) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2009.00723.x · Cited by 908
Boosting With the L2Loss: Regression and Classification
Bühlmann, P., & Yu, B. (2003) · Journal of the American Statistical Association · doi:10.1198/016214503000125 · Cited by 845
L1Penalized Estimation in the Cox Proportional Hazards Model
Goeman, J. J. (2010) · Biometrical Journal · doi:10.1002/bimj.200900028 · Cited by 816
Combining Instance-Based and Model-Based Learning
Quinlan, J. R. (1993) · Elsevier eBooks · doi:10.1016/b978-1-55860-307-3.50037-x · Cited by 773
Bayesian classification with Gaussian processes
Williams, C. K. I., & Barber, D. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.735807 · Cited by 766
Semi-Supervised Methods to Predict Patient Survival from Gene Expression Data
Bair, E., & Tibshirani, R. (2004) · PLoS Biology · doi:10.1371/journal.pbio.0020108 · Cited by 754
Boosting Algorithms: Regularization, Prediction and Model Fitting
Bühlmann, P., & Hothorn, T. (2007) · Statistical Science · doi:10.1214/07-sts242 · Cited by 726
BOOSTING ALGORITHMS: REGULARIZATION, PREDICTION AND MODEL FITTINGPreprint
Bühlmann, P., & Hothorn, T. (2007) · Cited by 705
One-step sparse estimates in nonconcave penalized likelihood models
Zou, H., & Li, R. (2008) · The Annals of Statistics · doi:10.1214/009053607000000802 · Cited by 683
Hainmueller, J., & Hazlett, C. (2014) · Political Analysis · doi:10.1093/pan/mpt019 · Cited by 517
Meinshausen, N. (2007) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2006.12.019 · Cited by 492
Cannon, A. J. (2011) · Computers & Geosciences · doi:10.1016/j.cageo.2010.07.005 · Cited by 404
Nonparametric statistical methods
(1974) · Behavior Therapy · doi:10.1016/s0005-7894(74)80108-5 · Cited by 342
Rännar, S., Lindgren, F., Geladi, P., & Wold, S. (1994) · Journal of Chemometrics · doi:10.1002/cem.1180080204 · Cited by 330
Weighted k-Nearest-Neighbor Techniques and Ordinal Classification
Hechenbichler, K., & Schliep, K. P. (2004) · Econstor (Econstor) · doi:10.5282/ubm/epub.1769 · Cited by 307
Gene selection with guided regularized random forest
Deng, H., & Runger, G. (2013) · Pattern Recognition · doi:10.1016/j.patcog.2013.05.018 · Cited by 304
bartMachine: Machine Learning with Bayesian Additive Regression Trees
Kapelner, A., & Bleich, J. (2016) · Journal of Statistical Software · doi:10.18637/jss.v070.i04 · Cited by 270
Mevik, B., & Cederkvist, H. R. (2004) · Journal of Chemometrics · doi:10.1002/cem.887 · Cited by 249
Adaptive Forward-Backward Greedy Algorithm for Learning Sparse Representations
Zhang, T. (2011) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2146690 · Cited by 248
Supervised Kohonen networks for classification problems
Melssen, W., Wehrens, R., & Buydens, L. (2006) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2006.02.003 · Cited by 212
The Design and Analysis of Benchmark Experiments
Hothorn, T., Leisch, F., Zeileis, A., & Hornik, K. (2005) · Journal of Computational and Graphical Statistics · doi:10.1198/106186005x59630 · Cited by 181
Random generalized linear model: a highly accurate and interpretable ensemble predictor
Lin, S. S., Langfelder, P., & Horvath, S. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-5 · Cited by 161
evtree: Evolutionary Learning of Globally Optimal Classification and Regression Trees in R
Grubinger, T., Zeileis, A., & Pfeiffer, K.-P. (2014) · Journal of Statistical Software · doi:10.18637/jss.v061.i01 · Cited by 159
Meinshausen, N. (2010) · The Annals of Applied Statistics · doi:10.1214/10-aoas367 · Cited by 151
Comments on the PLS kernel algorithm
de Jong, S., & ter Braak, C. J. F. (1994) · Journal of Chemometrics · doi:10.1002/cem.1180080208 · Cited by 77
Nonparametric statistical inference
Stoimenova, E. (2012) · Journal of Applied Statistics · doi:10.1080/02664763.2012.657415 · Cited by 74
partDSA: deletion/substitution/addition algorithm for partitioning the covariate space in prediction
Molinaro, A. M., Lostritto, K., & van der Laan, M. J. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq142 · Cited by 53
Koenker, R. W., & Mizera, I. (2014) · Journal of Statistical Software · doi:10.18637/jss.v060.i05 · Cited by 52
spikeslab: Prediction and Variable Selection Using Spike and Slab Regression
Ishwaran, H., Udaya, B. K., & Sunil Rao, J. (2010) · The R Journal · doi:10.32614/rj-2010-018 · Cited by 38
L-1 Penalized Estimation in the Cox Proportional Hazards Model
Goeman, J. J. (2010) · Data Archiving and Networked Services (DANS) · Cited by 26
Cited by
Machine Learning for Brain Stroke: A Review
Sirsat, M. S., Fermé, E., & Câmara, J. (2020) · Journal of Stroke and Cerebrovascular Diseases · doi:10.1016/j.jstrokecerebrovasdis.2020.105162 · Cited by 280
Self-citationMachine learning for hydrologic sciences: An introductory overview
Xu, T., & Liang, F. (2021) · WIREs Water · doi:10.1002/wat2.1533 · Cited by 267
On-device machine learning An algorithms and learning theory perspective
Dhar, S., Guo, J., Liu, J., Tripathi, S., Kurup, U., & Shah, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00623 · Cited by 47
Metadata
- Authors
- Fernández-Delgado, M., Sirsat, M. S., Cernadas, E., Alawadi, S., Barro, S., & Febrero-Bande, M.
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2019-03
- Volume
- 111
- Pages
- 11-34
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2906432502
- Cited by
- 377
- References
- 70
- FWCI
- 11.32
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.