Constructive Incremental Learning from Only Local Information
The DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index.
Abstract
We introduce a constructive, incremental learning system for regression problems that models data by means of spatially localized linear models. In contrast to other approaches, the size and shape of the receptive field of each locally linear model, as well as the parameters of the locally linear model itself, are learned independently, that is, without the need for competition or any other kind of communication. Independent learning is accomplished by incrementally minimizing a weighted local cross-validation error. As a result, we obtain a learning system that can allocate resources as needed while dealing with the bias-variance dilemma in a principled way. The spatial localization of the linear models increases robustness toward negative interference. Our learning system can be interpreted as a nonparametric adaptive bandwidth smoother, as a mixture of experts where the experts are trained in isolation, and as a learning system that profits from combining independent expert knowledge on the same problem. This article illustrates the potential learning capabilities of purely local learning and offers an interesting and powerful approach to learning with receptive fields.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Neural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
Probability, Random Variables, and Stochastic Processes.
Abrahams, J., & Papoulis, A. (1984) · Journal of the American Statistical Association · doi:10.2307/2288754 · Cited by 16,226
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
de Boor, C. (1978) · Applied Mathematical Sciences · doi:10.1007/978-1-4612-6333-3 · Cited by 11,944
Robust Locally Weighted Regression and Smoothing Scatterplots
Cleveland, W. S. (1979) · Journal of the American Statistical Association · doi:10.1080/01621459.1979.10481038 · Cited by 11,594
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881
Brown, R. A., Hastie, T. J., & Tibshirani, R. J. (1991) · Biometrics · doi:10.2307/2532174 · Cited by 8,176
Multivariate Adaptive Regression Splines
Friedman, J. H. (1991) · The Annals of Statistics · doi:10.1214/aos/1176347963 · Cited by 8,000
Regression Diagnostics -- Identifying Influential Data and Sources of Collinearity
Beaumont, C. D., Belsley, D. A., Kuh, E., & Welsch, R. E. (1981) · The Journal of the Operational Research Society · doi:10.2307/2581267 · Cited by 6,359
Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting
Cleveland, W. S., & Devlin, S. J. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478639 · Cited by 5,742
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 1×
“…suggests that information processing based on local receptive fields is a ubiquitous organizational principle in neurobiology that offers interesting computational opportunities (e.g., Lee, Rohrer, & Sparks, 1988; Georgopoulos, 1991; Field, 1994; Olshausen & Field, 1996; Daugman & Downing, 1995).”
BackgroundAdaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
Spline Models for Observational Data.
Muller, H.-G., & Wahba, G. (1991) · Journal of the American Statistical Association · doi:10.2307/2290434 · Cited by 4,928
Receptive fields of single neurones in the cat's striate cortex
Hubel, D. H., & Wiesel, T. N. (1959) · The Journal of Physiology · doi:10.1113/jphysiol.1959.sp006308 · Cited by 4,653
Theory and Practice of Recursive Identification
Lennart Ljung, A., & Söderström, T. (1983) · Lund University Publications (Lund University) · Cited by 3,989
The Handbook of brain theory and neural networks
(1996) · Choice Reviews Online · doi:10.5860/choice.33-4484 · Cited by 3,870
Multivariate Density Estimation: Theory, Practice, and Visualization
Scott, D. W. (1992) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316849 · Cited by 3,696
Classical and Modern Regression With Applications.
Steinhorst, R. K., & Myers, R. H. (1988) · Journal of the American Statistical Association · doi:10.2307/2288958 · Cited by 3,690
Neural Networks and the Bias/Variance Dilemma
Geman, S., Bienenstock, E., & Doursat, R. (1992) · Neural Computation · doi:10.1162/neco.1992.4.1.1 · Cited by 3,578 · cited 2×
“Due to the bias-variance tradeoff (Geman et al., 1992), learning algorithms have to include a model selection phase in order to find an appropriate compromise between oversmoothing and overfitting.”
Nadaraya, E. A. (1964) · Theory of Probability & Its Applications · doi:10.1137/1109020 · Cited by 3,462
Journal of the Royal Statistical Society (B)
Fildes, R. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90088-5 · Cited by 3,437
Journal of the Royal Statistical Society.
(1923) · The Economic Journal · doi:10.1093/ej/33.130.269 · Cited by 3,219
Hierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.181 · Cited by 2,673
Watson, G. S. (1964) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 2,661
Adaptive representation of dynamics during learning of a motor task
Shadmehr, R., & Mussa-Ivaldi, F. (1994) · The Journal of Neuroscience · doi:10.1523/jneurosci.14-05-03208.1994 · Cited by 2,612
A massively parallel architecture for a self-organizing neural pattern recognition machine
Carpenter, G. A., & Grossberg, S. (1987) · Computer Vision, Graphics, and Image Processing · doi:10.1016/s0734-189x(87)80014-2 · Cited by 2,525 · cited 6×
“Cleveland (1979) introduced the idea of employing locally linear models for memory-based function approximation, called locally weighted regression (LWR).”
BackgroundMethodInfluentialMODALITY AND TOPOGRAPHIC PROPERTIES OF SINGLE NEURONS OF CAT'S SOMATIC SENSORY CORTEX
Mountcastle, V. B. (1957) · Journal of Neurophysiology · doi:10.1152/jn.1957.20.4.408 · Cited by 2,400
The Cascade-Correlation learning architecture
Fahlman, S. E., & Lebiere, C. J. (2018) · doi:10.1184/r1/6610403 · Cited by 2,372
Friedman, J. H., & Stuetzle, W. (1981) · Journal of the American Statistical Association · doi:10.1080/01621459.1981.10477729 · Cited by 2,216
Predicting chaotic time series
Farmer, J. D., & Sidorowich, J. J. (1987) · Physical Review Letters · doi:10.1103/physrevlett.59.845 · Cited by 1,929
Churchland, P. S., & Sejnowski, T. J. (1992) · doi:10.7551/mitpress/2010.001.0001 · Cited by 1,757
Atkeson, C. G., Moore, A. W., & Schaal, S. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006559212014 · Cited by 1,671
Self-citationA Resource-Allocating Network for Function Interpolation
Platt, J. (1991) · Neural Computation · doi:10.1162/neco.1991.3.2.213 · Cited by 1,354
Growing cell structures—A self-organizing network for unsupervised and supervised learning
Fritzke, B. (1994) · Neural Networks · doi:10.1016/0893-6080(94)90091-4 · Cited by 1,270
Radial basis functions for multivariable interpolation: a review
Powell, M. J. D. (1987) · Clarendon Press eBooks · Cited by 1,197
Zipser, D., & Andersen, R. A. (1988) · Nature · doi:10.1038/331679a0 · Cited by 1,187
What Is the Goal of Sensory Coding?
Field, D. J. (1994) · Neural Computation · doi:10.1162/neco.1994.6.4.559 · Cited by 1,183
Merzenich, M. M., Kaas, J. H., Wall, J., Nelson, R. J., Sur, M., & Felleman, D. (1983) · Neuroscience · doi:10.1016/0306-4522(83)90024-6 · Cited by 1,042
Regularization Algorithms for Learning That Are Equivalent to Multilayer Networks
Poggio, T., & Girosi, F. (1990) · Science · doi:10.1126/science.247.4945.978 · Cited by 1,026
Topology representing networks
Martinetz, T., & Schulten, K. (1994) · Neural Networks · doi:10.1016/0893-6080(94)90109-0 · Cited by 883
The neural basis of cognitive development: A constructivist manifesto
Quartz, S. R., & Sejnowski, T. J. (1997) · Behavioral and Brain Sciences · doi:10.1017/s0140525x97001581 · Cited by 864
When networks disagree: Ensemble methods for hybrid neural networks
PERRONE, M. P., & COOPER, L. N. (1995) · World Scientific Series in 20th Century Physics · doi:10.1142/9789812795885_0025 · Cited by 755
Population coding of saccadic eye movements by neurons in the superior colliculus
Lee, C., Rohrer, W. H., & Sparks, D. L. (1988) · Nature · doi:10.1038/332357a0 · Cited by 732
Variable Bandwidth and Local Linear Regression Smoothers
Fan, J., & Gijbels, I. (1992) · The Annals of Statistics · doi:10.1214/aos/1176348900 · Cited by 635
Cleveland, W. S., Devlin, S. J., & Grosse, E. (1988) · Journal of Econometrics · doi:10.1016/0304-4076(88)90077-2 · Cited by 599
Fan, J., & Gijbels, I. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02034.x · Cited by 566
Model-based control of a robot manipulator
(1988) · Choice Reviews Online · doi:10.5860/choice.26-1542 · Cited by 553
Locally Weighted Learning for Control
Atkeson, C. G., Moore, A. W., & Schaal, S. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006511328852 · Cited by 527
Self-citationThe Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks
Frean, M. R. (1990) · Neural Computation · doi:10.1162/neco.1990.2.2.198 · Cited by 517
Smoothing by Local Regression: Principles and Methods
Cleveland, W. S., & Loader, C. (1996) · Contributions to Statistics · doi:10.1007/978-3-642-48425-4_2 · Cited by 456
Local Regression: Automatic Kernel Carpentry
Hastie, T., & Loader, C. (1993) · Statistical Science · doi:10.1214/ss/1177011002 · Cited by 415
A neural model for category learning
Reilly, D. L., Cooper, L. N., & Elbaum, C. (1982) · Biological Cybernetics · doi:10.1007/bf00387211 · Cited by 408
(2011) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_447 · Cited by 400
Aha, D. W. (1997) · Cited by 381
Friedman, J. H. (1984) · doi:10.21236/ada148241 · Cited by 379
A completely automatic french curve: fitting spline functions by cross validation
Wahba, G., & Wold, S. (1975) · Communications in Statistics · doi:10.1080/03610927508827223 · Cited by 366
Regularization in the Selection of Radial Basis Function Centers
Orr, M. J. L. (1995) · Neural Computation · doi:10.1162/neco.1995.7.3.606 · Cited by 357
Georgopoulos, A. P. (1991) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.14.030191.002045 · Cited by 305
An Information-Theoretic Approach to Neural Computing
Deco, G., & Obradovic, D. (1996) · Perspectives in Neural Computing · doi:10.1007/978-1-4612-4016-7 · Cited by 292
Exploiting Chaos to Predict the Future and Reduce Noise
Farmer, J. D., & Sidorowichl, J. J. (1989) · Evolution, Learning and Cognition · doi:10.1142/9789814434102_0011 · Cited by 278
Similarity Metric Learning for a Variable-Kernel Classifier
Lowe, D. G. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.72 · Cited by 267
LEARNING ARM KINEMATICS AND DYNAMICS
Atkeson, C. G. (1989) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.12.030189.001105 · Cited by 265
Self-citationRobot juggling: implementation of memory-based learning
(1994) · IEEE Control Systems · doi:10.1109/37.257895 · Cited by 242
Dusser de Barenne, J. G., Fulton, J. F., & Gerard, R. W. (1938) · The American Journal of the Medical Sciences · doi:10.1097/00000441-193808000-00024 · Cited by 204
From Statistics to Neural Networks: Theory and Pattern Recognition Applications
Cherkassky, V., Friedman, J. H., & Wechsler, H. L. (1996) · CERN Document Server (European Organization for Nuclear Research) · Cited by 175
Adapting bias by gradient descent: an incremental version of delta-bar-delta
Sutton, R. S. (1992) · Cited by 171
Aha, D. W. (1997) · Artificial Intelligence Review · doi:10.1023/a:1006538427943 · Cited by 162
Shin, Y., & Ghosh, J. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.377967 · Cited by 149
Modular features of motor control and learning
Mussa-Ivaldi, F. A. (1999) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(99)00029-x · Cited by 140
A course in nonparametric statistics
Hájek, J. (1969) · Cited by 134
What is the Goal of Sensory Coding?
Field, D. J. (1999) · The MIT Press eBooks · doi:10.7551/mitpress/7011.003.0008 · Cited by 130
A tree-structured adaptive network for function approximation in high-dimensional spaces
Sanger, T. D. (1991) · IEEE Transactions on Neural Networks · doi:10.1109/72.80339 · Cited by 127
Space-frequency localized basis function networks for nonlinear system estimation and control
Cannon, M., & Slotine, J.-J. E. (1995) · Neurocomputing · doi:10.1016/0925-2312(95)00036-1 · Cited by 124
Imamizu, H., Uno, Y., & Kawato, M. (1995) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.21.5.1174 · Cited by 123
Topology conserving mappings for learning motor tasks
Ritter, H. J., & Schulten, K. (1986) · AIP conference proceedings · doi:10.1063/1.36242 · Cited by 84
An incremental-learning neural network for the classification of remote-sensing images
Bruzzone, L., & Prieto, D. F. (1999) · Pattern Recognition Letters · doi:10.1016/s0167-8655(99)00091-4 · Cited by 71
Memory-based neural networks for robot learning
Atkeson, C. G., & Schaal, S. (1995) · Neurocomputing · doi:10.1016/0925-2312(95)00033-6 · Cited by 70
Self-citationImproved Gaussian Mixture Density Estimates Using Bayesian Penalty Terms and Network Averaging
Ormoneit, D., & Tresp, V. (1995) · Cited by 65
Fast, Robust Adaptive Control by Learning only Forward Models
Moore, A. (1991) · neural information processing systems · Cited by 52
Interference in Learning Internal Models of Inverse Dynamics in Humans
Shadmehr, R., Brashers-Krug, T., & Mussa-Ivaldi, F. A. (1995) · Cited by 28
Generalization Abilities of Cascade Network Architecture
Littmann, E., & Ritter, H. J. (1992) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 27
Local dimensionality reduction for locally weighted learning
Vijayakumar, S., & Schaal, S. (2002) · IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA) · doi:10.1109/cira.1997.613861 · Cited by 23
Self-citation
Cited by
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors
Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., & Schaal, S. (2013) · Neural Computation · doi:10.1162/neco_a_00393 · Cited by 1,598
Self-citationRobot Programming by Demonstration
Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008) · Springer Handbook of Robotics · doi:10.1007/978-3-540-30301-5_60 · Cited by 1,002
Self-citationLearning of action through adaptive combination of motor primitives
Thoroughman, K. A., & Shadmehr, R. (2000) · Nature · doi:10.1038/35037588 · Cited by 945
Movement imitation with nonlinear dynamical systems in humanoid robots
Ijspeert, A. J., Nakanishi, J., & Schaal, S. (2003) · Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292) · doi:10.1109/robot.2002.1014739 · Cited by 840
Self-citation
Metadata
- Authors
- Schaal, S., & Atkeson, C. G.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 1998-11-01
- Volume
- 10
- Issue
- 8
- Pages
- 2047-2084
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2167804690
- Cited by
- 517
- References
- 85
- FWCI
- 15.75
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.