The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
NONLINEAR GATED EXPERTS FOR TIME SERIES: DISCOVERING REGIMES AND AVOIDING OVERFITTING
Abstract
In the analysis and prediction of real-world systems, two of the key problems are nonstationarity (often in the form of switching between regimes), and overfitting (particularly serious for noisy processes). This article addresses these problems using gated experts, consisting of a (nonlinear) gating network, and several (also nonlinear) competing experts. Each expert learns to predict the conditional mean, and each expert adapts its width to match the noise level in its regime. The gating network learns to predict the probability of each expert, given the input. This article focuses on the case where the gating network bases its decision on information from the inputs. This can be contrasted to hidden Markov models where the decision is based on the previous state(s) (i.e. on the output of the gating network at the previous time step), as well as to averaging over several predictors. In contrast, gated experts soft-partition the input space, only learning to model their region. This article discusses the underlying statistical assumptions, derives the weight update rules, and compares the performance of gated experts to standard methods on three time series: (1) a computer-generated series, obtained by randomly switching between two nonlinear processes; (2) a time series from the Santa Fe Time Series Competition (the light intensity of a laser in chaotic state); and (3) the daily electricity demand of France, a real-world multivariate problem with structure on several time scales. The main results are: (1) the gating network correctly discovers the different regimes of the process; (2) the widths associated with each expert are important for the segmentation task (and they can be used to characterize the sub-processes); and (3) there is less overfitting compared to single networks (homogeneous multilayer perceptrons), since the experts learn to match their variances to the (local) noise levels. This can be viewed as matching the local complexity of the model to the local complexity of the data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743 · cited 1×
“If this mapping is linear (i.e., described by a transition matrix between these hidden states), this architecture is identical to a hidden Markov model (HMM) [Poritz, 1988, Rabiner, 1989, Baldi and Chauvin, 1994, Fraser and Dimitriadis, 1994, Ivanova et al., 1994, Nadas and Mercer, 1996].”
(2005) · Statistics for Biology and Health · doi:10.1007/0-387-27255-0_9 · Cited by 20,133
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
Breitung, J., & Hamilton, J. D. (1995) · Contemporary Sociology · doi:10.2307/2076916 · Cited by 10,966
Multivariate Adaptive Regression Splines
Friedman, J. H. (1991) · The Annals of Statistics · doi:10.1214/aos/1176347963 · Cited by 8,000
Bollerslev, T., Chou, R. Y., & Kroner, K. F. (1992) · Journal of Econometrics · doi:10.1016/0304-4076(92)90064-x · Cited by 4,270
Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks
Broomhead, D. S., & Lowe, D. (1988) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 3,443
Networks for approximation and learning
Poggio, T., & Girosi, F. (1990) · Proceedings of the IEEE · doi:10.1109/5.58326 · Cited by 3,252
Mixture Densities, Maximum Likelihood and the EM Algorithm
Redner, R. A., & Walker, H. F. (1984) · SIAM Review · doi:10.1137/1026034 · Cited by 2,693
Hierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.181 · Cited by 2,673
Engle, R. F. (1995) · Arch · doi:10.1093/oso/9780198774310.003.0001 · Cited by 1,960
Analysis of time series subject to changes in regime
Hamilton, J. D. (1990) · Journal of Econometrics · doi:10.1016/0304-4076(90)90093-9 · Cited by 1,926
Distributions in Statistics: Continuous Multivariate Distributions.
Gokhale, D. V., Johnson, N. L., & Kotz, S. (1973) · Biometrics · doi:10.2307/2529696 · Cited by 1,599
III. Contributions to the mathematical theory of evolution
Pearson, K. (1894) · Philosophical Transactions of the Royal Society of London. (A.) · doi:10.1098/rsta.1894.0003 · Cited by 1,550
Threshold Autoregression, Limit Cycles and Cyclical Data
Tong, H., & Lim, K. S. (1980) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1980.tb01126.x · Cited by 1,309
Modelling Nonlinear Economic Relationships
Jansen, D. W., Granger, C. W. J., & Terasvirta, T. (1995) · Southern Economic Journal · doi:10.2307/1060764 · Cited by 1,220
PREDICTING THE FUTURE: A CONNECTIONIST APPROACH
Weigend, A. S., Huberman, B. A., & Rumelhart, D. E. (1990) · International Journal of Neural Systems · doi:10.1142/s0129065790000102 · Cited by 721
Self-citationStatistical mechanics and phase transitions in clustering
Rose, K., Gurewitz, E., & Fox, G. C. (1990) · Physical Review Letters · doi:10.1103/physrevlett.65.945 · Cited by 458
Methods For Combining Experts' Probability Assessments
Jacobs, R. A. (1995) · Neural Computation · doi:10.1162/neco.1995.7.5.867 · Cited by 372
Buntine, W., & Weigend, A. S. (1991) · Complex Systems · Cited by 361
Self-citationBackpropagation: the basic theory
Rumelhart, D. E., Durbin, R. M., Golden, R. M., & Chauvin, Y. (1995) · Cited by 338
Measuring Business Cycles: A Modern Perspective
Diebold, F., & Rudebusch, G. (1994) · National Bureau of Economic Research · doi:10.3386/w4643 · Cited by 337
An Input Output HMM Architecture
Bengio, Y., & Frasconi, P. (1994) · Florence Research (University of Florence) · Cited by 332
Convergence results for the EM approach to mixtures of experts architectures
Jordan, M. I., & Xu, L. (1995) · Neural Networks · doi:10.1016/0893-6080(95)00014-3 · Cited by 305
Hidden Markov models: a guided tour
Poritz, A. B. (2003) · ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1988.196495 · Cited by 235
Neural modeling for time series: A statistical stepwise method for weight elimination
Cottrell, M., Girard, B., Girard, Y., Mangeas, M., & Muller, C. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.471372 · Cited by 209
Self-citationAnnealed Competition of Experts for a Segmentation and Classification of Switching Dynamics
Pawelzik, K. R., Kohlmorgen, J., & Müller, K. (1996) · Neural Computation · doi:10.1162/neco.1996.8.2.340 · Cited by 93
UNSUPERVISED DISCOVERY OF SPEECH SEGMENTS USING RECURRENT NETWORKS
Doutriaux, A., & Zipser, D. (1991) · Elsevier eBooks · doi:10.1016/b978-1-4832-1448-1.50036-6 · Cited by 11
Cited by
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687
“Contextual anomalies have been most commonly explored in time-series data [Weigend et al. 1995; Salvador and Chan 2003] and spatial data [Kou et al.”
BackgroundRokach, L. (2010) · Artificial Intelligence Review · doi:10.1007/s10462-009-9124-7 · Cited by 2,657
Is imitation learning the route to humanoid robots?
Schaal, S. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01327-3 · Cited by 1,314
Rose, K. (1998) · Proceedings of the IEEE · doi:10.1109/5.726788 · Cited by 876
Twenty Years of Mixture of Experts
Yuksel, S. E., Wilson, J. N., & Gader, P. D. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2200299 · Cited by 606
Metadata
- Authors
- WEIGEND, A. S., MANGEAS, M., & SRIVASTAVA, A. N.
- Venue
- International Journal of Neural Systems
- Publisher
- World Scientific Pub Co Pte Lt
- Published
- 1995-12
- Volume
- 06
- Issue
- 04
- Pages
- 373-399
- ISSN
- 0129-0657 (ISSN-L)
- 1793-6462
- OpenAlex
- W2009225465
- Cited by
- 253
- References
- 29
- FWCI
- 7.66
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.