The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Natural Gradient Works Efficiently in Learning
Abstract
When a parameter space has a certain underlying structure, the ordinary gradient of a function does not represent its steepest direction, but the natural gradient does. Information geometry is used for calculating the natural gradients in the parameter space of perceptrons, the space of matrices (for blind source separation), and the space of linear dynamical systems (for blind source deconvolution). The dynamical behavior of natural gradient online learning is analyzed and is proved to be Fisher efficient, implying that it has asymptotically the same performance as the optimal batch estimation of parameters. This suggests that the plateau phenomenon, which appears in the backpropagation learning algorithm of multilayer perceptrons, might disappear or might not be so serious when the natural gradient is used. An adaptive method of updating the learning rate is proposed and analyzed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Information-Maximization Approach to Blind Separation and Blind Deconvolution
Bell, A. J., & Sejnowski, T. J. (1995) · Neural Computation · doi:10.1162/neco.1995.7.6.1129 · Cited by 8,986
Differential-Geometrical Methods in Statistics
Amari, S.-I. (1985) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-5056-2 · Cited by 1,673
Research works printed but not linked: 21
- Amari S., 1967, IEEE Trans., EC-16(3), 299
- 10.1007/BF00365229 · DOI · Paper page
- 10.1007/BF01692059 · DOI
- 10.1016/0893-6080(93)90013-M · DOI
- 10.1090/conm/203/02554 · doi:10.1090/conm/203/02554
- Amari S., IEEE Trans. on Signal Processing.
- 10.2307/3318651 · doi:10.2307/3318651
- 10.1109/72.125867 · DOI
- 10.1162/neco.1993.5.1.140 · DOI
- 10.1103/PhysRevLett.75.1415 · doi:10.1103/physrevlett.75.1415
- 10.1016/0020-0255(85)90050-7 · DOI
- 10.1109/78.553476 · DOI · Paper page
- 10.1016/0165-1684(94)90029-9 · DOI · Paper page
- 10.1049/el:19961414 · doi:10.1049/el:19961414
- 10.1103/PhysRevA.44.2718 · DOI
- 10.1016/0165-1684(91)90079-X · DOI · Paper page
- 10.1088/0954-898X/5/4/008 · doi:10.1088/0954-898x/5/4/008 · Paper page
- 10.1103/PhysRevLett.77.4671 · doi:10.1103/physrevlett.77.4671
- Rao C. R., 1945, Bulletin of the Calcutta Mathematical Society, 37, 81
- 10.1103/PhysRevE.52.4225 · DOI
- 10.1103/PhysRevLett.76.2188 · doi:10.1103/physrevlett.76.2188
Cited by
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Lee, T.-W., Girolami, M., & Sejnowski, T. J. (1999) · Neural Computation · doi:10.1162/089976699300016719 · Cited by 1,967
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259
(2003) · Cited by 1,733
Noisy intermediate-scale quantum algorithms
Bharti, K., Cervera-Lierta, A., Kyaw, T. H., Haug, T., Alperin-Lea, S., Anand, A., Degroote, M., Heimonen, H., Kottmann, J. S., Menke, T., et al. (2022) · Reviews of Modern Physics · doi:10.1103/revmodphys.94.015004 · Cited by 2,231
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
Chaudhry, A., Dokania, P. K., Ajanthan, T., & Torr, P. H. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_33 · Cited by 1,091
The Variational Quantum Eigensolver: A review of methods and best practices
Tilly, J., Chen, H., Cao, S., Picozzi, D., Setia, K., Li, Y., Grant, E., Wossnig, L., Rungger, I., Booth, G. H., et al. (2022) · Physics Reports · doi:10.1016/j.physrep.2022.08.003 · Cited by 1,031
The power of quantum neural networks
Abbas, A., Sutter, D., Zoufal, C., Lucchi, A., Figalli, A., & Woerner, S. (2021) · Nature Computational Science · doi:10.1038/s43588-021-00084-1 · Cited by 1,082
Reinforcement learning of motor skills with policy gradients
Peters, J., & Schaal, S. (2008) · Neural Networks · doi:10.1016/j.neunet.2008.02.003 · Cited by 831
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., & Levine, S. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.049 · Cited by 847
Peters, J., & Schaal, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.026 · Cited by 738
Stokes, J., Izaac, J., Killoran, N., & Carleo, G. (2020) · Quantum · doi:10.22331/q-2020-05-25-269 · Cited by 473
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., & Peste, A. (2021) · ETH Zurich · doi:10.3929/ethz-b-000519202 · Cited by 423
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
Zhuang, J., Tang, T., Ding, Y., Tatikonda, S., Dvornek, N., Papademetris, X., & Duncan, J. S. (2020) · arXiv · doi:10.48550/arxiv.2010.07468 · Cited by 382
Smigielski, L., Scheidegger, M., Kometer, M., & Vollenweider, F. X. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.04.009 · Cited by 248
A free energy principle for a particular physics
Friston, K. (2019) · arXiv · doi:10.48550/arxiv.1906.10184 · Cited by 222
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., Yu, F., Wang, W., & Gu, S. (2021) · arXiv · doi:10.48550/arxiv.2102.05426 · Cited by 190
Sentience and the Origins of Consciousness: From Cartesian Duality to Markovian Monism
Friston, K. J., Wiese, W., & Hobson, J. A. (2020) · Entropy · doi:10.3390/e22050516 · Cited by 186
The free energy principle made simpler but not too simple
Friston, K., Da Costa, L., Sajid, N., Heins, C., Ueltzhöffer, K., Pavliotis, G. A., & Parr, T. (2023) · Physics Reports · doi:10.1016/j.physrep.2023.07.001 · Cited by 163
“The second (dissipative) part performs a (natural) gradient descent on the steadystate surprisal and depends upon the amplitude of random fluctuations [33, 34].”
Avoiding local minima in variational quantum eigensolvers with the natural gradient optimizer
Wierichs, D., Gogolin, C., & Kastoryano, M. (2020) · Physical Review Research · doi:10.1103/physrevresearch.2.043246 · Cited by 172
Contraction theory for nonlinear stability analysis and learning-based control: A tutorial overview
Tsukamoto, H., Chung, S.-J., & Slotine, J.-J. E. (2021) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2021.10.001 · Cited by 145
NetKet 3: Machine Learning Toolbox for Many-Body Quantum Systems
Vicentini, F., Hofmann, D., Szabó, A., Wu, D., Roth, C., Giuliani, C., Pescia, G., Nys, J., Vargas-Calderón, V., Astrakhantsev, N., et al. (2022) · SciPost Physics Codebases · doi:10.21468/scipostphyscodeb.7 · Cited by 135
Beyond-classical computation in quantum simulation
King, A. D., Nocera, A., Rams, M. M., Dziarmaga, J., Wiersema, R., Bernoudy, W., Raymond, J., Kaushal, N., Heinsdorf, N., Harris, R., et al. (2025) · Science · doi:10.1126/science.ado6285 · Cited by 138
Capacity and Quantum Geometry of Parametrized Quantum Circuits
Haug, T., Bharti, K., & Kim, M. S. (2021) · PRX Quantum · doi:10.1103/prxquantum.2.040309 · Cited by 127
Harrow, A. W., & Napp, J. C. (2021) · Physical Review Letters · doi:10.1103/physrevlett.126.140502 · Cited by 121
Ossadtchi, A., Baillet, S., Mosher, J. C., Thyerlei, D., Sutherling, W., & Leahy, R. M. (2004) · Clinical Neurophysiology · doi:10.1016/j.clinph.2003.10.036 · Cited by 104
Adaptive Deep Models for Incremental Learning: Considering Capacity Scalability and Sustainability
Yang, Y., Zhou, D.-W., Zhan, D.-C., Xiong, H., & Jiang, Y. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330865 · Cited by 83
Classical variational simulation of the Quantum Approximate Optimization Algorithm
Medvidović, M., & Carleo, G. (2021) · npj Quantum Information · doi:10.1038/s41534-021-00440-z · Cited by 79
Variational benchmarks for quantum many-body problems
Wu, D., Rossi, R., Vicentini, F., Astrakhantsev, N., Becca, F., Cao, X., Carrasquilla, J., Ferrari, F., Georges, A., Hibat-Allah, M., et al. (2024) · Science · doi:10.1126/science.adg9774 · Cited by 79
Unbiasing time-dependent Variational Monte Carlo by projected quantum evolution
Sinibaldi, A., Giuliani, C., Carleo, G., & Vicentini, F. (2023) · Quantum · doi:10.22331/q-2023-10-10-1131 · Cited by 51
Noise-resilient variational hybrid quantum-classical optimization
Gentini, L., Cuccoli, A., Pirandola, S., Verrucchi, P., & Banchi, L. (2020) · Physical Review A · doi:10.1103/physreva.102.052414 · Cited by 46
Human primary motor cortex shows hemispheric specialization for speech
Terumitsu, M., Fujii, Y., Suzuki, K., Kwee, I. L., & Nakada, T. (2006) · Neuroreport · doi:10.1097/01.wnr.0000224778.97399.c4 · Cited by 45
On the natural gradient for variational quantum eigensolver
Yamamoto, N. (2019) · arXiv · doi:10.48550/arxiv.1909.05074 · Cited by 39
Neural dynamics under active inference: plausibility and efficiency of information processing
Da Costa, L., Parr, T., Sengupta, B., & Friston, K. (2021) · Entropy · doi:10.3390/e23040454 · Cited by 38
Being Bayesian about Categorical Probability
Joo, T., Chung, U., & Seo, M.-G. (2020) · arXiv · doi:10.48550/arxiv.2002.07965 · Cited by 37
Direct sampling of projected entangled-pair states
Vieijra, T., Haegeman, J., Verstraete, F., & Vanderstraeten, L. (2021) · Physical Review B · doi:10.1103/physrevb.104.235141 · Cited by 37
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 27
“where D is the divergence measure between two distributions; usually, KL-divergence is chosen: KL[πθ′ ||πθ ] ≈ (θ′ − θ)F(θ′ − θ), where F is the Fisher-information matrix [133]: F = E πθ [ ∇θ log πθ(at,i|st,i)∇θ log πθ(at,i|st,i) ] .”
Generalization Guarantees for Imitation Learning
Ren, A. Z., Veer, S., & Majumdar, A. (2020) · arXiv · doi:10.48550/arxiv.2008.01913 · Cited by 17
Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory
Suzgun, M., Yuksekgonul, M., Bianchi, F., Jurafsky, D., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.07952
Elastic Weight Removal for Faithful and Abstractive Dialogue Generation
Daheim, N., Dziri, N., Sachan, M., Gurevych, I., & Ponti, E. M. (2023) · arXiv · doi:10.48550/arxiv.2303.17574
LOGAN: Latent Optimisation for Generative Adversarial Networks
Wu, Y., Donahue, J., Balduzzi, D., Simonyan, K., & Lillicrap, T. (2019) · arXiv · doi:10.48550/arxiv.1912.00953
Metadata
- Authors
- Amari, S.-I.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 1998-02-01
- Volume
- 10
- Issue
- 2
- Pages
- 251-276
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W1970789124
- Cited by
- 2,681
- References
- 2
- FWCI
- 61.26
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
