The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
A review of automatic differentiation and its efficient implementation
Open accessAbstract
Derivatives play a critical role in computational statistics, examples being Bayesian inference using Hamiltonian Monte Carlo sampling and the training of neural networks. Automatic differentiation (AD) is a powerful tool to automate the calculation of derivatives and is preferable to more traditional methods, especially when differentiating complex algorithms and mathematical functions. The implementation of AD, however, requires some care to insure efficiency. Modern differentiation packages deploy a broad range of computational techniques to improve applicability, run time, and memory management. Among these techniques are operation overloading, region‐based memory, and expression templates. There also exist several mathematical techniques which can yield high performance gains when applied to complex algorithms. For example, semi‐analytical derivatives can reduce by orders of magnitude the runtime required to numerically solve and differentiate an algebraic equation. Open and practical problems include the extension of current packages to provide more specialized routines, and finding optimal methods to perform higher‐order differentiation. This article is categorized under: Algorithmic Development > Scalable Statistical Methods
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Stan : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · cited 2×
“NUTS, supported by AD, indeed provides the algorithmic foundation for the probabilistic languages Stan (Carpenter et al., 2017) and PyMC3 (Salvatier, Wiecki, & Fonnesbeck, 2016).”
BackgroundCasADi: a software framework for nonlinear optimization and optimal control
Andersson, J. A. E., Gillis, J., Horn, G., Rawlings, J. B., & Diehl, M. (2019) · Mathematical Programming Computation · doi:10.1007/s12532-018-0139-4 · Cited by 3,868
The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte CarloPreprint
Homan, M. D., & Gelman, A. (2014) · arXiv (Cornell University) · Cited by 3,405
MCMC Using Hamiltonian Dynamics
Neal, R. M. (2011) · Handbook of Markov Chain Monte Carlo · doi:10.1201/b10905-6 · Cited by 2,574
Probabilistic programming in Python using PyMC3
Salvatier, J., Wiecki, T. V., & Fonnesbeck, C. (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.55 · Cited by 2,544
Nineteen Dubious Ways to Compute the Exponential of a Matrix, Twenty-Five Years Later
Moler, C., & Van Loan, C. (2003) · SIAM Review · doi:10.1137/s00361445024180 · Cited by 2,357
30 years of adaptive neural networks: perceptron, Madaline, and backpropagation
Widrow, B., & Lehr, M. A. (1990) · Proceedings of the IEEE · doi:10.1109/5.58323 · Cited by 2,242
Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation
Griewank, A., & Walther, A. (1987) · Cited by 2,161
The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo
Hoffman, M. D., & Gelman, A. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1111.4246 · Cited by 1,713
Numerical methods for nonlinear algebraic equations
Rabinowitz, P. (1970) · Gordon and Breach Science Publishers eBooks · Cited by 900
Automatic differentiation in machine learning: a survey
Baydin, A. G., Pearlmutter, B. A., Radul, A., & Siskind, J. M. (2015) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2019.25 · Cited by 823
Fast Exact Multiplication by the Hessian
Pearlmutter, B. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.1.147 · Cited by 540
Automatic differentiation variational inference
Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., & Blei, D. M. (2016) · arXiv · doi:10.48550/arxiv.1603.00788 · Cited by 515
The Tapenade automatic differentiation tool: Principles, model, and specification
Hascoet, L., & Pascual, V. (2013) · ACM Transactions on Mathematical Software · doi:10.1145/2450153.2450158 · Cited by 498
Adifor 2.0: automatic differentiation of Fortran 77 programs
Bischof, C., Khademi, P., Mauer, A., & Carle, A. (1996) · IEEE Computational Science and Engineering · doi:10.1109/99.537089 · Cited by 366
Veldhuizen, T. L. (1996) · Cited by 366
Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation
Griewank, A. (1992) · Optimization Methods and Software · doi:10.1080/10556789208805505 · Cited by 360
Walther, A., & Griewank, A. (2012) · Chapman & Hall/CRC computational science series/Chapman & Hall/CRC computational science · doi:10.1201/b11644-8 · Cited by 228
Automatic Differentiation: Applications, Theory, and Implementations
(2006) · Lecture Notes in Computational Science and Engineering · doi:10.1007/3-540-28438-9 · Cited by 182
Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation
Giles, M. B. (2008) · Lecture notes in computational science and engineering · doi:10.1007/978-3-540-68942-3_4 · Cited by 118
A General Metric for Riemannian Manifold Hamiltonian Monte Carlo
Betancourt, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40020-9_35 · Cited by 88
Extending R with C++ : A Brief Introduction to RcppPreprint
Eddelbüttel, D., & Balamuta, J. (2017) · doi:10.7287/peerj.preprints.3188v1 · Cited by 74
The Stan Math Library: Reverse-Mode Automatic Differentiation in C++Preprint
Carpenter, B., Hoffman, M. D., Brubaker, M. A., Lee, D. D., Li, P., & Betancourt, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.07164 · Cited by 68
Algorithmic Differentiation of Implicit Functions and Optimal Values
Bell, B. M., & Burke, J. V. (2008) · Lecture notes in computational science and engineering · doi:10.1007/978-3-540-68942-3_7 · Cited by 56
Gay, D. E., & Aiken, A. (2001) · ACM SIGPLAN Notices · doi:10.1145/381694.378815 · Cited by 14
Cited by
DeepXDE: A Deep Learning Library for Solving Differential Equations
Lu, L., Meng, X., Mao, Z., & Karniadakis, G. E. (2021) · SIAM Review · doi:10.1137/19m1274067 · Cited by 2,512
Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
Yu, J., Lu, L., Meng, X., & Karniadakis, G. E. (2022) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2022.114823 · Cited by 722
DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting
Heiden, E., Macklin, M., Narang, Y., Fox, D., Garg, A., & Ramos, F. (2021) · Robotics: Science and Systems XVII · doi:10.15607/rss.2021.xvii.067 · Cited by 66
A computational toolbox for the assembly yield of complex and heterogeneous structures
Curatolo, A. I., Kimchi, O., Goodrich, C. P., Krueger, R. K., & Brenner, M. P. (2023) · Nature Communications · doi:10.1038/s41467-023-43168-4 · Cited by 15
Metadata
- Authors
- Margossian, C. C.
- Venue
- WIREs Data Mining and Knowledge Discovery
- Publisher
- Wiley
- Published
- 2019-03-20
- Volume
- 9
- Issue
- 4
- ISSN
- 1942-4787 (ISSN-L)
- 1942-4795
- OpenAlex
- W2901943374
- Cited by
- 321
- References
- 27
- FWCI
- 14.73
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.