The DOI matches its DataCite record; 50 of the 55 references it cites were found in a registry or an index.
Neural Ordinary Differential Equations
Open accessAbstract
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Compared to well-known deep learning models like ResNet and U-Net, neural ODE models are more efficient in terms of memory and parameters.”
From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [163]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, the dynamics of optimization are naturally continuous.”
From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [163]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When the score function is approximated by the time-dependent score-based model, which is typically a neural network, this is an example of a neural ODE (Chen et al., 2018).”
From Score-Based Generative Modeling through Stochastic Differential Equations (2020) · cites this paper as Chen et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Neural ODE models have the advantage of adaptive computing, making them potentially appropriate for application in medical applications.”
From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [163]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
(2017) · Cited by 11,625
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Theory of Ordinary Differential Equations
Coddington, E. A., Levinson, N., & Teichmann, T. (1956) · Physics Today · doi:10.1063/1.3059875 · Cited by 6,221
The Mathematical Theory of Optimal Processes
Bellman, R., Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V., Mishchenko, E. F., Trirogoff, Κ. N., & Neustadt, L. W. (1965) · Econometrica · doi:10.2307/1911909 · Cited by 4,212
CasADi: a software framework for nonlinear optimization and optimal control
Andersson, J. A. E., Gillis, J., Horn, G., Rawlings, J. B., & Diehl, M. (2018) · Mathematical Programming Computation · doi:10.1007/s12532-018-0139-4 · Cited by 4,200
Solving ordinary differential equations I. nonstiff problems
Hairer, E., Nørsett, S. P., & Wanner, G. (1987) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(87)90083-8 · Cited by 4,077
Research works printed but not linked: 8
- Brandon Amos and J Zico Kolter. OptNet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning , pages 136-145, 2017.
- David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016. · arXiv
- Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Proceedings of the 1st Machine Learning for Healthcare Conference , volume 56 of Proceedings of Machine Learning Research , pages 253270. PMLR, 18-19 Aug 2016. URL http://proceedings.mlr.press/v56/Lipton16.html .
- Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. arXiv preprint arXiv:1710.10121 , 2017. · arXiv
- Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Autograd: Reverse-mode differentiation of native Python. In ICML workshop on Automatic Machine Learning , 2015.
- Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770 , 2015. · arXiv
- Hossein Soleimani, James Hensman, and Suchi Saria. Scalable joint models for reliable uncertaintyaware event prediction. IEEE transactions on pattern analysis and machine intelligence , 2017a.
- Jos Stam. Stable fluids. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques , pages 121-128. ACM Press/Addison-Wesley Publishing Co., 1999.
Cited by
Sorted by citations
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
Sherstinsky, A. (2020) · Physica D: Nonlinear Phenomena · doi:10.1016/j.physd.2019.132306 · Cited by 5,680
Denoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
Score-Based Generative Modeling through Stochastic Differential Equations
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020) · arXiv · doi:10.48550/arxiv.2011.13456 · Cited by 1,953
Metadata
- Authors
- Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D.
- Affiliations
- University of Toronto (CA)
- ORCID
- Rubanova, Yulia
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 31
- Pages
- 6572-6583
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1806.07366
- OpenAlex
- W2808746463
- Cited by
- 2,097
- References
- 55
- Citations by year
- 2026: 38 · 2025: 370 · 2024: 509 · 2023: 387 · 2022: 258 · 2021: 297 · 2020: 153 · 2019: 76 · 2018: 8
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Model Reduction and Neural Networks
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Mathematics › Numerical Analysis › Numerical Solvers for Ordinary Differential Equations
- Keywords
- neural ordinary differential equations, continuous-time neural networks, continuous normalizing flows, ResNet, memory-efficient neural networks, generative models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
