PreprintCited by 2,097Model Reduction and Neural NetworksNeural Networks and ApplicationsNumerical Solvers for Ordinary Differential Equations

The DOI matches its DataCite record; 50 of the 55 references it cites were found in a registry or an index.

Neural Ordinary Differential Equations

Open access

Abstract

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.07366

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 55 references

5 of the 55 references couldn’t be matched to a known paper.

Research works printed but not linked: 8

  1. Brandon Amos and J Zico Kolter. OptNet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning , pages 136-145, 2017.
  2. David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016. · arXiv
  3. Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Proceedings of the 1st Machine Learning for Healthcare Conference , volume 56 of Proceedings of Machine Learning Research , pages 253270. PMLR, 18-19 Aug 2016. URL http://proceedings.mlr.press/v56/Lipton16.html .
  4. Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. arXiv preprint arXiv:1710.10121 , 2017. · arXiv
  5. Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Autograd: Reverse-mode differentiation of native Python. In ICML workshop on Automatic Machine Learning , 2015.
  6. Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770 , 2015. · arXiv
  7. Hossein Soleimani, James Hensman, and Suchi Saria. Scalable joint models for reliable uncertaintyaware event prediction. IEEE transactions on pattern analysis and machine intelligence , 2017a.
  8. Jos Stam. Stable fluids. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques , pages 121-128. ACM Press/Addison-Wesley Publishing Co., 1999.

Cited by

Sorted by citations

Items 201–300 of 1,007

Metadata

Authors
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D.
Affiliations
University of Toronto (CA)
Venue
arXiv
Publisher
arXiv
Published
2018
Volume
31
Pages
6572-6583
ISSN
2331-8422 (ISSN-L)
OpenAlex
W2808746463
Cited by
2,097
References
55
Citations by year
2026: 38 · 2025: 370 · 2024: 509 · 2023: 387 · 2022: 258 · 2021: 297 · 2020: 153 · 2019: 76 · 2018: 8
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Model Reduction and Neural Networks
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Mathematics › Numerical Analysis › Numerical Solvers for Ordinary Differential Equations
Keywords
neural ordinary differential equations, continuous-time neural networks, continuous normalizing flows, ResNet, memory-efficient neural networks, generative models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review