CoChat
Sign Up
PreprintCited by 2,097Model Reduction and Neural NetworksNeural Networks and ApplicationsNumerical methods for differential equations

Neural ordinary differential equations

Open access
Chen, Ricky T. Q., Rubanova, Yulia, Bettencourt, Jesse, Duvenaud, David·2018·arXiv
DOI 10.48550/arxiv.1806.07366arXiv 1806.07366
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.07366

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Compared to well-known deep learning models like ResNet and U-Net, neural ODE models are more efficient in terms of memory and parameters.

    “Compared to well-known deep learning models like ResNet and U-Net, neural ODE models are more efficient in terms of memory and parameters.”

    From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [163]

    Table 1 in Section 3 (p. 3) reports that the ODE-Net used fewer parameters than the tested ResNet (0.22M versus 0.60M) and had constant rather than depth-dependent memory cost. The comparison is specific to those MNIST models; our copy does not provide a memory or parameter comparison with U-Net. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The dynamics of optimization in neural ODE models are naturally continuous.

    “Additionally, the dynamics of optimization are naturally continuous.”

    From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [163]

    The Introduction (p. 1) describes the hidden-state dynamics of neural ODEs as continuous, and Section 2 (p. 2) describes computing training gradients by solving an ODE backward in time. Neither passage says that the dynamics of optimization itself are naturally continuous; the claim shifts the continuity from model states and gradient computation to optimization. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Neural ODE models have the advantage of adaptive computing, making them potentially appropriate for application in medical applications.

    “Neural ODE models have the advantage of adaptive computing, making them potentially appropriate for application in medical applications.”

    From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [163]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

5 of the 55 references couldn’t be matched to a known paper.

Research works printed but not linked: 8

  1. Brandon Amos and J Zico Kolter. OptNet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning , pages 136-145, 2017.
  2. David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016. · arXiv
  3. Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Proceedings of the 1st Machine Learning for Healthcare Conference , volume 56 of Proceedings of Machine Learning Research , pages 253270. PMLR, 18-19 Aug 2016. URL http://proceedings.mlr.press/v56/Lipton16.html .
  4. Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. arXiv preprint arXiv:1710.10121 , 2017. · arXiv
  5. Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Autograd: Reverse-mode differentiation of native Python. In ICML workshop on Automatic Machine Learning , 2015.
  6. Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770 , 2015. · arXiv
  7. Hossein Soleimani, James Hensman, and Suchi Saria. Scalable joint models for reliable uncertaintyaware event prediction. IEEE transactions on pattern analysis and machine intelligence , 2017a.
  8. Jos Stam. Stable fluids. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques , pages 121-128. ACM Press/Addison-Wesley Publishing Co., 1999.

Cited by

Sorted by citations

Related

Authors
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D.
Affiliations
University of Toronto (CA)
ORCID
Rubanova, Yulia
Venue
arXiv
Publisher
arXiv
Published
2018
Volume
31
Pages
6572-6583
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1806.07366
arXiv
1806.07366
OpenAlex
W2808746463
Cited by
2,097
References
55
Citations by year
2026: 38 · 2025: 370 · 2024: 509 · 2023: 387 · 2022: 258 · 2021: 297 · 2020: 153 · 2019: 76 · 2018: 8 · 2016: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Model Reduction and Neural Networks
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Mathematics › Numerical Analysis › Numerical methods for differential equations
Keywords
neural ordinary differential equations, continuous-time neural networks, continuous normalizing flows, ResNet, memory-efficient neural networks, generative models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review