The DOI matches its DataCite record; 50 of the 55 references it cites were found in a registry or an index.
Neural Ordinary Differential Equations
Open accessAbstract
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Compared to well-known deep learning models like ResNet and U-Net, neural ODE models are more efficient in terms of memory and parameters.”
From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [163]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, the dynamics of optimization are naturally continuous.”
From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [163]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When the score function is approximated by the time-dependent score-based model, which is typically a neural network, this is an example of a neural ODE (Chen et al., 2018).”
From Score-Based Generative Modeling through Stochastic Differential Equations (2020) · cites this paper as Chen et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Neural ODE models have the advantage of adaptive computing, making them potentially appropriate for application in medical applications.”
From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [163]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
(2017) · Cited by 11,625
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Theory of Ordinary Differential Equations
Coddington, E. A., Levinson, N., & Teichmann, T. (1956) · Physics Today · doi:10.1063/1.3059875 · Cited by 6,221
The Mathematical Theory of Optimal Processes
Bellman, R., Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V., Mishchenko, E. F., Trirogoff, Κ. N., & Neustadt, L. W. (1965) · Econometrica · doi:10.2307/1911909 · Cited by 4,212
CasADi: a software framework for nonlinear optimization and optimal control
Andersson, J. A. E., Gillis, J., Horn, G., Rawlings, J. B., & Diehl, M. (2018) · Mathematical Programming Computation · doi:10.1007/s12532-018-0139-4 · Cited by 4,200
Solving ordinary differential equations I. nonstiff problems
Hairer, E., Nørsett, S. P., & Wanner, G. (1987) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(87)90083-8 · Cited by 4,077
Stochastic Backpropagation and Approximate Inference in Deep Generative ModelsPreprint
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 3,921
Recurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370
Automatic Differentiation in Machine Learning: a Survey
Baydin, A. G., Pearlmutter, B. A., Radul, A., & Siskind, J. M. (2015) · Maynooth University ePrints and eTheses Archive (Maynooth University) · Cited by 2,014
NICE: Non-linear Independent Components Estimation
Dinh, L., Krueger, D., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1410.8516 · Cited by 1,493
Hidden physics models: Machine learning of nonlinear partial differential equations
Raissi, M., & Karniadakis, G. E. (2017) · Journal of Computational Physics · doi:10.1016/j.jcp.2017.11.039 · Cited by 1,349
Improving Variational Inference with Inverse Autoregressive FlowPreprint
Kingma, D., Salimans, T., Józefowicz, R., Chen, X., Sutskever, I., & Welling, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04934 · Cited by 1,168
Doctor AI: Predicting Clinical Events via Recurrent Neural Networks
Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F., & Sun, J. (2015) · Europe PMC (PubMed Central) · doi:10.48550/arxiv.1511.05942 · Cited by 873
Ueber die numerische Aufl�sung von Differentialgleichungen
Runge, C. (1895) · Mathematische Annalen · doi:10.1007/bf01446807 · Cited by 728
Stable architectures for deep neural networks
Haber, E., & Ruthotto, L. (2017) · Inverse Problems · doi:10.1088/1361-6420/aa9a90 · Cited by 689
Recurrent Marked Temporal Point Processes
Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez, M., & Song, L. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939875 · Cited by 614
A theoretical framework for back-propagation
LeCun, Y. (1988) · Cited by 583
Gradient calculations for dynamic recurrent neural networks: a survey
Pearlmutter, B. A. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.410363 · Cited by 578
Deep Neural Networks Motivated by Partial Differential Equations
Ruthotto, L., & Haber, E. (2019) · Journal of Mathematical Imaging and Vision · doi:10.1007/s10851-019-00903-1 · Cited by 514
PDE-Net: Learning PDEs from DataPreprint
Long, Z., Lu, Y., Ma, X., & Dong, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.09668 · Cited by 507
The Reversible Residual Network: Backpropagation Without Storing ActivationsPreprint
Gomez, A. N., Ren, M., Urtasun, R., & Grosse, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04585 · Cited by 415
The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point ProcessPreprint
Mei, H., & Eisner, J. M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.09328 · Cited by 386
Beitrag zur Naherungsweisen Integration Totaler Differentialgleichungen
Kutta, W. (1901) · Cited by 380
Spatially Adaptive Computation Time for Residual Networks
Figurnov, M., Collins, M. D., Zhu, Y., Zhang, L., Huang, J., Vetrov, D., & Salakhutdinov, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.194 · Cited by 345
Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations
Raissi, M., Perdikaris, P. G., & Karniadakis, G. E. (2018) · SIAM Journal on Scientific Computing · doi:10.1137/17m1120762 · Cited by 341
Adaptive Computation Time for Recurrent Neural Networks
Graves, A. (2016) · arXiv · doi:10.48550/arxiv.1603.08983 · Cited by 335
Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical SystemsPreprint
Raissi, M., Perdikaris, P. G., & Karniadakis, G. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01236 · Cited by 330
A General-Purpose Software Framework for Dynamic Optimization
Andersson, J. (2013) · Lirias · Cited by 328
Automated Derivation of the Adjoint of High-Level Transient Finite Element Programs
Farrell, P. E., Ham, D. A., Funke, S. W., & Rognes, M. E. (2013) · SIAM Journal on Scientific Computing · doi:10.1137/120873558 · Cited by 303
Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow
Wiewel, S., Becher, M., & Thuerey, N. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13620 · Cited by 281
Reversible Architectures for Arbitrarily Deep Residual Neural Networks
Chang, B., Meng, L., Haber, E., Ruthotto, L., Begert, D., & Holtham, E. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11668 · Cited by 263
Computationally Efficient Convolved Multiple Output Gaussian Processes
Álvarez, M. A., & Lawrence, N. D. (2009) · Apollo (University of Cambridge) · doi:10.5555/1953048.2021048 · Cited by 218
Improving Variational Auto-Encoders using Householder FlowPreprint
Tomczak, J. M., & Welling, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.09630 · Cited by 149
Learning to Detect Sepsis with a Multitask Gaussian Process RNN ClassifierPreprint
Futoma, J., Hariharan, S., & Heller, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04152 · Cited by 124
Multi-level Residual Networks from Dynamical Systems ViewPreprint
Chang, B., Meng, L., Haber, E., Tung, F., & Begert, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10348 · Cited by 116
Sylvester Normalizing Flows for Variational Inference
van den Berg, R., Hasenclever, L., Tomczak, J. M., & Welling, M. (2018) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.1803.05649 · Cited by 113
Probabilistic ODE Solvers with Runge-Kutta MeansPreprint
Schober, M., Duvenaud, D., & Hennig, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2582 · Cited by 75
Self-citationThe Stan Math Library: Reverse-Mode Automatic Differentiation in C++Preprint
Carpenter, B., Hoffman, M. D., Brubaker, M. A., Lee, D. D., Li, P., & Betancourt, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.07164 · Cited by 68
Time-Dependent Representation for Neural Event Sequence PredictionPreprint
Li, Y., Du, N., & Samy Bengio, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.00065 · Cited by 59
FATODE: A Library for Forward, Adjoint, and Tangent Linear Integration of ODEs
Zhang, H., & Sandu, A. (2014) · SIAM Journal on Scientific Computing · doi:10.1137/130912335 · Cited by 55
Variable Computation in Recurrent Neural NetworksPreprint
Jernite, Y., Grave, É., Joulin, A., & Mikolov, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06188 · Cited by 54
Black-box Variational Inference for Stochastic Differential EquationsPreprint
Ryder, T., Golightly, A., McGough, A. S., & Prangle, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03335 · Cited by 31
Optimization and profile calculation of ODE models using second order adjoint sensitivity analysis
Stapor, P. L., Fröhlich, F., & Hasenauer, J. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty230 · Cited by 30
Soleimani, H., Subbaswamy, A., & Saria, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.02038 · Cited by 23
What-If Reasoning with Counterfactual Gaussian ProcessesPreprint
Schulam, P., & Saria, S. (2017) · arXiv (Cornell University) · Cited by 8
Fast derivatives of likelihood functionals for ODE based models using adjoint-state method
Melicher, V., Haber, T., & Vanroose, W. (2017) · Computational Statistics · doi:10.1007/s00180-017-0765-8 · Cited by 8
Research works printed but not linked: 8
- Brandon Amos and J Zico Kolter. OptNet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning , pages 136-145, 2017.
- David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016. · arXiv
- Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Proceedings of the 1st Machine Learning for Healthcare Conference , volume 56 of Proceedings of Machine Learning Research , pages 253270. PMLR, 18-19 Aug 2016. URL http://proceedings.mlr.press/v56/Lipton16.html .
- Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. arXiv preprint arXiv:1710.10121 , 2017. · arXiv
- Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Autograd: Reverse-mode differentiation of native Python. In ICML workshop on Automatic Machine Learning , 2015.
- Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770 , 2015. · arXiv
- Hossein Soleimani, James Hensman, and Suchi Saria. Scalable joint models for reliable uncertaintyaware event prediction. IEEE transactions on pattern analysis and machine intelligence , 2017a.
- Jos Stam. Stable fluids. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques , pages 121-128. ACM Press/Addison-Wesley Publishing Co., 1999.
Cited by
Sorted by citations
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
Sherstinsky, A. (2020) · Physica D: Nonlinear Phenomena · doi:10.1016/j.physd.2019.132306 · Cited by 5,680
Denoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
Score-Based Generative Modeling through Stochastic Differential Equations
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020) · arXiv · doi:10.48550/arxiv.2011.13456 · Cited by 1,953
Metadata
- Authors
- Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D.
- Affiliations
- University of Toronto (CA)
- ORCID
- Rubanova, Yulia
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 31
- Pages
- 6572-6583
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1806.07366
- OpenAlex
- W2808746463
- Cited by
- 2,097
- References
- 55
- Citations by year
- 2026: 38 · 2025: 370 · 2024: 509 · 2023: 387 · 2022: 258 · 2021: 297 · 2020: 153 · 2019: 76 · 2018: 8
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Model Reduction and Neural Networks
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Mathematics › Numerical Analysis › Numerical Solvers for Ordinary Differential Equations
- Keywords
- neural ordinary differential equations, continuous-time neural networks, continuous normalizing flows, ResNet, memory-efficient neural networks, generative models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
