The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction
Open accessAbstract
Leveraging physical knowledge described by partial differential equations (PDEs) is an appealing way to improve unsupervised video forecasting models. Since physics is too restrictive for describing the full visual content of generic video sequences, we introduce PhyDNet, a two-branch deep architecture, which explicitly disentangles PDE dynamics from unknown complementary information. A second contribution is to propose a new recurrent physical cell (PhyCell), inspired from data assimilation techniques, for performing PDE-constrained prediction in latent space. Extensive experiments conducted on four various datasets show the ability of PhyDNet to outperform state-of-the-art methods. Ablation studies also highlight the important gain brought out by both disentanglement and PDE-constrained prediction. Finally, we show that PhyDNet presents interesting features for dealing with missing data and long-term forecasting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
A New Approach to Linear Filtering and Prediction Problems
Kalman, R. E. (1960) · Journal of Basic Engineering · doi:10.1115/1.3662552 · Cited by 31,751
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
A Wavelet Tour of Signal Processing
(1999) · Elsevier eBooks · doi:10.1016/b978-0-12-466606-1.x5000-4 · Cited by 16,227
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
Discovering governing equations from data by sparse identification of nonlinear dynamical systems
Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1517384113 · Cited by 4,796
Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653
Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction
Zhang, J., Zheng, Y., & Qi, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10735 · Cited by 2,282
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861
Data-driven discovery of partial differential equations
Rudy, S. H., Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2017) · Science Advances · doi:10.1126/sciadv.1602614 · Cited by 1,557
PredRNN: recurrent neural networks for predictive learning using spatiotemporal LSTMs
Wang, Y., Long, M., Wang, J., Gao, Z., & Yu, P. S. (2017) · neural information processing systems · Cited by 1,366
MoCoGAN: Decomposing Motion and Content for Video Generation
Tulyakov, S., Liu, M.-Y., Yang, X., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00165 · Cited by 1,076
Interaction Networks for Learning about Objects, Relations and Physics
Battaglia, P. W., Pascanu, R., Lai, M., Rezende, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1612.00222 · Cited by 969
Raissi, M., Perdikaris, P. G., & Karniadakis, G. E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10561 · Cited by 925
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759
Video Frame Synthesis Using Deep Voxel Flow
Liu, Z., Yeh, R. A., Tang, X., Liu, Y., & Agarwala, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.478 · Cited by 743
A Proposal on Machine Learning via Dynamical Systems
E, W. (2017) · Communications in Mathematics and Statistics · doi:10.1007/s40304-017-0103-z · Cited by 607
PDE-Net 2.0: Learning PDEs from data with a numeric-symbolic hybrid deep network
Long, Z., Lu, Y., & Dong, B. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.108925 · Cited by 549
Decomposing Motion and Content for Natural Video Sequence Prediction
Villegas, R., Yang, J., Hong, S., Lin, X., & Lee, H. (2017) · arXiv · doi:10.48550/arxiv.1706.08033 · Cited by 488
Wang, Y., Zhang, J., Zhu, H., Long, M., Wang, J., & Yu, P. S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00937 · Cited by 480
Neural Relational Inference for Interacting Systems
Kipf, T., Fetaya, E., Wang, K.-C. J., Welling, M., & Zemel, R. S. (2018) · UvA-DARE (University of Amsterdam) · Cited by 475
Brabandere, B. D., Jia, X., Tuytelaars, T., & Gool, L. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09673 · Cited by 461
Generating Videos with Scene Dynamics
Vondrick, C., Pirsiavash, H., & Torralba, A. (2016) · Maryland Shared Open Access Repository · doi:10.13016/m26gih-tnyz · Cited by 441
Learning partial differential equations via data discovery and sparse optimization
Schaeffer, H. (2017) · Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rspa.2016.0446 · Cited by 432
Graph networks as learnable physics engines for inference and control
Sanchez-Gonzalez, A., Heess, N., Springenberg, J. T., Merel, J., Riedmiller, M., Hadsell, R., & Battaglia, P. (2018) · arXiv · doi:10.48550/arxiv.1806.01242 · Cited by 387
Eidetic 3D LSTM: A Model for Video Prediction and Beyond
Wang, Y., Jiang, L., Yang, M. H., Li, L.-J., Long, M., & Fei-Fei, L. (2019) · International Conference on Learning Representations · Cited by 372
Learning to Decompose and Disentangle Representations for Video PredictionPreprint
Hsieh, J.-T., Liu, B., Huang, D.-A., Fei-Fei, L., & Niebles, J. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04166 · Cited by 354
The Pose Knows: Video Forecasting by Generating Pose Futures
Walker, J., Marino, K., Gupta, A., & Hebert, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.361 · Cited by 330
Dual Motion GAN for Future-Flow Embedded Video Prediction
Liang, X., Lee, L., Dai, W., & Xing, E. P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.194 · Cited by 328
PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
Wang, Y., Gao, Z., Long, M., Wang, J., & Yu, P. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.06300 · Cited by 324
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
Xue, T., Wu, J., Bouman, K. L., & Freeman, W. T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02586 · Cited by 320
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their InteractionsPreprint
van Steenkiste, S., Chang, M., Greff, K., & Schmidhuber, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10353 · Cited by 293
Data driven governing equations approximation using deep neural networks
Qin, T., Wu, K., & Xiu, D. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.06.042 · Cited by 284
Visual Interaction NetworksPreprint
Watters, N., Tacchetti, A., Weber, T., Pascanu, R., Battaglia, P., & Zoran, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01433 · Cited by 283
Sequential Attend, Infer, Repeat: Generative Modelling of Moving ObjectsPreprint
Kosiorek, A. R., Kim, H., Posner, I., & Teh, Y. W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01794 · Cited by 272
Flexible Neural Representation for Physics PredictionPreprint
Mrowca, D., Zhuang, C., Wang, E., Haber, N., Fei-Fei, L., Tenenbaum, J. B., & Yamins, D. L. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.08047 · Cited by 267
A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
Fraccaro, M., Kamronn, S., Paquet, U., & Winther, O. (2017) · arXiv · doi:10.48550/arxiv.1710.05741 · Cited by 229
Spatio-temporal video autoencoder with differentiable memory
Pătrăucean, V., Handa, A., & Cipolla, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06309 · Cited by 211
Addressing Failure Prediction by Learning Model Confidence
Corbière, C., Thome, N., Bar-Hen, A., Cord, M., & Pérez, P. (2019) · arXiv · doi:10.48550/arxiv.1910.04851 · Cited by 211
Self-citationKrishnan, R. G., Shalit, U., & Sontag, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05121 · Cited by 206
Unsupervised Learning of Disentangled Representations from Video
Denton, E., & Birodkar, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10915 · Cited by 205
Improved Conditional VRNNs for Video Prediction
Castrejón, L., Ballas, N., & Courville, A. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00770 · Cited by 200
Unsupervised Learning of Long-Term Motion Dynamics for Videos
Luo, Z., Peng, B., Huang, D.-A., Alahi, A., & Fei-Fei, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.751 · Cited by 189
Predicting Future Frames Using Retrospective Cycle GAN
Kwon, Y.-H., & Park, M.-G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00191 · Cited by 184
Folded Recurrent Neural Networks for Future Video Prediction
Oliu, M., Selva, J., & Escalera, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_44 · Cited by 173
ContextVP: Fully Context-Aware Video Prediction
Byeon, W., Wang, Q., Srivastava, R. K., & Koumoutsakos, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01270-0_46 · Cited by 168
Backprop KF: Learning Discriminative Deterministic State EstimatorsPreprint
Haarnoja, T., Ajay, A., Levine, S., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07148 · Cited by 167
Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces
Becker, P., Pandya, H., Gebhardt, G. H. W., Zhao, C., Taylor, C. J., & Neumann, G. M. (2019) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.48550/arxiv.1905.07357 · Cited by 159
Neural Jump Stochastic Differential EquationsPreprint
Jia, J., & Benson, A. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10403 · Cited by 151
SDC-Net: Video Prediction Using Spatially-Displaced Convolution
Reda, F. A., Liu, G., Shih, K. J., Kirby, R. M., Barker, J., Tarjan, D., Tao, A., & Catanzaro, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_44 · Cited by 142
Flow-Grounded Spatial-Temporal Video Prediction from Still Images
Li, Y., Chen, F., Yang, J., Wang, Z., Lu, X., & Yang, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01240-3_37 · Cited by 139
Disentangling Propagation and Generation for Video Prediction
Gao, H., Xu, H., Cai, Q.-Z., Wang, R., Yu, F., & Darrell, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00910 · Cited by 121
(2018) · Lecture notes in computer science · doi:10.1007/978-3-030-04167-0 · Cited by 118
Bocquet, M., Brajard, J., Carrassi, A., & Bertino, L. (2019) · Nonlinear processes in geophysics · doi:10.5194/npg-26-143-2019 · Cited by 115
Learning to See Physics via Visual De-animation
Wu, J., Lu, E., Kohli, P., Freeman, B., & Tenenbaum, J. B. (2017) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 113
Shape and Time Distortion Loss for Training Deep Time Series Forecasting ModelsPreprint
Guen, V. L., & Thome, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09020 · Cited by 113
Self-citationCompositional Video Prediction
Ye, Y., Singh, M., Gupta, A., & Tulsiani, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01045 · Cited by 89
Image Restoration: Wavelet Frame Shrinkage, Nonlinear Evolution PDEs, and Beyond
Dong, B., Jiang, Q., & Shen, Z. (2017) · Multiscale Modeling and Simulation · doi:10.1137/15m1037457 · Cited by 88
Bilinear Residual Neural Network for the Identification and Forecasting of Geophysical Dynamics
Fablet, R., Ouala, S., & Herzet, C. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553492 · Cited by 79
Convolutional neural networks combined with Runge–Kutta methods
Zhu, M., Chang, B., & Fu, C. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07785-2 · Cited by 75
Differentiable Physics-informed Graph NetworksPreprint
Seo, S., & Liu, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02950 · Cited by 67
Structure Preserving Video Prediction
Xu, J., Ni, B., Li, Z., Cheng, S., & Yang, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00158 · Cited by 64
HybridNet: Classification and Reconstruction Cooperation for Semi-supervised Learning
Robert, T., Thome, N., & Cord, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_10 · Cited by 56
Self-citation
Cited by
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
Wang, Y., Wu, H., Zhang, J., Gao, Z., Wang, J., Yu, P. S., & Long, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3165153 · Cited by 677
Can deep learning beat numerical weather prediction?
Schultz, M. G., Betancourt, C., Gong, B., Kleinert, F., Langguth, M., Leufen, L. H., Mozaffari, A., & Stadtler, S. (2021) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0097 · Cited by 562
SimVP: Simpler yet Better Video Prediction
Gao, Z., Tan, C., Wu, L., & Li, S. Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00317 · Cited by 403
VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting
Tang, Y., Dong, P., Tang, Z., Chu, X., & Liang, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00575 · Cited by 70
CMS-LSTM: Context Embedding and Multi-Scale Spatiotemporal Expression LSTM for Predictive Learning
Chai, Z., Xu, Z., Bail, Y., Lin, Z., & Yuan, C. (2022) · 2022 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859659 · Cited by 24
Metadata
- Authors
- Le Guen, V., & Thome, N.
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 11471-11481
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3009163129
- Cited by
- 306
- References
- 64
- FWCI
- 49.45
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.