The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Memory in Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity From Spatiotemporal Dynamics
Open accessAbstract
Natural spatiotemporal processes can be highly non-stationary in many ways, e.g. the low-level non-stationarity such as spatial correlations or temporal dependencies of local pixel values; and the high-level variations such as the accumulation, deformation or dissipation of radar echoes in precipitation forecasting. From Cramer's Decomposition, any non-stationary process can be decomposed into deterministic, time-variant polynomials, plus a zero-mean stochastic term. By applying differencing operations appropriately, we may turn time-variant polynomials into a constant, making the deterministic component predictable. However, most previous recurrent neural networks for spatiotemporal prediction do not use the differential signals effectively, and their relatively simple state transition functions prevent them from learning too complicated variations in spacetime. We propose the Memory In Memory (MIM) networks and corresponding recurrent blocks for this purpose. The MIM blocks exploit the differential signals between adjacent recurrent states to model the non-stationary and approximately stationary properties in spatiotemporal dynamics with two cascaded, self-renewed memory modules. By stacking multiple MIM blocks, we could potentially handle higher-order non-stationarity. The MIM networks achieve the state-of-the-art results on four spatiotemporal prediction tasks across both synthetic and real-world datasets. We believe that the general idea of this work can be potentially applied to other time-series forecasting tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709
Time Series Analysis: Forecasting and Control.
Anderson, O. D., Box, G. E. P., & Jenkins, G. M. (1978) · The Statistician · doi:10.2307/2988198 · Cited by 5,298
Generative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653 · cited 1×
“6M dataset [13] contains human actions of 17 scenarios, including 3.”
BackgroundDeep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction
Zhang, J., Zheng, Y., & Qi, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10735 · Cited by 2,282
Self-citationSpectral Analysis for Physical Applications
Percival, D. B., & Walden, A. T. (1993) · Cambridge University Press eBooks · doi:10.1017/cbo9780511622762 · Cited by 2,161
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
Denton, E., Chintala, S., Szlam, A. D., & Fergus, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.05751 · Cited by 1,541
MoCoGAN: Decomposing Motion and Content for Video Generation
Tulyakov, S., Liu, M.-Y., Yang, X., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00165 · Cited by 1,076
Time Series Analysis Forecasting and Control.
Chakravarti, I. M., Box, G. E. P., & Jenkins, G. M. (1973) · Journal of the American Statistical Association · doi:10.2307/2284112 · Cited by 972
Generating Videos with Scene Dynamics
Vondrick, C., Pirsiavash, H., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.02612 · Cited by 785
PredRNN: recurrent neural networks for predictive learning using spatiotemporal LSTMs
Wang, Y., Long, M., Wang, J., Gao, Z., & Yu, P. S. (2017) · neural information processing systems · Cited by 734
Self-citationDeep Learning for Precipitation Nowcasting: A Benchmark and A New Model
Shi, X., Gao, Z., Lausen, L., Wang, H., Yeung, D., Wong, W. K., & Woo, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.03458 · Cited by 596
Decomposing Motion and Content for Natural Video Sequence Prediction
Villegas, R., Yang, J., Hong, S., Lin, X., & Lee, H. (2017) · arXiv · doi:10.48550/arxiv.1706.08033 · Cited by 488
Brabandere, B. D., Jia, X., Tuytelaars, T., & Gool, L. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09673 · Cited by 461
Generating Videos with Scene Dynamics
Vondrick, C., Pirsiavash, H., & Torralba, A. (2016) · Maryland Shared Open Access Repository · doi:10.13016/m26gih-tnyz · Cited by 441
Generative Adversarial Networks
Salvaris, M., Dean, D., & Tok, W. H. (2018) · Deep Learning with Azure · doi:10.1007/978-1-4842-3679-6_8 · Cited by 375
PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
Wang, Y., Gao, Z., Long, M., Wang, J., & Yu, P. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.06300 · Cited by 324
Self-citationVisual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
Xue, T., Wu, J., Bouman, K. L., & Freeman, W. T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02586 · Cited by 320
Stochastic Video Generation with a Learned Prior
Denton, E., & Fergus, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07687 · Cited by 278
Video (language) modeling: a baseline for generative models of natural videos
Ranzato, M., Szlam, A. D., Bruna, J., Mathieu, M., Collobert, R., & Chopra, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6604 · Cited by 254
Spatio-temporal video autoencoder with differentiable memory
Pătrăucean, V., Handa, A., & Cipolla, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06309 · Cited by 211
Unsupervised Learning of Disentangled Representations from Video
Denton, E., & Birodkar, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10915 · Cited by 205
Stochastic Adversarial Video Prediction
Lee, A. X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.01523 · Cited by 184
Folded Recurrent Neural Networks for Future Video Prediction
Oliu, M., Selva, J., & Escalera, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_44 · Cited by 150
PredCNN: Predictive Learning with Cascade Convolutions
Xu, Z., Wang, Y., Long, M., & Wang, J. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/408 · Cited by 106
Self-citationHierarchical Long-term Video Prediction without SupervisionPreprint
Wichers, N., Villegas, R., Erhan, D., & Lee, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04768 · Cited by 96
Cited by
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
Wang, Y., Wu, H., Zhang, J., Gao, Z., Wang, J., Yu, P. S., & Long, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3165153 · Cited by 677
Self-citationCan deep learning beat numerical weather prediction?
Schultz, M. G., Betancourt, C., Gong, B., Kleinert, F., Langguth, M., Leufen, L. H., Mozaffari, A., & Stadtler, S. (2021) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0097 · Cited by 562
SmaAt-UNet: Precipitation nowcasting using a small attention-UNet architecture
Trebing, K., Staǹczyk, T., & Mehrkanoon, S. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.01.036 · Cited by 509
SimVP: Simpler yet Better Video Prediction
Gao, Z., Tan, C., Wu, L., & Li, S. Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00317 · Cited by 403
Self-Attention ConvLSTM for Spatiotemporal Prediction
Lin, Z., Li, M., Zheng, Z., Cheng, Y., & Yuan, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6819 · Cited by 378
Metadata
- Authors
- Wang, Y., Zhang, J., Zhu, H., Long, M., Wang, J., & Yu, P. S.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 9146-9154
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2967033144
- Cited by
- 480
- References
- 35
- FWCI
- 14.02
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.