The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
MARS: Motion-Augmented RGB Stream for Action Recognition
Abstract
Most state-of-the-art methods for action recognition consist of a two-stream architecture with 3D convolutions: an appearance stream for RGB frames and a motion stream for optical flow frames. Although combining flow with RGB improves the performance, the cost of computing accurate optical flow is high, and increases action recognition latency. This limits the usage of two-stream approaches in real-world applications requiring low latency. In this paper, we introduce two learning approaches to train a standard 3D CNN, operating on RGB frames, that mimics the motion stream, and as a result avoids flow computation at test time. First, by minimizing a feature-based loss compared to the Flow stream, we show that the network reproduces the motion stream with high fidelity. Second, to leverage both appearance and motion information effectively, we train with a linear combination of the feature-based loss and the standard cross-entropy loss for action recognition. We denote the stream trained using this combined loss as Motion-Augmented RGB Stream (MARS). As a single stream, MARS performs better than RGB or Flow alone, for instance with 72.7% accuracy on Kinetics compared to 72.0% and 65.6% with RGB and Flow streams respectively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Van Gool, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_2 · Cited by 3,958
A Closer Look at Spatiotemporal Convolutions for Action Recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00675 · Cited by 3,516
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.179 · Cited by 3,152
Convolutional Two-Stream Network Fusion for Video Action Recognition
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.213 · Cited by 2,684
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
Sun, D., Yang, X., Liu, M.-Y., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00931 · Cited by 2,648
The Kinetics Human Action Video Dataset
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 2,593
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,570
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
Hara, K., Kataoka, H., & Satoh, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00685 · Cited by 2,205
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658 · cited 5×
“Irrespective of their differences, all the methods discussed above use hand-crafted optical flow features such as [1, 45] for the motion stream, creating a bottleneck for fast and online inference.”
BackgroundMethodInfluentialThe kinetics human action video dataset
Kay, W. (2024) · TIB Data Manager · doi:10.57702/jtuxnt5m · Cited by 1,646
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification
Xie, S., Sun, C., Huang, J., Tu, Z., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_19 · Cited by 1,415
Optical Flow Estimation Using a Spatial Pyramid Network
Ranjan, A., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.291 · Cited by 1,344
Temporal Relational Reasoning in Videos
Zhou, B., Andonian, A., Oliva, A., & Torralba, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_49 · Cited by 1,067
Long-Term Temporal Convolutions for Action Recognition
Varol, G., Laptev, I., & Schmid, C. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2712608 · Cited by 915
Self-citationEpicFlow: Edge-preserving interpolation of correspondences for optical flow
Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298720 · Cited by 697
Self-citationOptical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
Sun, S., Kuang, Z., Sheng, L., Ouyang, W., & Zhang, W. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00151 · Cited by 343
RESOUND: Towards Action Recognition Without Representation Bias
Li, Y., Li, Y., & Vasconcelos, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_32 · Cited by 332
Hidden Two-Stream Convolutional Networks for Action Recognition
Zhu, Y., Lan, Z., Newsam, S., & Hauptmann, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20893-6_23 · Cited by 278
Unifying distillation and privileged information
Lopez-Paz, D., Bottou, L., Schölkopf, B., & Vapnik, V. (2015) · arXiv · doi:10.48550/arxiv.1511.03643 · Cited by 270
Efficient Feature Extraction, Encoding, and Classification for Action Recognition
Kantorov, V., & Laptev, I. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.332 · Cited by 239
End-to-End Learning of Motion Representation for Video Understanding
Fan, L., Huang, W., Gan, C., Ermon, S., Gong, B., & Huang, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00630 · Cited by 230
Learning with Side Information through Modality Hallucination
Hoffman, J., Gupta, S., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.96 · Cited by 207
Modality Distillation with Multiple Stream Networks for Action Recognition
Garcia, N. C., Morerio, P., & Murino, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01237-3_7 · Cited by 194
On the Integration of Optical Flow and Action Recognition
Sevilla-Lara, L., Liao, Y., Güney, F., Jampani, V., Geiger, A., & Black, M. J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-12939-2_20 · Cited by 187
Motion Feature Network: Fixed Motion Filter for Action Recognition
Lee, M., Lee, S., Son, S., Park, G., & Kwak, N. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_24 · Cited by 147
ActionFlowNet: Learning Motion Representation for Action Recognition
Ng, J. Y.-H., Choi, J., Neumann, J., & Davis, L. S. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00179 · Cited by 111
Graph Distillation for Action Detection with Privileged Modalities
Luo, Z., Hsieh, J.-T., Jiang, L., Niebles, J. C., & Fei-Fei, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_11 · Cited by 111
Cited by
Human Action Recognition From Various Data Modalities: A Review
Sun, Z., Ke, Q., Rahmani, H., Bennamoun, M., Wang, G., & Liu, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183112 · Cited by 601
Human activity recognition in artificial intelligence framework: a narrative review
Gupta, N., Gupta, S. K., Pathak, R. K., Jain, V., Rashidi, P., & Suri, J. S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10116-x · Cited by 337
Fine-grained activity classification in assembly based on multi-visual modalities
Chen, H., Zendehdel, N., Leu, M. C., & Yin, Z. (2024) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02152-x · Cited by 47
Metadata
- Authors
- Crasto, N., Weinzaepfel, P., Alahari, K., & Schmid, C.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 7874-7883
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2947084868
- Cited by
- 281
- References
- 49
- FWCI
- 15.60
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.