The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Balanced Multimodal Learning via On-the-fly Gradient Modulation
Open accessAbstract
Multimodal learning helps to comprehensively understand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multimodal model outperforms its uni-modal counterpart. Specifically, in this paper we point out that existing multimodal discriminative models, in which uniform objective is designed for all modalities, could remain under-optimized uni-modal representations, caused by another dominated modality in some scenarios, e.g., sound in blowing wind event, vision in drawing picture event, etc. To alleviate this optimization imbalance, we propose on-the-fly gradient modulation to adaptively control the optimization of each modality, via monitoring the discrepancy of their contribution towards the learning objective. Further, an extra Gaussian noise that changes dynamically is introduced to avoid possible generalization drop caused by gradient modulation. As a result, we achieve considerable improvement over common fusion methods on different multimodal tasks, and this simple strategy can also boost existing multimodal methods, which illustrates its efficacy and versatility. The source code is available at https://github.com/GeWu-Lab/OGM-GE_CVPR2022.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
SlowFast Networks for Video Recognition
Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00630 · Cited by 4,007
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Van Gool, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_2 · Cited by 3,958
librosa: Audio and Music Signal Analysis in Python
McFee, B., Raffel, C., Liang, D., Ellis, D., McVicar, M., Battenberg, E., & Nieto, O. (2015) · Proceedings of the Python in Science Conference · doi:10.25080/majora-7b98e3ed-003 · Cited by 3,033
The Kinetics Human Action Video Dataset
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 2,593
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490
TSM: Temporal Shift Module for Efficient Video Understanding
Lin, J., Gan, C., & Han, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00718 · Cited by 2,055
Salakhutdinov, R., & Hinton, G. E. (2009) · Cited by 1,890
The kinetics human action video dataset
Kay, W. (2024) · TIB Data Manager · doi:10.57702/jtuxnt5m · Cited by 1,646
CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2336244 · Cited by 914
Arandjelovic, R., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 782
What Makes Training Multi-modal Classification Networks Hard?
Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588
Audio-Visual Event Localization in Unconstrained Videos
Tian, Y., Shi, J., Li, B., Duan, Z., & Xu, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_16 · Cited by 493
Vggsound: A Large-Scale Audio-Visual Dataset
Chen, H., Xie, W., Vedaldi, A., & Zisserman, A. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053174 · Cited by 430
Stochastic Gradient Descent as Approximate Bayesian Inference
Mandt, S., Hoffman, M. D., & Blei, D. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04289 · Cited by 369
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition
Kazakos, E., Nagrani, A., Zisserman, A., & Damen, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00559 · Cited by 302
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
Korbar, B., Tran, D., & Torresani, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00230 · Cited by 299
Self-Supervised Learning by Cross-Modal Audio-Video Clustering
Alwassel, H., Mahajan, D., Korbar, B., Torresani, L., Ghanem, B., & Tran, D. (2019) · arXiv · doi:10.48550/arxiv.1911.12667 · Cited by 298
Audiovisual SlowFast Networks for Video RecognitionPreprint
Xiao, F., Lee, Y. J., Grauman, K. L., Malik, J., & Feichtenhofer, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08740 · Cited by 266
Listen to Look: Action Recognition by Previewing Audio
Gao, R., Oh, T.-H., Grauman, K., & Torresani, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01047 · Cited by 262
Three Factors Influencing Minima in SGD
Jastrzȩbski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y., & Storkey, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04623 · Cited by 217
Chaudhari, P., & Soatto, S. (2018) · 2018 Information Theory and Applications Workshop (ITA) · doi:10.1109/ita.2018.8503224 · Cited by 213
Deep Multimodal Clustering for Unsupervised Audiovisual Learning
Hu, D., Nie, F., & Li, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00947 · Cited by 194
Self-citationSTAR: A Benchmark for Situated Reasoning in Real-World VideosPreprint
Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., & Gan, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.09711 · Cited by 142
Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence
He, F., Liu, T., & Tao, D. (2019) · neural information processing systems · Cited by 132
Positive Sample Propagation along the Audio-Visual Event Line
Zhou, J., Zheng, L., Zhong, Y., Hao, S., & Wang, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00833 · Cited by 131
X-Vectors Meet Emotions: A Study On Dependencies Between Emotion and Speaker Recognition
Pappagari, R. R., Wang, T., Villalba, J., Chen, N., & Dehak, N. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054317 · Cited by 118
Temporal Multimodal Learning in Audiovisual Speech Recognition
Hu, D., Li, X., & Lu, X. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.389 · Cited by 108
Self-citationMultimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition
Ghaleb, E., Popa, M. C., & Asteriadis, S. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925444 · Cited by 107
Xie, Z., He, F., Fu, S., Sato, I., Tao, D., & Sugiyama, M. (2020) · Neural Computation · doi:10.1162/neco_a_01403 · Cited by 75
Learning to Balance the Learning Rates Between Various Modalities via Adaptive Tracking Factor
Ya, S., Mai, S., & Hu, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3101421 · Cited by 67
Improving Multi-Modal Learning with Uni-Modal TeachersPreprint
Du, C., Li, T., Liu, Y., Wen, Z., Hua, T., Wang, Y., & Zhao, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11059 · Cited by 66
Temporal Relational Modeling with Self-Supervision for Action Segmentation
Wang, D., Hu, D., Li, X., & Dou, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16377 · Cited by 55
Self-citationParida, K. K., Matiyali, N., Guha, T., & Sharma, G. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093438 · Cited by 49
Dense Multimodal Fusion for Hierarchically Joint Representation
Hu, D., Wang, C., Nie, F., & Li, X. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683898 · Cited by 47
Self-citationMultimodal Learning and Reasoning for Visual Question Answering
Ilievski, I., & Feng, J. (2017) · neural information processing systems · Cited by 46
Heracleous, P., & Yoneyama, A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0220386 · Cited by 38
On the Noisy Gradient Descent that Generalizes as SGD
Wu, J., Hu, W., Xiong, H., Huan, J., Braverman, V., & Zhu, Z. (2019) · ePrints Soton (University of Southampton) · doi:10.48550/arxiv.1906.07405 · Cited by 35
Improving Multimodal Accuracy Through Modality Pre-training and AttentionPreprint
Ismail, A. A., Hasan, M. M., & Ishtiaq, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06102 · Cited by 28
Agrillo, C., Ranpura, A. L., Butterworth, B. L., Ranpura, A. L., & Butterworth, B. L. (2013) · Psychology Press eBooks · doi:10.4324/9780203765401 · Cited by 17
Speech2Action: Cross-Modal Supervision for Action Recognition
Nagrani, A., Sun, C., Ross, D., Sukthankar, R., Schmid, C., & Zisserman, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01033 · Cited by 12
Cited by
Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., & Gao, K. (2022) · Proceedings of the 2022 International Conference on Multimodal Interaction · doi:10.1145/3536221.3556630 · Cited by 64
Decoupling Common and Unique Representations for Multimodal Self-supervised Learning
Wang, Y., Albrecht, C. M., Braham, N. A. A., Liu, C., Xiong, Z., & Zhu, X. X. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73397-0_17 · Cited by 49
$360+x$: A Panoptic Multi-modal Scene Understanding Dataset
Chen, H., Hou, Y., Qu, C., Testini, I., Hong, X., & Jiao, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01833 · Cited by 17
“For I3D, VGGish, and ResNet-18 networks, the training settings are 200 epochs with the parameters described in [19].”
MethodContinual Egocentric Activity Recognition With Foreseeable-Generalized Visual–IMU Representations
He, C., Cheng, S., Qiu, Z., Xu, L., Meng, F., Wu, Q., & Li, H. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3371975 · Cited by 11
Utilizing Data Imbalance to Enhance Compound–Protein Interaction Prediction Models
(2025) · doi:10.1002/aisy.202400985 · Cited by 3
Metadata
- Authors
- Peng, X., Wei, Y., Deng, A., Wang, D., & Hu, D.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 8228-8237
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312639100
- Cited by
- 349
- References
- 42
- FWCI
- 24.24
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.