Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
Open accessAbstract
Despite having impressive vision-language (VL) pretraining with BERT-based encoder for VL understanding, the pretraining of a universal encoder-decoder for both VL understanding and generation remains challenging. The difficulty originates from the inherently different peculiarities of the two disciplines, e.g., VL understanding tasks capitalize on the unrestricted message passing across modalities, while generation tasks only employ visual-to-textual message passing. In this paper, we start with a two-stream decoupled design of encoder-decoder structure, in which two decoupled cross-modal encoder and decoder are involved to separately perform each type of proxy tasks, for simultaneous VL understanding and generation pretraining. Moreover, for VL pretraining, the dominant way is to replace some input visual/word tokens with mask tokens and enforce the multi-modal encoder/decoder to reconstruct the original tokens, but no mask token is involved when fine-tuning on downstream tasks. As an alternative, we propose a primary scheduled sampling strategy that elegantly mitigates such discrepancy via pretraining encoder-decoder in a two-pass manner. Extensive experiments demonstrate the compelling generalizability of our pretrained encoder-decoder by fine-tuning on four VL understanding and generation downstream tasks. Source code is available at https://github.com/YehLi/TDEN.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For image-language Transformers, verb understanding is harder than subject or object understanding.
“Hendricks et al. [301] probe the image-language Transformers by fine-grained image-sentence pairs, and find that verb understanding is harder than subject or object understanding.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [301]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714 · see citing works
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,177 · see citing works
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.571 · Cited by 1,810 · see citing works
Self-citationScheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621 · see citing works
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402 · see citing works
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225 · see citing works
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · see citing works
Attention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092 · see citing works
Self-citationDeep Modular Co-Attention Networks for Visual Question Answering
Zhou, Y., Yu, J., Cui, Y., Tao, D., & Tian, Q. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00644 · Cited by 1,013 · see citing works
Exploring Visual Relationship for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_42 · Cited by 962 · see citing works
Self-citationVL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · see citing works
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864 · see citing works
MASS: Masked Sequence to Sequence Pre-training for Language GenerationPreprint
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02450 · Cited by 834 · see citing works
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795 · see citing works
From Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769 · see citing works
Kim, J.-H., Jun, J., & Zhang, B.-T. (2018) · arXiv · doi:10.48550/arxiv.1805.07932 · Cited by 761 · see citing works
Boosting Image Captioning with Attributes
Yao, T., Pan, Y., Li, Y., Qiu, Z., & Mei, T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.524 · Cited by 752 · see citing works
Self-citationX-Linear Attention Networks for Image Captioning
Pan, Y., Yao, T., Li, Y., & Mei, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01098 · Cited by 720 · see citing works
Self-citationJointly Modeling Embedding and Translation to Bridge Video and Language
Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.497 · Cited by 573 · see citing works
Self-citationLXMERT: Learning Cross-Modality Encoder Representations from TransformersPreprint
Tan, H. X., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07490 · Cited by 433 · see citing works
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering
Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C. H., Wang, X., & Li, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00680 · Cited by 415 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0965-7 · Cited by 375 · see citing works
Video Captioning with Transferred Semantic Attributes
Pan, Y., Yao, T., Li, H., & Mei, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.111 · Cited by 368 · see citing works
Self-citationUNITER: Learning UNiversal Image-TExt Representations
Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2019) · Cited by 280 · see citing works
Hierarchy Parsing for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00271 · Cited by 219 · see citing works
Self-citationJointly Localizing and Describing Events for Dense Video Captioning
Li, Y., Yao, T., Pan, Y., Chao, H., & Mei, T. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00782 · Cited by 196 · see citing works
Self-citationIncorporating Copying Mechanism in Image Captioning for Learning Novel Objects
Yao, T., Pan, Y., Li, Y., & Mei, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.559 · Cited by 157 · see citing works
Self-citationExploring Visual Relationship for Image CaptioningPreprint
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07041 · Cited by 141 · see citing works
Self-citationTemporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning
Chen, J., Pan, Y., Li, Y., Yao, T., Chao, H., & Mei, T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018167 · Cited by 113 · see citing works
Self-citationTemporal Deformable Convolutional Encoder-Decoder Networks for Video CaptioningPreprint
Chen, J., Pan, Y.-W., Li, Y.-H., Yao, T., Chao, H., & Mei, T. (2019) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1905.01077 · Cited by 110 · see citing works
Self-citationDeep Modular Co-Attention Networks for Visual Question AnsweringPreprint
Zhou, Y., Yu, J., Cui, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10770 · Cited by 98 · see citing works
Pointing Novel Objects in Image Captioning
Li, Y., Yao, T., Pan, Y., Chao, H., & Mei, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01278 · Cited by 79 · see citing works
Self-citationUnified Vision-Language Pre-Training for Image Captioning and VQAPreprint
Zhou, L., Palangi, H., Zhang, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11059 · Cited by 74 · see citing works
Learning a Metric Embedding for Face Recognition using the Multibatch MethodPreprint
Tadmor, O., Wexler, Y., Rosenwein, T., Shalev‐Shwartz, S., & Shashua, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07270 · Cited by 40 · see citing works
Jointly Localizing and Describing Events for Dense Video CaptioningPreprint
Li, Y., Yao, T., Pan, Y., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.08274 · Cited by 38 · see citing works
Self-citationIncorporating Copying Mechanism in Image Captioning for Learning Novel ObjectsPreprint
Yao, T., Pan, Y., Li, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05271 · Cited by 33 · see citing works
Self-citationAuto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-trainingPreprint
Pan, Y., Li, Y., Luo, J., Xu, J., Yao, T., & Mei, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02375 · Cited by 24 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 2
- Chen, J.; Pan, Y.; Li, Y.; Yao, T.; Chao, H.; and Mei, T. 2019a. Temporal deformable convolutional encoderdecoder networks for video captioning. In AAAI .
- Tan, H.; and Bansal, M. 2019. LXMERT: Learning CrossModality Encoder Representations from Transformers. In EMNLP-IJCNLP .
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
Contextual Transformer Networks for Visual Recognition
Li, Y., Yao, T., Pan, Y., & Mei, T. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3164083 · Cited by 771 · see citing works
Self-citationTransformers in computational visual media: A survey
Xu, Y., Wei, H., Lin, M., Deng, Y., Sheng, K., Zhang, M., Tang, F., Dong, W., Huang, F., & Xu, C. (2021) · Computational Visual Media · doi:10.1007/s41095-021-0247-3 · Cited by 137 · see citing works
Region-Aware Image Captioning via Interaction Learning
Liu, A.-A., Zhai, Y., Xu, N., Nie, W., Li, W., & Zhang, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3107035 · Cited by 128 · see citing works
Semantic-Conditional Diffusion Networks for Image Captioning*
Luo, J., Li, Y., Pan, Y., Yao, T., Feng, J., Chao, H., & Mei, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02237 · Cited by 128 · see citing works
Self-citationZhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., & Chen, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01894 · Cited by 100 · see citing works
Self-citationTowards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks
Luo, G., Zhou, Y., Sun, X., Wang, Y., Cao, L., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3139234 · Cited by 66 · see citing works
Vision-Enhanced and Consensus-Aware Transformer for Image Captioning
Cao, S., An, G., Zheng, Z., & Wang, Z. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3178844 · Cited by 61 · see citing works
Improving Medical Vision-Language Contrastive Pretraining With Semantics-Aware Triage
Liu, B., Lu, D., Wei, D., Wu, X., Wang, Y., Zhang, Y., & Zheng, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3294980 · Cited by 52 · see citing works
ESA: External Space Attention Aggregation for Image-Text Retrieval
Zhu, H., Zhang, C., Wei, Y., Huang, S., & Zhao, Y. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3253548 · Cited by 49 · see citing works
Vision-and-Language Pretrained Models: A Survey
Long, S., Cao, F., Han, S. C., & Yang, H. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/773 · Cited by 46 · see citing works
Retrieval Augmented Convolutional Encoder-decoder Networks for Video Captioning
Chen, J., Pan, Y.-W., Li, Y.-H., Yao, T., Chao, H., & Mei, T. (2022) · ACM Trans. Multim. Comput. Commun. Appl. · doi:10.1145/3539225 · Cited by 51 · see citing works
Self-citationZhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., & Chen, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00285 · Cited by 44 · see citing works
Self-citationHGNet: Learning Hierarchical Geometry from Points, Edges, and Surfaces
Yao, T., Li, Y., Pan, Y., & Mei, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02092 · Cited by 41 · see citing works
Self-citationNeuron-Based Spiking Transmission and Reasoning Network for Robust Image-Text Retrieval
Li, W., Ma, Z., Deng, L.-J., Xiaopeng Fan, B., & Tian, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3233042 · Cited by 40 · see citing works
Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training
Lei, C., Luo, S., Liu, Y., He, W., Wang, J., Wang, G., Tang, H., Miao, C., & Li, H. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475431 · Cited by 37 · see citing works
See and Learn More: Dense Caption-Aware Representation for Visual Question Answering
Bi, Y., Jiang, H., Hu, Y., Sun, Y., & Yin, B. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3291379 · Cited by 34 · see citing works
Feng, J., Wang, G., Zheng, C., Cai, Y., Fu, Z., Wang, Y., Wei, X.-Y., & Li, Q. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3284474 · Cited by 33 · see citing works
A thorough review of models, evaluation metrics, and datasets on image captioning
Luo, G., Cheng, L., Jing, C., Zhao, C., & Song, G. (2021) · IET Image Processing · doi:10.1049/ipr2.12367 · Cited by 28 · see citing works
Li, Y., Pan, Y., Chen, J., Yao, T., & Mei, T. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3478331 · Cited by 26 · see citing works
Self-citationCoT-UNet++: A medical image segmentation method based on contextual transformer and dense connection
Yin, Y., Xu, W. Z., Chen, L., & Wu, H. (2023) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2023364 · Cited by 25 · see citing works
Cross-Modal Alternating Learning With Task-Aware Representations for Continual Learning
Li, W., Gao, B.-B., Xia, B., Wang, J., Liu, J., Liu, Y., Wang, C., & Zheng, F. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3340551 · Cited by 26 · see citing works
Multimodal detection of hateful memes by applying a vision-language pre-training model
Chen, Y., & Pan, F. (2022) · PLoS ONE · doi:10.1371/journal.pone.0274300 · Cited by 22 · see citing works
Commonsense-Guided Semantic and Relational Consistencies for Image-Text Retrieval
Li, W., Yang, S. S., Li, Q., Li, X., & Liu, A.-A. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3289753 · Cited by 23 · see citing works
Jiang, J., Liu, Z., & Zheng, N. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01858-y · Cited by 22 · see citing works
Decoupled Cross-Modal Phrase-Attention Network for Image-Sentence Matching
Shi, Z., Zhang, T., Wei, X., Wu, F. H., & Zhang, Y. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3197972 · Cited by 20 · see citing works
Bottom-up and Top-down Object Inference Networks for Image Captioning
Pan, Y., Li, Y., Yao, T., & Mei, T. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3580366 · Cited by 16 · see citing works
Self-citationEnd-to-End Video Scene Graph Generation With Temporal Propagation Transformer
Zhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., & Chen, C. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3283879 · Cited by 15 · see citing works
Self-citationA contrastive framework for enhancing Knowledge Graph Question Answering: Alleviating exposure bias
Du, H., Zhang, X., Wang, M., Chen, Y., Ji, D., Ma, J., & Wang, H. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110996 · Cited by 15 · see citing works
Bridging the Cross-Modality Semantic Gap in Visual Question Answering
Wang, B., Ma, Y., Li, X., Gao, J., Hu, Y., & Yin, B. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3370925 · Cited by 12 · see citing works
A Critical Analysis of Benchmarks, Techniques, and Models in Medical Visual Question Answering
Al-Hadhrami, S. A., Menaï, M. E. B., Al-Ahmadi, S. A., & Alnafessah, A. (2023) · IEEE Access · doi:10.1109/access.2023.3335216 · Cited by 13 · see citing works
Wu, M., Hu, Y., Zhang, Y., Zeng, Z., Su, G., & Sha, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29893 · Cited by 10 · see citing works
Multi-Level Visual Representation with Semantic-Reinforced Learning for Video Captioning
Dong, C., Chen, X., Chen, A., Hu, F., Wang, Z., & Li, X. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3479217 · Cited by 6 · see citing works
Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioning
Luo, J., Chen, J., Li, Y., Pan, Y., Feng, J., Chao, H., & Yao, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_14 · Cited by 5 · see citing works
Self-citationSimultaneously Training and Compressing Vision-and-Language Pre-Training Model
Qi, Q., Zhang, A., Liao, Y., Sun, W., Wang, Y., Li, X., & Liu, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3233258 · Cited by 5 · see citing works
A Survey on Learning Objects’ Relationship for Image Captioning
Du, R., Zhang, W., Guo, Z. M., & Sun, X. (2023) · Computational Intelligence and Neuroscience · doi:10.1155/2023/8600853 · Cited by 5 · see citing works
A Scientometric Visualization Analysis of Image Captioning Research From 2010 to 2020
Liu, W., Wu, H., Hu, K., Luo, Q., & Cheng, X. (2021) · IEEE Access · doi:10.1109/access.2021.3129782 · Cited by 4 · see citing works
CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and DenoisingPreprint
Luo, J., Li, Y., Pan, Y., Yao, T., Chao, H., & Mei, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.07515 · Cited by 4 · see citing works
Self-citationXu, J., Zhuang, S., He, Y., Wang, H., Zhuang, Z., & Zeng, H. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.103966 · Cited by 4 · see citing works
An, B., Deng, J., Yang, M., Tian, Y., & Zhang, H. (2025) · Physics of Fluids · doi:10.1063/5.0250138 · Cited by 3 · see citing works
A Pretrained Spatio-Temporal Hypergraph Transformer for Multi-Stock Trend Forecasting
Wu, Y., Xie, L., Wan, H., & Xu, H. (2025) · Mathematics · doi:10.3390/math13101565 · Cited by 3 · see citing works
Deep-learning-based image captioning： analysis and prospects
Zhao, Y., Jin, Z. G., Zhang, F. S., Zhao, H., Tao, Z., Dou, C., Xu, X., & Liu, D. (2023) · Journal of Image and Graphics · doi:10.11834/jig.220660 · Cited by 4 · see citing works
X-modaler: A Versatile and High-performance Codebase for Cross-modal AnalyticsPreprint
Li, Y., Pan, Y., Chen, J., Yao, T., & Mei, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.08217 · Cited by 2 · see citing works
Self-citationMeasuring Similarity of Dual-Modal Academic Data Based on Multi-Fusion Representation Learning
Zhang, L., Gao, Q., Liu, M., Gu, Z., & Lang, B. (2024) · IEEE Access · doi:10.1109/access.2024.3427731 · Cited by 1 · see citing works
LFEN: A language feature enhanced network for scene text recognition
Chen, H., Jiang, R., Hu, F., Chen, M., & Zhang, Y. (2025) · Cognitive Robotics · doi:10.1016/j.cogr.2025.08.001 · Cited by 1 · see citing works
Embedding Coordinates in Transformer for Behavior Recognition in Classroom Scenarios
Zhu, H., & Zhao, J. (2023) · International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir59294.2023.10391226 · Cited by 1 · see citing works
MirrorDiff: Learning Mirror Diffusion for Image Captioning via Regeneration
Wang, J., Fu, L., Zhu, Y., Jin, Q., Wang, H., Li, Y., Wu, X., & Hu, K. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733389
Self-citationCausality guided co-attention network for visual question answering
Miao, J., Yu, K., Fang, B., Le, W., & Hong, R. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108200
DSACap: Enhancing Visual-Semantic Alignment with Diffusion-based Framework for Image Captioning
Fu, L., Wang, J., Li, Y., Jin, Q., Wang, H., Jing, Y., Huang, L., Yao, L., Zheng, J., Wu, X., et al. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755156
Self-citation
- Authors
- Li, Y., Pan, Y., Yao, T., Chen, J., & Mei, T.
- Affiliations
- Jingdong (China) (CN)
- Sun Yat-sen University (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2021-05-18
- Volume
- 35
- Issue
- 10
- Pages
- 8518-8526
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W3175095612
- Cited by
- 49
- References
- 50
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 4 · 2023: 18 · 2022: 11 · 2021: 9
- FWCI
- 3.43
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language pre-training, scheduled sampling, encoder-decoder architecture, visual language understanding, dual-stream architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
