Transfer learning for multimodal dialog
Open accessAbstract
Audio-Visual Scene-Aware Dialog (AVSD) is best understood as an extension of Visual Question Answering, the task of generating a textual answer in response to a textual question on multi-media content. In AVSD, the answer-relevant “context” is expanded to include past dialog turns, which we view as a specialized form of extra textual knowledge (in addition to the standard video features). We have developed a framework that uses hierarchical attention to fuse contributions from different modalities, and had shown how it can be used to generate textual summaries from multi-modal sources, specifically videos with accompanying commentary. In this paper, we transfer the algorithmic approach, models, and data from this background corpus of 2000 h of how-to videos to the AVSD task, and report our findings. Our approach uses dialog context, but makes no assumption about the ordering of the history. Our system achieves the best performance in both automatic and human evaluations in the 7th Dialog State Tracking Challenge (AVSD).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transfer learning can improve the quality of the information transferred by eliminating latent noise and conflict.
“Moreover, it can improve the quality of the information transferred by eliminating any latent noise and conflict [113, 115–117].”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [113]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Proceedings of IEEE International Conference on Computer Vision
(1995) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.1995.466933 · Cited by 13,117 · see citing works
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
2017 IEEE International Conference on Computer Vision (ICCV)
(2017) · doi:10.1109/iccv37128.2017 · Cited by 3,129 · see citing works
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952132 · Cited by 2,605 · see citing works
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
Hara, K., Kataoka, H., & Satoh, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00685 · Cited by 2,300 · see citing works
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · see citing works
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,633 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works
On the Properties of Neural Machine Translation: Encoder-Decoder ApproachesPreprint
Cho, K., van Merriënboer, B., Bahdanau, D., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.1259 · Cited by 1,112 · see citing works
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968 · see citing works
Visual7W: Grounded Question Answering in Images
Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794 · see citing works
Lin, C.-Y., & Och, F. J. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219032 · Cited by 737 · see citing works
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Yu, H., Wang, J., Huang, Z., Yang, Y., & Xu, W. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.496 · Cited by 568 · see citing works
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Wouters, J., Kloosterman, F., & Bertrand, A. (2020) · doi:10.1109/icassp40776.2020 · Cited by 547 · see citing works
Das, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · Cited by 432 · see citing works
Attention-Based Multimodal Fusion for Video Description
Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.450 · Cited by 401 · see citing works
Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
Nallapati, R. M., Zhou, B., dos Santos, C. N., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.06023 · Cited by 371 · see citing works
Visual Turing test for computer vision systems
Geman, D., Geman, S., Hallonquist, N., & Laurent Younes, B. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1422953112 · Cited by 323 · see citing works
2006 International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2006)
(2005) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.849938 · Cited by 313 · see citing works
A Shared Task on Multimodal Machine Translation and Crosslingual Image Description
Specia, L., Frank, S., Sima’an, K., & Elliott, D. (2016) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w16-2346 · Cited by 228 · see citing works
Hollywood in Homes: Crowdsourcing Data Collection for Activity UnderstandingPreprint
Sigurdsson, G. A., Varol, G., Wang, X., Farhadi, A., Laptev, I., & Gupta, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.01753 · Cited by 221 · see citing works
How2: A Large-scale Dataset for Multimodal Language UnderstandingPreprint
Sanabria, R., Çağlayan, O., Palaskar, S., Elliott, D., Barrault, L., Specia, L., & Metze, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00347 · Cited by 179 · see citing works
Self-citationAudio Visual Scene-Aware Dialog
Alamri, H., Cartillier, V., Das, A., Wang, J., Cherian, A., Essa, I. A., Batra, D., Marks, T. K., Hori, C., Anderson, P., et al. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00774 · Cited by 172 · see citing works
Multimodal Abstractive Summarization for How2 Videos
Palaskar, S., Libovický, J., Gella, S., et al. (2019) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p19-1659 · Cited by 113 · see citing works
Self-citationProceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation
Vecchi, E. M., Carreras, X., & Wu, D. (2014) · doi:10.3115/v1/w14-40 · Cited by 112 · see citing works
End-to-end Audio Visual Scene-aware Dialog Using Multimodal Attention-based Video Features
Hori, C., Alamri, H., Wang, J., Wichern, G., Hori, T., Cherian, A., Marks, T. K., Cartillier, V., Lopes, R. G., Das, A., et al. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682583 · Cited by 112 · see citing works
NMTPY: A Flexible Toolkit for Advanced Neural Machine Translation Systems
Caglayan, O., García-Martínez, M., Bardet, A., Aransa, W., Bougares, F., & Barrault, L. (2017) · The Prague Bulletin of Mathematical Linguistics · doi:10.1515/pralin-2017-0035 · Cited by 61 · see citing works
End-to-end Multimodal Speech Recognition
Palaskar, S., Sanabria, R., & Metze, F. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462439 · Cited by 39 · see citing works
Self-citationVisual features for context-aware speech recognition
Gupta, A., Miao, Y., Neves, L., & Metze, F. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953112 · Cited by 37 · see citing works
Self-citationAttention Strategies for Multi-Source Sequence-to-Sequence Learning
Libovický, J., & Helcl, J. (2017) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p17-2031 · Cited by 33 · see citing works
Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7Preprint
Alamri, H., Cartillier, V., Lopes, R. G., Das, A., Wang, J., Essa, I. A., Batra, D., Parikh, D., Cherian, A., Marks, T. K., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00525 · Cited by 21 · see citing works
Computational Natural Language Learning
Seel, P. D. N. M. (2012) · Springer eBooks · doi:10.1007/978-1-4419-1428-6_3521 · Cited by 15 · see citing works
CNN Architectures for Large-Scale Audio ClassificationPreprint
Hershey, S., Chaudhuri, S., Ellis, D. P. W., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.09430 · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Libovick y, J., Palaskar, S., Gella, S., Metze, F., 2018. Multimodal abstractive summarization of open-domain videos. NeurIPS Workshop on Visually Grounded Inter- action and Language (ViGIL).
- Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y., 2015. Show, attend and tell: neural image caption generation with visual atten- tion. International conference on machine learning, pp. 2048-2057.
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54 · see citing works
- Authors
- Palaskar, S., Sanabria, R., & Metze, F.
- Affiliations
- Carnegie Mellon University (US)
- Corresponding author
- Palaskar, Shruti
- Venue
- Computer Speech & Language
- Publisher
- Elsevier BV
- Published
- 2020-11
- Volume
- 64
- Pages
- 101093
- ISSN
- 0885-2308 (ISSN-L)
- OpenAlex
- W3012039651
- Cited by
- 6
- References
- 41
- Citations by year
- 2023: 1 · 2022: 2 · 2021: 3
- FWCI
- 0.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Computer Speech & Language (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- transfer learning, visual question answering, multimodal fusion, hierarchical attention, video tutorials, dialogue state tracking, multimodal summarization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
