CoChat
Sign Up
PreprintCited by 2,264Multimodal Machine Learning ApplicationsTopic ModelingVideo Analysis and Summarization

Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

Open access
Zhu, Yukun, Kiros, Ryan, Zemel, Richard, Salakhutdinov, Ruslan, Urtasun, Raquel, Torralba, Antonio, Fidler, Sanja·2015·arXiv
DOI 10.48550/arxiv.1506.06724arXiv 1506.06724
Open this paper in CoChat — free accountOpen PDF

Abstract

Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This paper aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in current datasets. To align movies and books we exploit a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1506.06724

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Dynamic time warping and conditional random fields do not explicitly model event boundaries.

    “However, these methods do not explicitly model event boundaries.”

    From Discovering Event Structure in Continuous Narrative Perception and Memory (2017) · cites this paper as Zhu et al., 2015

    Section 4.4 (p. 6) defines the paper’s CRF in terms of shot-to-sentence alignments and consistency between nearby alignments, with no explicit event-boundary variable. It describes dynamic time warping only as enforcing forward-in-time matching; that description does not establish whether the dynamic-time-warping method models event boundaries. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. [1][1] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. ICLR , 2015. 4 -
  2. [2][2] K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. EMNLP , 2014. 4 -
  3. [3][3] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014. 4 - · arXiv
  4. [4][4] T. Cour, C. Jordan, E. Miltsakaki, and B. Taskar. Movie/script: Alignment and parsing of video and text transcription. In ECCV , 2008. 2 -
  5. [5][5] M. Everingham, J. Sivic, and A. Zisserman. 'Hello! My name is... Buffy' - Automatic Naming of Characters in TV Video. BMVC , pages 899-908, 2006. 2 -
  6. [6][6] A. Farhadi, M. Hejrati, M. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences for images. In ECCV , 2010. 2 -
  7. [7][7] S. Fidler, A. Sharma, and R. Urtasun. A sentence is worth a thousand pixels. In CVPR , 2013. 2 -
  8. [8][8] A. Gupta and L. Davis. Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers. In ECCV , 2008. 1 -
  9. [9][9] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997. 4 -
  10. [10][10] N. Kalchbrenner and P. Blunsom. Recurrent continuous translation models. In EMNLP , pages 1700-1709, 2013. 4 -
  11. [11][11] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR , 2015. 1, 2 -
  12. [13][13] R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. CoRR , abs/1411.2539, 2014. 1, 2, 3, 5, 9, 10 - · arXiv
  13. [14][14] R. Kiros, Y. Zhu, R. Salakhutdinov, R. S. Zemel, A. Torralba, R. Urtasun, and S. Fidler. Skip-Thought Vectors. In Arxiv , 2015. 3, 4 -
  14. [15][15] C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What are you talking about? text-to-image coreference. In CVPR , 2014. 1, 2 -
  15. [16][16] G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y . Choi, A. Berg, and T. Berg. Baby talk: Understanding and generating simple image descriptions. In CVPR , 2011. 2 -
  16. [17][17] D. Lin, S. Fidler, C. Kong, and R. Urtasun. Visual Semantic Search: Retrieving Videos via Complex Textual Queries. CVPR , pages 2657-2664, 2014. 1, 2 -
  17. [18][18] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755. 2014. 1, 19 -
  18. [19][19] X. Lin and D. Parikh. Don't just listen, use your imagination: Leveraging visual common sense for non-visual tasks. In CVPR , 2015. 1 -
  19. [20][20] M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS , 2014. 1 -
  20. [21][21] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. In arXiv:1410.1090 , 2014. 1, 2 - · arXiv
  21. [22][22] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013. 4 - · arXiv
  22. [23][23] K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. BLEU: a method for automatic evaluation of machine translation. In ACL , pages 311-318, 2002. 6 -
  23. [24][24] H. Pirsiavash, C. Vondrick, and A. Torralba. Inferring the why in images. arXiv.org , jun 2014. 2 -
  24. [25][25] V. Ramanathan, A. Joulin, P. Liang, and L. Fei-Fei. Linking People in Videos with 'Their' Names Using Coreference Resolution. In ECCV , pages 95-110. 2014. 2 -
  25. [26][26] V. Ramanathan, P. Liang, and L. Fei-Fei. Video event understanding using natural language descriptions. In ICCV , 2013. 1 -
  26. [27][27] A. Rohrbach, M. Rohrbach, N. Tandon, and B. Schiele. A dataset for movie description. In CVPR , 2015. 2, 5 -
  27. [28][28] P. Sankar, C. V. Jawahar, and A. Zisserman. Subtitle-free Movie to Script Alignment. In BMVC , 2009. 2 -
  28. [29][29] A. Schwing, T. Hazan, M. Pollefeys, and R. Urtasun. Efficient Structured Prediction with Latent Variables for General Graphical Models. In ICML , 2012. 6 -
  29. [30][30] J. Sivic, M. Everingham, and A. Zisserman. 'Who are you?' -Learning person specific classifiers from video. CVPR , pages 1145-1152, 2009. 2 -
  30. [31][31] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS , 2014. 4 -
  31. [32][32] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842 , 2014. 5 - · arXiv
  32. [33][33] M. Tapaswi, M. Bauml, and R. Stiefelhagen. Book2Movie: Aligning Video scenes with Book chapters. In CVPR , 2015. 2 -
  33. [35][35] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. J. Mooney, and K. Saenko. Translating Videos to Natural Language Using Deep Recurrent Neural Networks. CoRR abs/1312.6229 , cs.CV, 2014. 1, 2 - · arXiv
  34. [36][36] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In arXiv:1411.4555 , 2014. 1, 2 - · arXiv
  35. [37][37] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In arXiv:1502.03044 , 2015. 2 - · arXiv
  36. [38][38] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning Deep Features for Scene Recognition using Places Database. In NIPS , 2014. 5, 7

Cited by

Sorted by citations

Authors
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S.
Affiliations
  • University of Toronto (CA)
  • Massachusetts Institute of Technology (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2015
Pages
19-27
DOI
10.48550/arxiv.1506.06724
arXiv
1506.06724
OpenAlex
W1566289585
Cited by
2,264
References
1
Citations by year
2026: 35 · 2025: 135 · 2024: 212 · 2023: 345 · 2022: 352 · 2021: 427 · 2020: 363 · 2019: 208 · 2018: 84 · 2017: 56 · 2016: 44 · 2015: 3
FWCI
25.57
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • Canadian Institute for Advanced Research
  • Samsung
  • Natural Sciences and Engineering Research Council of Canada (ONR-N00014-14-1-0232)
  • Office of Naval Research (ONR-N00014, N00014)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
Keywords
visual explanations, multimodal fusion, visual captioning, narrative semantics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review