CoChat
Sign Up
Cited by 46

Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation

Yariv, Guy, Gat, Itai, Benaim, Sagie, Wolf, Lior, Schwartz, Idan, Adi, Yossi·2024·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v38i7.28486
Open this paper in CoChat — free account

Abstract

We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. For this task, the videos are required to be aligned both globally and temporally with the input audio: globally, the input audio is semantically associated with the entire output video, and temporally, each segment of the input audio is associated with a corresponding segment of that video. We utilize an existing text-conditioned video generation model and a pre-trained audio encoder model. The proposed method is based on a lightweight adaptor network, which learns to map the audio-based representation to the input representation expected by the text-to-video generation model. As such, it also enables video generation conditioned on text, audio, and, for the first time as far as we can ascertain, on both text and audio. We validate our method extensively on three datasets demonstrating significant semantic diversity of audio-video samples and further propose a novel evaluation metric (AV-Align) to assess the alignment of generated videos with input audio samples. AV-Align is based on the detection and comparison of energy peaks in both modalities. In comparison to recent state-of-the-art approaches, our method generates videos that are better aligned with the input sound, both with respect to content and temporal axis. We also show that videos produced by our method present higher visual quality and are more diverse. Code and samples are available at: https://pages.cs.huji.ac.il/adiyoss-lab/TempoTokens/.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v38i7.28486

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AV-Align is a metric for assessing temporal alignment between audio-video pairs, based on energy peaks of both modalities.

    “With this respect, AV-Align (Yariv et al., 2024) is a metric for assessing temporal alignment between audio-video pairs. It is based on energy peaks of both modalities, i.e., the highest mean magnitude of optical flow for video frames and the onset of the audio waveform.”

    From Deepfake: definitions, performance metrics and standards, datasets, and a meta-review (2024) · cites this paper as Yariv et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. Adi, Y.; Zeghidour, N.; Collobert, R.; Usunier, N.; Liptchin- sky, V.; and Synnaeve, G. 2019. To reverse the gradient or not: An empirical comparison of adversarial and multi-task learning in speech recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 3742-3746. IEEE.
  2. Ali, A.; Schwartz, I.; Hazan, T.; and Wolf, L. 2022. Video and text matching with conditioned embeddings. In Pro- ceedings of the IEEE/CVF winter conference on applica- tions of computer vision, 1565-1574.
  3. NeurIPS.
  4. Chatterjee, M.; and Cherian, A. 2020. Sound2sight: Gener- ating visual dynamics from sound and context. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, 701- 719. Springer.
  5. Chen, H.; Xie, W.; Vedaldi, A.; and Zisserman, A. 2020. Vg- gsound: A large-scale audio-visual dataset. In ICASSP.
  6. Chen, L.; Srivastava, S.; Duan, Z.; and Xu, C. 2017. Deep cross-modal audio-visual generation. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017, 349-357.
  7. Chen, S.; Wu, Y.; Wang, C.; Liu, S.; Tompkins, D.; Chen, Z.; and Wei, F. 2022. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058. · arXiv
  8. Copet, J.; Kreuk, F.; Gat, I.; Remez, T.; Kant, D.; Synnaeve, G.; Adi, Y.; and Défossez, A. 2023. Simple and Controllable Music Generation. arXiv preprint arXiv:2306.05284. · arXiv
  9. Crowson, K.; Biderman, S.; Kornis, D.; Stander, D.; Halla- han, E.; Castricato, L.; and Raff, E. 2022. Vqgan-clip: Open domain image generation and editing with natural language guidance. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceed- ings, Part XXXVII, 88-105. Springer.
  10. Esser, P.; Rombach, R.; and Ommer, B. 2021. Taming trans- formers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12873-12883.
  11. Gafni, O.; Polyak, A.; Ashual, O.; Sheynin, S.; Parikh, D.; and Taigman, Y. 2022. Make-a-scene: Scene-based text-to- image generation with human priors. In European Confer- ence on Computer Vision, 89-106. Springer.
  12. Gat, I.; Lorberbom, G.; Schwartz, I.; and Hazan, T. 2022. Latent space explanation by intervention. In AAAI.
  13. Ge, S.; Hayes, T.; Yang, H.; Yin, X.; Pang, G.; Jacobs, D.; Huang, J.-B.; and Parikh, D. 2022. Long video gen- eration with time-agnostic vqgan and time-sensitive trans- former. In European Conference on Computer Vision, 102- 118. Springer.
  14. Gemmeke, J. F.; Ellis, D. P.; Freedman, D.; Jansen, A.; Lawrence, W.; Moore, R. C.; Plakal, M.; and Ritter, M. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), 776-780. IEEE.
  15. Hao, W.; Guan, H.; and Zhang, Z. 2022. Vag: A uni- form model for cross-modal visual-audio mutual generation. IEEE Transactions on Neural Networks and Learning Sys- tems.
  16. Hassid, M.; Remez, T.; Nguyen, T. A.; Gat, I.; Conneau, A.; Kreuk, F.; Copet, J.; Defossez, A.; Synnaeve, G.; Dupoux, E.; et al. 2023. Textually Pretrained Speech Language Mod- els. arXiv preprint arXiv:2305.13009. · arXiv
  17. He, Y.; Yang, T.; Zhang, Y.; Shan, Y.; and Chen, Q. 2022. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221. · arXiv
  18. Ho, J.; Chan, W.; Saharia, C.; Whang, J.; Gao, R.; Gritsenko, A.; Kingma, D. P.; Poole, B.; Norouzi, M.; Fleet, D. J.; et al. 2022. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303. · arXiv
  19. Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. NeurIPS.
  20. Hong, W.; Ding, M.; Zheng, W.; Liu, X.; and Tang, J. 2022. Cogvideo: Large-scale pretraining for text-to-video genera- tion via transformers. arXiv preprint arXiv:2205.15868. · arXiv
  21. Horn, B. K.; and Schunck, B. G. 1981. Determining optical flow. Artificial Intelligence, 17(1): 185-203.
  22. Kreuk, F.; Synnaeve, G.; Polyak, A.; Singer, U.; Défossez, A.; Copet, J.; Parikh, D.; Taigman, Y.; and Adi, Y. 2022. Au- diogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352. · arXiv
  23. Kumar, N.; Goel, S.; Narang, A.; and Hasan, M. 2020. Ro- bust one shot audio to video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 770-771.
  24. Lee, S. H.; Oh, G.; Byeon, W.; Kim, C.; Ryoo, W. J.; Yoon, S. H.; Cho, H.; Bae, J.; Kim, J.; and Kim, S. 2022. Sound- guided semantic video generation. In European Conference on Computer Vision, 34-50. Springer.
  25. Mama, R.; Tyndel, M. S.; Kadhim, H.; Clifford, C.; and Thu- rairatnam, R. 2021. NWT: towards natural audio-to-video generation with representation learning. arXiv preprint arXiv:2106.04283. · arXiv
  26. Nichol, A. Q.; Dhariwal, P.; Ramesh, A.; Shyam, P.; Mishkin, P.; Mcgrew, B.; Sutskever, I.; and Chen, M. 2022. GLIDE: Towards Photorealistic Image Generation and Edit- ing with Text-Guided Diffusion Models. In International Conference on Machine Learning, 16784-16804.
  27. PMLR.
  28. Park, S. J.; Kim, M.; Hong, J.; Choi, J.; and Ro, Y. M. 2022. Synctalkface: Talking face generation with pre- cise lip-syncing via audio-lip memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2062-2070.
  29. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In ICML.
  30. Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Rad- ford, A.; Chen, M.; and Sutskever, I. 2021. Zero-shot text-to- image generation. In International Conference on Machine Learning, 8821-8831.
  31. PMLR.
  32. Ruan, L.; Ma, Y.; Yang, H.; He, H.; Liu, B.; Fu, J.; Yuan, N. J.; Jin, Q.; and Guo, B. 2023. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video gen- eration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10219-10228.
  33. Schwartz, I.; Yu, S.; Hazan, T.; and Schwing, A. G. 2019. Factor graph attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2039-2048.
  34. Sheffer, R.; and Adi, Y. 2023. I hear your true colors: Im- age guided audio generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1-5. IEEE.
  35. Singer, U.; Polyak, A.; Hayes, T.; Yin, X.; An, J.; Zhang, S.; Hu, Q.; Yang, H.; Ashual, O.; Gafni, O.; et al. 2022. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792. · arXiv
  36. Soomro, K.; Zamir, A. R.; and Shah, M. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. · arXiv
  37. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. · arXiv
  38. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  39. Tran, D.; Bourdev, L.; Fergus, R.; Torresani, L.; and Paluri, M. 2015. Learning Spatiotemporal Features with 3D Con- volutional Networks. arXiv:1412.0767. · arXiv
  40. Unterthiner, T.; van Steenkiste, S.; Kurach, K.; Marinier, R.; Michalski, M.; and Gelly, S. 2019. Towards Accurate Generative Models of Video: A New Metric Challenges. arXiv:1812.01717. · arXiv
  41. Wan, C.-H.; Chuang, S.-P.; and Lee, H.-Y. 2019. Towards audio to scene image synthesis using generative adversarial network. In ICASSP 2019-2019 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), 496-500. IEEE.
  42. Wu, C.; Liang, J.; Ji, L.; Yang, F.; Fang, Y.; Jiang, D.; and Duan, N. 2022a. Nüwa: Visual synthesis pre-training for neural visual world creation. In European conference on computer vision, 720-736. Springer.
  43. Wu, H.-H.; Seetharaman, P.; Kumar, K.; and Bello, J. P. 2022b. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4563-4567. IEEE.
  44. Yariv, G.; Gat, I.; Wolf, L.; Adi, Y.; and Schwartz, I. 2023. AudioToken: Adaptation of Text-Conditioned Diffu- sion Models for Audio-to-Image Generation. arXiv preprint arXiv:2305.13050. · arXiv
  45. Yu, J.; Xu, Y.; Koh, J. Y.; Luong, T.; Baid, G.; Wang, Z.; Vasudevan, V.; Ku, A.; Yang, Y.; Ayan, B. K.; et al. 2022. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2(3): 5. · arXiv
  46. Yu, L.; Cheng, Y.; Sohn, K.; Lezama, J.; Zhang, H.; Chang, H.; Hauptmann, A. G.; Yang, M.-H.; Hao, Y.; Essa, I.; et al. 2023. Magvit: Masked generative video transformer. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10459-10469.

Cited by

Sorted by citations

Authors
Yariv, G., Gat, I., Benaim, S., Wolf, L., Schwartz, I., & Adi, Y.
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2024-03-24
Volume
38
Issue
7
Pages
6639-6647
ISSN
  • 2374-3468
  • 2159-5399
DOI
10.1609/aaai.v38i7.28486
Cited by
46
References
7

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review