Speech2Action: Cross-Modal Supervision for Action Recognition
Open accessAbstract
Is it possible to guess human action from dialogue alone? In this work we investigate the link between spoken words and actions in movies. We note that movie screenplays describe actions, as well as contain the speech of characters and hence can be used to learn this correlation with no additional supervision. We train a BERT-based Speech2Action classifier on over a thousand movie screenplays, to predict action labels from transcribed speech segments. We then apply this model to the speech segments of a large unlabelled movie corpus (188M speech segments from 288K movies). Using the predictions of this model, we obtain weak action labels for over 800K video clips. By training on these video clips, we demonstrate superior action recognition performance on standard action recognition benchmarks, without using a single manually labelled action example.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent attempts mainly use two straightforward methods to improve robustness for multimodal Transformer models: augmentation and adversarial learning based strategies, and fine-grained loss functions.
“Recent attempts mainly use two straightforward methods to improve the robustness for multimodal Transformer models: (1) augmentation and adversarial learning based strategies [293], [294], (2) fine-grained loss functions [295].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [295]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 59 · Cited by 12
Research works printed but not linked: 1
- Marcin Marszałek, Ivan Laptev, and Cordelia Schmid. Ac- tions in context. In CVPR 2009-IEEE Conference on Com- puter Vision & Pattern Recognition, pages 2929-2936. IEEE Computer Society, 2009.
- Authors
- Nagrani, A., Sun, C., Ross, D., Sukthankar, R., Schmid, C., & Zisserman, A.
- Affiliations
- Google (United States) (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 10314-10323
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3013845856
- Cited by
- 12
- References
- 59
- Citations by year
- 2025: 1 · 2022: 4 · 2021: 4 · 2020: 3
- FWCI
- 0.85
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- human action recognition, cross-modal supervision, weak labels, BERT, speech transcription, video annotation, weakly supervised learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
