The DOI matches its DataCite record; 74 of the 74 references it cites were found in a registry or an index; none of the 4 sources its claims cite is checked for support yet.
Token Turing Machines
Open accessAbstract
We propose Token Turing Machines (TTM), a sequential, autoregressive Transformer model with memory for real-world sequential visual understanding. Our model is inspired by the seminal Neural Turing Machine, and has an external memory consisting of a set of tokens which summarise the previous history (i.e., frames). This memory is efficiently addressed, read and written using a Transformer as the processing unit/controller at each step. The model's memory module ensures that a new observation will only be processed with the contents of the memory (and not the entire history), meaning that it can efficiently process long sequences with a bounded computational cost at each step. We show that TTM outperforms other alternatives, such as other Transformer models designed for long sequences and recurrent neural networks, on two real-world sequential visual understanding tasks: online temporal activity detection from videos and vision-based robot action policy learning. Code is publicly available at: https://github.com/google-research/scenic/tree/main/scenic/projects/token_turing
Overview
CoChat Summary
TL;DR
Token Turing Machines (TTM) are sequential autoregressive Transformers with an external token memory that is read and written via token summarization, giving bounded per-step computation for long visual sequences. TTM improves over strong baselines on online temporal activity detection and real-robot action policy learning.
Key findings
- On Charades temporal activity detection, ViViT-B + TTM reaches 26.34 mAP versus 23.18 for ViViT-B alone, and ViViT-L + TTM reaches 28.79 mAP versus 26.01 for ViViT-L.
- TTM-Transformer with n = 16 uses 0.228 GFLOPS and reaches 26.24 mAP, compared with causal/recurrent Transformers at 0.523/0.410 GFLOPS and 25.85/25.97 mAP.
- On AVA v2.2, ViViT-B + TTM per box with 4 layers reaches 31.5 mAP (+6.3 over ViViT-B) with +2.0 GFLOPS; MeMViT reaches 28.5 mAP (+2.3) with +1.3 GFLOPS.
- In real-robot evaluation, TTM achieves 89.26 average task success rate versus 79.80 for BC ResNet, 80.08 for BC EfficientNet, and 79.26 for a no-memory version.
Methods
- TTM maintains an external memory of m tokens and uses token summarization for read and write operations; the processing unit can be a Transformer or MLPMixer, and a linear output head produces per-step predictions.
- Video experiments use ViViT-B/L backbones, Charades with ∼9.8k videos of 157 activities (∼7.9k training, ∼1.8k validation) and 25 validation frames for mAP, plus AVA v2.2 with 80 atomic actions in 430 clips.
- Robot experiments use a real2real behavioral cloning dataset of 89,000 teleoperated episodes across 551 tasks in a mock kitchen, with TTM using EfficientNet outputs as tokens, memory size m = 96, reads r = 16, n = 48, and an 8-layer Transformer processing unit.
Limitations
- The paper does not include a dedicated limitations section; it states that its focus is computer vision problems with sequential visual data and that the chosen tasks require very long token sequences.
- Evident from the text: evaluation is limited to two task families and specific datasets, and the AVA experiment uses only 4 steps at a time due to memory constraints.
- Evident from the text: the robot evaluation is in the same kitchen environment used for training, and the paper reports no explicit failure-case analysis or broader deployment evaluation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 3 Not checked · 25 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that TTM outperforms other alternatives, such as other Transformer models designed for long sequences and recurrent neural networks, on two real-world sequential visual understanding tasks: online temporal activity detection from videos and vision-based robot action policy learning.”
- Also stated in: § 1. Introduction · boosted
“We demonstrate its capabilities by achieving substantial improvements over strong baselines in two diverse and challenging tasks: (1) online temporal action detection (i.e., localisation) from videos and (2) vision-based robot action policy learning.”
§ Token Turing Machines > Abstract
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1. Introduction · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We are able to confirm that TTM, due to its external memory interactions, enables much more efficient online inference compared to other types of sequential/temporal models.”
§ Token Turing Machines > 4.1.3 Temporal Activity Detection Results
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When TTM-Transformer and causal/recurrent Transformers are using the same number of input tokens (i.e., n = 16), TTM spends around 1/2 FLOPS compared to the causal/recurrent Transformers (TTM 0.228 vs. causal/recurrent Transformers 0.523/0.410GFLOPS), while TTM still outperforms them (TTM 26.24 vs. causal/recurrent Transformers 25.85/25.97mAP).”
§ Token Turing Machines > 4.1.3 Temporal Activity Detection Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When TTM-Transformer and causal/recurrent Transformers are using the same number of input tokens (i.e., n = 16), TTM spends around 1/2 FLOPS compared to the causal/recurrent Transformers (TTM 0.228 vs. causal/recurrent Transformers 0.523/0.410GFLOPS), while TTM still outperforms them (TTM 26.24 vs. causal/recurrent Transformers 25.85/25.97mAP).”
§ Token Turing Machines > 4.1.3 Temporal Activity Detection Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We are able to observe the benefit of TTM, which improves the accuracy by managing the external memory, with a little added compute.”
- Also stated in: § 4.1.5 Spatio-temporal Activity Detection Results · boosted
“What we confirm in this experiment is the relative gain over the backbone; TTM with a small added compute, by utilizing the memory, improves AVA activity detection.”
§ Token Turing Machines > 4.1.5 Spatio-temporal Activity Detection Results
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1.5 Spatio-temporal Activity Detection Results · boosted
8 more claims
Create a free account to see all 30 claims and every source verdict.
- Probably not a substantial claim. 10
- It may state two separate claims as one. 9
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Token Turing Machines > 4.3. Real-robot Results
“Fig. 5 and Table 6 show the results.”
§ Token Turing Machines > A.2. Robot Policy Training
“Table: 6 Table 6 . Results on AVA 2.2, with the vanilla Kinetics-400 pretraining.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Token Turing Machines > 4.1.4 Ablations
“The number of input tokens per step, n = 3176.”
§ Token Turing Machines > 4.1.4 Ablations
“The number of input tokens per step is n = 3136.Table 5.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Token Turing Machines > 3.1.4 Writing to Memory
“Therefore, we formulate our write operations as selecting n tokens (i.e., the size of the memory) from the concatenation of current memory, input and output tokens, as denoted by M t+1 = Write(M t , O t , I t ) = S n ([M t ||O t ||I t ]).”
§ Token Turing Machines > 3. Token Turing Machines
“The memory at time step t, M t ∈ R m×d , consists of a set of m tokens of dimensionality d.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Research works printed but not linked: 1
- DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block-recurrent trans- formers. In NeurIPS, 2022. 2
Cited by
Streaming Dense Video Captioning
Zhou, X., Arnab, A., Buch, S., Shen, Y., Myers, A., Xiong, X., Nagrani, A., & Schmid, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01727 · Cited by 83
Self-citationVicTR: Video-conditioned Text Representations for Activity Recognition
Kahatapitiya, K., Arnab, A., Nagran, A., & Ryoo, M. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01755 · Cited by 36
Self-citationShea, N. J. (2024) · doi:10.1093/9780191997167.001.0001 · Cited by 29
Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities
Piergiovanni, A., Noble, I., Kim, D., Ryoo, M. S., Gomes, V., & Angelova, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02531 · Cited by 26
Self-citationGeneralization to New Sequential Decision Making Tasks with In-Context LearningPreprint
Raparthy, S. C., Hambro, E., Kirk, R., Henaff, M., & Răileanu, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.03801 · Cited by 26
Metadata
- Authors
- Ryoo, M. S., Gopalakrishnan, K., Kahatapitiya, K., Xiao, T., Rao, K., Stone, A., Lu, Y., Ibarz, J., & Arnab, A.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 19070-19081
- ISSN
- 1063-6919 (ISSN-L)
- arXiv
- 2211.09119
- OpenAlex
- W4386076008
- Cited by
- 21
- References
- 74
- Citations by year
- 2026: 7 · 2025: 5 · 2024: 6 · 2023: 3
- FWCI
- 1.85
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- neural Turing machines, external memory, long sequence processing, video analysis, vision-based reinforcement learning, autoregressive transformers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
