Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling
Open accessAbstract
While Large Language Models require more and more data to train and scale, rather than looking for any data to acquire, we should consider what types of tasks are more likely to benefit from data scaling. We should be intentional in our data acquisition. We argue that the shape of the data itself, such as its compositional and structural patterns, informs which tasks to prioritize in data scaling, and shapes the development of the next generation of compute paradigms for tasks where data scaling is inefficient, or even insufficient.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pure model- and data-scaling based on recurrent models will not be enough to match the performance of attention-based architecture.
“This finding is non-surprising given the impact of the attention mechanism on the field of machine translation (Bahdanau et al., 2014), indicating that pure model- and data-scaling based on recurrent models will not be enough to match the performance of attention-based architecture (Rodchenko et al., 2025).”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Rodchenko et al., 2025
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
“This finding is non-surprising given the impact of the attention mechanism on the field of machine translation [6], indicating that pure model-and data-scaling based on recurrent models will not be enough to match the performance of attention-based architecture [84].”
Self-citation
- Authors
- Rodchenko, T., Noy, N., & Scherrer, N.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2501.13779
- OpenAlex
- W4406793945
- Cited by
- 0
- References
- 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- data scaling, large language models, data acquisition, compositional data, structural patterns, computational efficiency, training data requirements, data-centric AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
