Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 1,740

The DOI matches its Crossref record; 108 of the 108 references it cites were found in a registry or an index.

Text Data Augmentation for Deep Learning

Open access
Shorten, Connor, Khoshgoftaar, Taghi M., Furht, Borko·2021·Journal of Big Data
DOI 10.1186/s40537-021-00492-0

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Natural Language Processing (NLP) is one of the most captivating applications of Deep Learning. In this survey, we consider how the Data Augmentation training strategy can aid in its development. We begin with the major motifs of Data Augmentation summarized into strengthening local decision boundaries, brute force training, causality and counterfactual examples, and the distinction between meaning and form. We follow these motifs with a concrete list of augmentation frameworks that have been developed for text data. Deep Learning generally struggles with the measurement of generalization and characterization of overfitting. We highlight studies that cover how augmentations can construct test sets for generalization. NLP is at an early stage in applying Data Augmentation compared to Computer Vision. We highlight the key differences and promising ideas that have yet to be tested in NLP. For the sake of practical implementation, we describe tools that facilitate Data Augmentation such as the use of consistency regularization, controllers, and offline and online augmentation pipelines, to preview a few. Finally, we discuss interesting topics around Data Augmentation in NLP such as task-specific augmentations, the use of prior knowledge in self-supervised learning versus Data Augmentation, intersections with transfer and multi-task learning, and ideas for AI-GAs (AI-Generating Algorithms). We hope this paper inspires further research interest in Text Data Augmentation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s40537-021-00492-0

Cites

108 of 108 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 108 references

Research works printed but not linked: 36

  1. Tang R, Nogueira R, Zhang E, Gupta N, Cam P, Cho K, Lin J. Rapidly bootstrapping a question answering dataset for covid-19. 2020. arXiv:2004.11339. Accessed Jul 2021 · arXiv
  2. Kukačka J, Golkov V, Cremers D. Regularization for deep learning: a taxonomy 2017 . arXiv:1710.10686. Accessed Jul 2021 · arXiv
  3. Ho D, Liang E, Stoica I, Abbeel P, Chen X. Population based augmentation: efficient learning of augmentation policy schedules 2019. arXiv:1905.05393. Accessed Jul 2021 · arXiv
  4. Clune J. AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence 2019. arXiv:1905.10985. Accessed Jul 2021 · arXiv
  5. Such FP, Rawal A, Lehman J, Stanley KO, Clune J. Generative teaching networks: accelerating neural architecture search by learning to generate synthetic training data 2019. arXiv:1912.07768. Accessed Jul 2021 · arXiv
  6. Rawal A, Lehman J, Such FP, Clune J, Stanley KO. Synthetic petri dish: a novel surrogate model for rapid architecture search 2020. arXiv:2005.13092. Accessed Jul 2021 · arXiv
  7. Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler DM, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D. Language models are few-shot learners 2020. arXiv:2005.14165. Accessed Jul 2021 · arXiv
  8. van der Maaten L, Hinton G. Viualizing data using t-sne. J Mach Learn Res. 2008;9:2579–605.
  9. Levine S, Kumar A, Tucker G, Fu J. Offline Reinforcement learning: tutorial, review, and perspectives on open problems 2020. arXiv:2005.01643. Accessed Jul 2021 · arXiv
  10. Kaushik D, Hovy E, Lipton ZC. Learning the difference that makes a difference with counterfactually-augmented data 2019. arXiv:1909.12434. Accessed Jul 2021 · arXiv
  11. Weber L, Jumelet J, Bruni E, Hupkes D. Language modelling as a multi-task problem. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2049–2060. Association for Computational Linguistics, Online 2021. https://www.aclweb.org/anthology/2021.eacl-main.176. Accessed Jul 2021 · DOI
  12. Li MM, Huang K, Zitnik M. Representation learning for networks in biology and medicine: advancements, challenges, and opportunities 2021. arXiv:2104.04883. Accessed Jul 2021 · arXiv
  13. Kong K, Li G, Ding M, Wu Z, Zhu C, Ghanem B, Taylor G, Goldstein T. FLAG: adversarial data augmentation for graph neural networks 2020. arXiv:2010.09891. Accessed Jul 2021 · arXiv
  14. Gopalan A, Juan D-C, Magalhaes CI, Ferng C-S, Heydon A, Lu C-T, Pham P, Yu G, Fan Y, Wang Y. Neural structured learning: Training neural networks with structured signals. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. WSDM ’21, pp. 1150–1153. Association for Computing Machinery, New York, NY, USA 2021. https://doi.org/10.1145/3437963.3441666. · doi:10.1145/3437963.3441666
  15. Guo H, Mao Y, Zhang R. Augmenting data with mixup for sentence classification: an empirical study. arXiv:1905.08941 . Accessed Jul 2021 · arXiv
  16. Khosla P, Teterwak P, Wang C, Sarna A, Tian Y, Isola P, Maschinot A, Liu C, Krishnan D. Supervised contrastive learning 2020. arXiv:2004.11362. Accessed Jul 2021 · arXiv
  17. Minderer M, Bachem O, Houlsby N, Tschannen M. Automatic shortcut removal for self-supervised representation learning 2020. arXiv:2002.08822. Accessed Jul 2021 · arXiv
  18. Gao L, Biderman S, Black S, Golding L, Hoppe T, Foster C, Phang J, He H, Thite A, Nabeshima N, Presser S, Leahy C. The pile: an 800GB dataset of diverse text for language modeling 2020. arXiv:2101.00027. Accessed Jul 2021 · arXiv
  19. Scao TL, Rush AM. How many data points is a prompt worth? 2021. arXiv:2103.08493. Accessed Jul 2021 · arXiv
  20. Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, de Laroussilhe Q, Gesmundo A, Attariyan M, Gelly S. Parameter-efficient transfer learning for NLP 2019. arXiv:1902.00751. Accessed Jul 2021 · arXiv
  21. Arjovsky M. Out of Distribution generalization in machine learning 2021. arXiv:2103.02667. Accessed Jul 2021 · arXiv
  22. Chollet F. On the measure of intelligence 2019. arXiv:1911.01547. Accessed Jul 2021 · arXiv
  23. Koh PW, Sagawa S, Marklund H, Xie SM, Zhang M, Balsubramani A, Hu, W.-h., Yasunaga M, Phillips RL, Beery S, Leskovec J, Kundaje A, Pierson E, Levine S, Finn C, Liang P. Wilds: a benchmark of in-the-wild distribution shifts. arXiv:2012.07421. Accessed Jul 2021 · arXiv
  24. Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models 2020. arXiv:2006.11239. Accessed Jul 2021 · arXiv
  25. Garg S, Balakrishnan S, Kolter JZ, Lipton ZC. RATT: Leveraging unlabeled data to guarantee generalization 2021. arXiv:2105.00303. Accessed Jul 2021 · arXiv
  26. Zhang H, Zhang Z, Odena A, Lee H. Consistency regularization for generative adversarial networks. In: International Conference on Learning Representations 2020. https://openreview.net/forum?id=S1lxKlSKPH. Accessed Jul 2021
  27. Caron M, Misra I, Mairal J, Goyal P, Bojanowski P, Joulin A. Unsupervised learning of visual features by contrasting cluster assignments 2020. arXiv:2006.09882. Accessed Jul 2021 · arXiv
  28. Gunel B, Du J, Conneau A, Stoyanov V. Supervised contrastive learning for pre-trained language model fine-tuning. In: International Conference on Learning Representations 2021. https://openreview.net/forum?id=cu7IUiOhujH. Accessed Jul 2021
  29. Li L, Jamieson K, Rostamizadeh A, Gonina E, Hardt M, Recht B, Talwalkar A. A system for massively parallel hyperparameter tuning 2018. arXiv:1810.05934. Accessed Jul 2021 · arXiv
  30. Wang F, Zhang Y, Zheng Y, Ruan W. Gradient-guided dynamic efficient adversarial training. arXiv:2103.03076. Accessed Jul 2021 · arXiv
  31. Choi D, Passos A, Shallue CJ, Dahl GE. Faster neural network training with data echoing 2019. arXiv:1907.05550. Accessed Jul 2021 · arXiv
  32. Fort S, Brock A, Pascanu R, De S, Smith SL. Drawing multiple augmentation samples per image during training efficiently decreases test error 2021. arXiv:2105.13343. Accessed Jul 2021 · arXiv
  33. Kuchnik M, Smith V. Efficient augmentation via data subsampling. In: International Conference on Learning Representations 2019. https://openreview.net/forum?id=Byxpfh0cFm. Accessed Jul 2021
  34. Wei J, Huang C, Vosoughi S, Cheng Y, Xu S. Few-shot text classification with triplet networks, data augmentation, and curriculum learning 2021. arXiv:2103.07552. Accessed Jul 2021 · arXiv · doi:10.18653/v1/2021.naacl-main.434
  35. Jun H, Child R, Chen M, Schulman J, Ramesh A, Radford A, Sutskever I. Distribution augmentation for generative modeling. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 5006–5019. PMLR? 2020. http://proceedings.mlr.press/v119/jun20a.html. Accessed Jul 2021
  36. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ. Exploring the limits of transfer learning with a unified text-to-text transformer 2019. arXiv:1910.10683. Accessed Jul 2021 · arXiv

Cited by

See all 1,740

Metadata

Authors
Shorten, C., Khoshgoftaar, T. M., & Furht, B.
Venue
Journal of Big Data
Publisher
Springer Science and Business Media LLC
Published
2021-07-19
Volume
8
Issue
1
Pages
101-101
ISSN
2196-1115 (ISSN-L)
DOI
10.1186/s40537-021-00492-0
OpenAlex
W3176923149
Cited by
1,740
References
108
FWCI
118.83
Type
journal article
Language
en
Open access
gold
Licence
Funders
National Science Foundation (IIS-2027890)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Text Data Augmentation for Deep Learning. https://app.cochat.ai/papers/10.1186/s40537-021-00492-0
The paper
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021). Text Data Augmentation for Deep Learning. Journal of Big Data. https://doi.org/10.1186/s40537-021-00492-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.