Synthetic Text as Data: On Usefulness and Limitations
Open accessAbstract
This study investigates the utility of GPT-generated text as a training resource in supervised learning, focusing on two perspectives: its effectiveness as an augmentation tool in data-scarce or class-imbalanced settings and its potential as a substitute for human-written data. Using MBTI personality classification as a benchmark task, we conducted controlled experiments under both class imbalance and few-shot learning conditions. Results showed that GPT-generated text could improve classification performance when used to supplement underrepresented classes. However, when synthetic data fully replace real data, performance declines significantly—particularly in tasks requiring fine-grained semantic distinctions. Further analysis reveals that GPT outputs often capture only partial personality traits, enabling coarse-level classification but falling short in nuanced cases. These findings suggest that GPT-generated text can function as a conditional training resource, with its effectiveness closely tied to the granularity of the classification task.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Choi et al. showed that while GPT-generated text can serve as a practical augmentation tool in data-scarce settings, it falls short as a substitute for real data in fine-grained classification tasks, as synthetic outputs often capture only partial class-discriminative signals.
“Complementarily, Choi et al. [31] showed that while GPT-generated text can serve as a practical augmentation tool in data-scarce settings, it falls short as a substitute for real data in finegrained classification tasks, as synthetic outputs often capture only partial class-discriminative signals.”
From Evaluation of NeMo Guardrails as a Firewall for User–LLM Interaction (2026) · cites this paper as [31]
Choi et al. [31] showed that while GPT-generated text can serve as a practical augmentation tool in data-scarce settings, it falls short as a substitute for real data in fine-grained classification tasks.
“Complementarily, Choi et al. [31] showed that while GPT-generated text can serve as a practical augmentation tool in data-scarce settings, it falls short as a substitute for real data in fine-grained classification tasks-as synthetic outputs often capture only partial class-discriminative signals, enabling coarse-level distinctions but failing in more nuanced cases.”
From Evaluation of NeMo Guardrails as a Firewall for User–LLM Interaction (2026) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N., & Choi, Y. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.501 · Cited by 118 · see citing works
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N. A., & Choi, Y. (2021) · arXiv · doi:10.48550/arxiv.2107.01294 · Cited by 90 · see citing works
Flagged first, then by citations
Research works printed but not linked: 19
- Feng, S.Y.; Gangal, V.; Wei, J.; Chandar, S.; Vosoughi, S.; Mitamura, T.; Hovy, E. A survey of data augmentation approaches for NLP. arXiv 2021, arXiv:2105.03075. · arXiv
- Sennrich, R.; Haddow, B.; Birch, A. Improving neural machine translation models with monolingual data. arXiv 2015, arXiv:1511.06709. · arXiv
- Bayer, M.; Kaufhold, M.A.; Reuter, C. A survey on data augmentation for text classification. ACM Comput. Surv. 2022, 55, 1-39.
- Dai, H.; Liu, Z.; Liao, W.; Huang, X.; Cao, Y.; Wu, Z.; Zhao, L.; Xu, S.; Liu, W.; Liu, N.; et al. AugGPT: Leveraging ChatGPT for Text Data Augmentation. arXiv 2023, arXiv:2302.13007. · arXiv
- Abaskohi, A.; Rothe, S.; Yaghoobzadeh, Y. LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning. arXiv 2023, arXiv:2305.18169. · arXiv
- Oh, S.; Jung, W.; Lee, S.A. Data augmentation for neural machine translation using generative language model. arXiv 2023, arXiv:2307.16833. · arXiv
- Cegin, J.; Simko, J.; Brusilovsky, P. ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness. arXiv 2023, arXiv:2305.12947. · arXiv
- Nori, H.; King, N.; McKinney, S.M.; Carignan, D.; Horvitz, E. Capabilities of gpt-4 on medical challenge problems. arXiv 2023, arXiv:2303.13375. · arXiv
- Lammerse, M.; Hassan, S.Z.; Sabet, S.S.; Riegler, M.A.; Halvorsen, P. Human vs. GPT-3: The challenges of extracting emotions from child responses. In Proceedings of the 2022 14th International Conference on Quality of Multimedia Experience (QoMEX), Lippstadt, Germany, 5-7 September 2022; pp. 1-4.
- Zhang, J.; Qiao, D.; Yang, M.; Wei, Q. Regurgitative Training: The Value of Real Data in Training Large Language Models. arXiv 2024, arXiv:2407.12835. · arXiv
- McCoy, R.T.; Smolensky, P.; Linzen, T.; Gao, J.; Celikyilmaz, A. How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN. arXiv 2021, arXiv:2111.09509. · arXiv
- Meng, Q.; Ke, G.; Wang, T.; Chen, W.; Ye, Q.; Ma, Z.M.; Liu, T.Y. A communication-efficient parallel algorithm for decision tree. Adv. Neural Inf. Process. Syst. 2016, 29, 1279-1287.
- Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; Liu, T.Y. Lightgbm: A highly efficient gradient boosting decision tree. Adv. Neural Inf. Process. Syst. 2017, 30, 3149-3157.
- Dorogush, A.V.; Ershov, V.; Gulin, A. CatBoost: gradient boosting with categorical features support. arXiv 2018, arXiv:1810.11363. · arXiv
- Chen, T.; Guestrin, C. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 9 March 2016; pp. 785-794.
- Caruana, R.; Niculescu-Mizil, A.; Crew, G.; Ksikes, A. Ensemble selection from libraries of models. In Proceedings of the Twenty-First International Conference on Machine Learning, Banff, AB, Canada, 4-8 July 2004; p. 18.
- Fakoor, R.; Mueller, J.W.; Erickson, N.; Chaudhari, P.; Smola, A.J. Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation. In Proceedings of the NIPS'20: Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 6-12 December 2020.
- Shi, X.; Mueller, J.; Erickson, N.; Li, M.; Smola, A. Multimodal AutoML on Structured Tables with Text Fields. In Proceedings of the 8th ICML Workshop on Automated Machine Learning (AutoML), Online, 23 July 2021.
- Xu, Z.; Peng, K.; Ding, L.; Tao, D.; Lu, X. Take care of your prompt bias! Investigating and. mitigating prompt bias in factual knowledge extraction. arXiv 2024, arXiv:2403.09963. · arXiv
Cited by
Sorted by citations
Evaluation of NeMo Guardrails as a Firewall for User–LLM Interaction
Azambuja, A. J., Guilherme, M., Castro, J. V. F. D., Lima, J. P. D. O., Oliveira, L. B., & Soares, A. D. S. (2026) · Future Internet · doi:10.3390/fi18050252 · Cited by 2 · see citing works
- Authors
- Choi, S., Sim, J., & Choi, G.
- Affiliations
- Research and Development Center, Nanum Space Co., Ltd., Jeonju 54907, Republic of Korea
- Jeonbuk National University (KR)
- Corresponding author
- Choi, Guebin
- ORCID
- Choi, Guebin
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2025-05-13
- Volume
- 15
- Issue
- 10
- Pages
- 5460
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4410316383
- Cited by
- 4
- References
- 2
- Citations by year
- 2026: 2 · 2025: 2
- FWCI
- 1.62
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Research Foundation of Korea (NRF) (RS-2023-00249743)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- synthetic data, data augmentation, class imbalance, few-shot learning, fine-grained classification, supervised learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
