The DOI matches its Crossref record; 49 of the 54 references it cites were found in a registry or an index.
Generating Datasets with Pretrained Language Models
Open accessAbstract
To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pretraining objectives or finetuned on a large set of labeled text pairs.While the latter approach typically outperforms the former, it requires great human effort to generate suitable datasets of sufficient size.In this paper, we show how PLMs can be leveraged to obtain high-quality sentence embeddings without the need for labeled data, finetuning or modifications to the pretraining objective: We utilize the generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch, which we then use for finetuning much smaller and more efficient models.Our fully unsupervised approach outperforms strong baselines on several semantic textual similarity datasets.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Schick and Schütze [119] propose to use pre-trained LMs to generate datasets given certain instructions.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [119]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
(2017) · Cited by 11,625
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645
Research works printed but not linked: 1
- Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Cited by
Sorted by citations
Items 101–129 of 129
A Simple Method to Improve the Performance of Small Pre-trained Language Models on Few-shot Tasks
Zhang, Y., Wu, C., Shi, R., & Zhang, Y. (2024) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd61410.2024.10580281 · Cited by 1
EBFP: Example-Based Further Pre-training
Zhang, Y., Wu, C., Zhao, W., & Zhang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5669-8_35 · Cited by 1
AraT5GQA: Arabic Question Answering model using automatic generated dataset
Nakhleh, S., Mustafa, A., & Najadat, H. (2024) · International Conference on Information and Communication Systems (ICICS) · doi:10.1109/icics63486.2024.10638274 · Cited by 1
Towards Human-Like Virtual Beings: Simulating Human Behavior in 3D Scenes
Liang, C., Wang, W., & Yang, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01001 · Cited by 1
Plot Writing From Pre-Trained Language Models
Jin, Y., Kadam, V., & Wanvarie, D. (2022) · International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.5 · Cited by 7
ConvFiT: Conversational Fine-Tuning of Pretrained Language ModelsPreprint
Vulić, I., Su, P.-H., Coope, S., Gerz, D., Budzianowski, P., Casanueva, I., Mrkšić, N., & Wen, T.-H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10126 · Cited by 3
Training Conversational Agents with Generative Conversational NetworksPreprint
Lin, Y.-T. R., Papangelis, A., Kim, S., & Hakkani-Tür, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08383
Zaninello, A., & Magnini, B. (2023) · NLRSE · doi:10.18653/v1/2023.nlrse-1.3
KEST: Kernel Distance Based Efficient Self-Training for Improving Controllable Text Generation
Feng, Y., Yi, X., Lakshmanan, L. V. S., & Xie, X. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/561 · Cited by 1
A Glitch in the Matrix? Locating and Detecting Language Model Grounding with FakepediaPreprint
Monea, G., Peyrard, M., Josifoski, M., Chaudhary, V., Eisner, J. M., Kıcıman, E., Palangi, H., Patra, B., & West, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.02073
End-to-End Autoregressive Retrieval via Bootstrapping for Smart Reply Systems
Towle, B., & Zhou, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.510
Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning
Shivagunde, N., Lialin, V., & Rumshisky, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.130 · Cited by 4
Generate to Understand for Representation in One Pre-training Stage
Xue, C., Zhong, X., & Liu, X. (2023) · International Conference on Information Systems and Computing Technology (ISCTech) · doi:10.1109/isctech60480.2023.00054
Generate-then-Revise: An Effective Synthetic Training Data Generation Framework for Event Detection
Du, H., Sun, H., Liu, P., & Yu, D. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_4
Enhancing Short Text Semantic Similarity with Word2Vec and Maximum Matching
Xiao, Y., & Huang, G. (2024) · Asia-Pacific Artificial Intelligence and Big Data Forum (AIBDF) · doi:10.1145/3718491.3718605
Military Fine-tuning Dataset Construction Method for Chinese Large Language Model
Dong, Z., Yu, T., & Wu, Y. (2024) · International Conference on Computers and Artificial Intelligence Technology (CAIT) · doi:10.1109/cait64506.2024.10963020
Kandasamy, L., Bajaj, A., Dhindsa, G. S., & Tagalpallewar, A. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-7260-9.ch013
Qi, Z. (2024) · Young Researchers' Roundtable on Spoken Dialogue Systems (YRRSDS) · doi:10.18653/v1/2024.yrrsds-1.31
Causal Relation-Aware Data Augmentation for Legal Textual Entailment
Chu, H., Chu, H., Nguyen, P., & Nguyen, L.-M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97141-9_27
Text Generator and Text Discriminator for NIST GenAI T2T Challenge
Sarker, S., Qian, L., & Dong, X. (2025) · International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc64931.2025.11077532
Leveled Learning: An Interpolation-Based Data Augmentation Method on Few-Shot Text Classification
Wang, Y., Xu, F., Wang, B., & Liu, P. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7008-6_13
Kim, R. S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5717409
DiverSeed: Integrating Active Learning for Target Domain Data Generation in Instruction Tuning
Gao, J., Qi, M., Gao, X., Xiang, S., Ji, K., Ruan, J., Liu, T., & Fu, Y. (2025) · Machine Learning · doi:10.1007/s10994-025-06926-7
Qi, Z., & Inaba, M. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.1030
AdaGen: Adaptive Domain-Specific Classification via Zero-Shot Dataset Generation
Xu, Z.-H., Liang, F.-Y., Zhou, H.-Y., & Wang, C. (2025) · Proceedings · doi:10.1109/icdm65498.2025.00088
TaiChiGPT: complex sports action generation based on large language models
Li, J., Ran, K., Ma, Y., Xie, H., & Wu, Y. (2026) · The Visual Computer · doi:10.1007/s00371-026-04428-8
Improving the Efficiency of Lossless Embedding in Text Steganography
Teterev, N. V., & Levina, A. B. (2026) · International Russian Smart Industry Conference (SmartIndustryCon) · doi:10.1109/smartindustrycon68821.2026.11492963
Synthetic Interaction Data for Scalable Personalization in Large Language Models
Ma, Y., Huang, Y., Wang, W., Luo, X., Zhang, X., & Feuerriegel, S. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817563 · Cited by 8
Salamea, C., Vasquez-Avila, J., Viñanzaca, J. F. V., & Sacoto-Cabrera, E. J. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-27538-7_33
Metadata
- Authors
- Schick, T., & Schütze, H.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- ORCID
- Schütze, Hinrich
- Venue
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Published
- 2021
- Pages
- 6943-6951
- OpenAlex
- W3154863804
- Cited by
- 122
- References
- 47
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 27 · 2023: 39 · 2022: 26 · 2021: 8
- FWCI
- 11.55
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Open access LMU (Ludwid Maxmilian's Universitat Munchen) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Universitätsbibliothek der LMU
- Funders
- H2020 European Research Council (740516)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- sentence embeddings, pre-trained language models, semantic textual similarity, generative language models, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
