GPT Understands, Too
Open accessAbstract
Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU). However, our preliminary study reveals that manual discrete prompts often lead to unstable performance -- e.g., changing a single word in the prompt might result in substantial performance drop. We propose a novel method P-Tuning that employs trainable continuous prompt embeddings in concatenation with discrete prompts. Empirically, P-Tuning not only stabilizes training by minimizing the gap between various discrete prompts, but also improves performance by a sizeable margin on a wide range of NLU tasks including LAMA and SuperGLUE. P-Tuning is generally effective for both frozen and tuned language models, under both the fully-supervised and few-shot settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
P-tuning represents prompt embeddings as the output of a BiLSTM to account for interaction between prompt tokens.
“To account for interaction between prompt tokens, they represent prompt embeddings as the output of a BiLSTM [35].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [77]
The Introduction (p. 1) says P-Tuning uses an LSTM or MLP prompt encoder to model dependencies between continuous prompt embeddings. Section 2.3 (p. 3) describes mapping trainable prompt embeddings to model inputs, but our copy does not specify a bidirectional LSTM; it also presents MLP and identity mappings as alternatives. (LLM, conf=0.85)
Experimental — accuracy not yet measuredP-tuning introduces task-related anchor tokens within the template for further improvement, and these anchor tokens are not tuned during training.
“P-tuning also introduces the use of task-related anchor tokens (such as”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [77]
Section 2.2 (p. 2) says P-Tuning concatenates discrete prompts with continuous prompts and that this combination performs better. The Introduction (p. 1) says the continuous prompts are updated by backpropagation, but our copy does not call the discrete tokens “anchor tokens” or explicitly establish that their embeddings are never tuned during training; the method also permits finetuning the language model. (LLM, conf=0.80)
Experimental — accuracy not yet measuredTo achieve strong SuperGLUE results, P-tuning has to be used in conjunction with model tuning, jointly updating both the prompt and the main model parameters, whereas prompt tuning keeps the original language model frozen.
“To achieve strong SuperGLUE results, P-tuning has to be used in conjunction with model tuning, that is, models jointly update both the prompt and the main model parameters, whereas our approach keeps the original language model frozen.”
From The Power of Scale for Parameter-Efficient Prompt Tuning (2021) · cites this paper as Liu et al. (2021)
The Experiments section (p. 3) says the paper's SuperGLUE setup jointly optimized the language-model parameters and continuous prompts, and the Introduction (p. 2) reports improved SuperGLUE results with the model fine-tuned. But the paper describes this as its experimental setup, not as something P-tuning *has to* do to achieve strong SuperGLUE results. The comparison with the citing authors' frozen-model approach is not a finding of this source. (LLM, conf=0.88)
Experimental — accuracy not yet measuredPrior soft prompt-tuning and discrete prompt optimization approaches become inapplicable when there is only API access to the LLM.
“These approaches become inapplicable when there is only API access to the LLM.”
From Large Language Models as Optimizers (2023) · cites this paper as Liu et al., 2021
The Introduction (p. 1) says P-Tuning updates continuous prompt embeddings by backpropagation, supporting the limitation for that soft-prompt method. But Section 2.1 (p. 2) also describes discrete-prompt search by corpus mining and pretrained generative models, not just gradient-based search. The paper does not say that discrete-prompt optimization as a whole becomes inapplicable with API-only access. (LLM, conf=0.76)
Experimental — accuracy not yet measuredLiu et al. propose P-tuning, where continuous prompts are learned by inserting trainable variables into the embedded input.
“Liu et al. [77] propose”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [77]
ChatGPT is a highly sophisticated chatbot capable of fulfilling a wide range of text-based requests, including answering simple questions and completing more advanced tasks such as generating thank you letters and guiding individuals through tough discussions about productivity issues.
“It is a highly sophisticated chatbot that is capable of fulfilling a wide range of text-based requests, including answering simple questions and completing more advanced tasks such as generating thank you letters and guiding individuals through tough discussions about productivity issues (Liu et al., 2021).”
From Chatting about ChatGPT: how may AI and GPT impact academia and libraries? (2023) · cites this paper as Liu et al., 2021
The Introduction (p. 1) and Setup (p. 4) discuss prompting language models and evaluating question-answering tasks, not ChatGPT as a chatbot. Searches for thank-you letters, productivity discussions, and conversational assistance found no passage supporting the specific capabilities attributed to ChatGPT. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 39
- Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 . · arXiv
- Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019a. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2924-2936.
- Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019b. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341 . · arXiv
- Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop , pages 177-190. Springer.
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 . · arXiv
- Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung , volume 23, pages 107-124.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . · arXiv
- Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723 . · arXiv
- John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) . Association for Computational Linguistics.
- Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-factr: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 5943-5959.
- Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020b. How can we know what language models know? Transactions of the Association for Computational Linguistics , 8:423-438.
- Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 252-262.
- Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. ArXiv , abs/2104.08691. · arXiv
- Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning . Citeseer.
- Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 . · arXiv
- Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt3 ? arXiv preprint arXiv:2101.06804 . · arXiv
- Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, and Jie Tang. 2020. Selfsupervised learning: Generative or contrastive. arXiv preprint arXiv:2006.08218 , 1(2). · arXiv
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 . · arXiv
- Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. CoRR , abs/2104.08786. · arXiv
- Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2020. How context affects language models' factual predictions. arXiv preprint arXiv:2005.04611 . · arXiv
- Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066 . · arXiv
- Mohammad Taher Pilehvar and José Camacho-Collados. 2018. Wic: 10, 000 example pairs for evaluating context-sensitive representations. CoRR , abs/1808.09121. · arXiv
- Guanghui Qin and J. Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. ArXiv , abs/2104.06599. · arXiv
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 . · arXiv
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning , pages 90-95.
- Timo Schick and Hinrich Schütze. 2020. It's not just size that matters: Small language models are also few-shot learners. Computing Research Repository , arXiv:2009.07118. · arXiv
- Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 . · arXiv
- Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. arXiv preprint arXiv:1906.05714 . · arXiv
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537 . · arXiv
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In NeurIPS 2019 , pages 3261-3275.
- Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. ArXiv , abs/1804.07461. · arXiv
- Chenguang Wang, Xiao Liu, and Dawn Song. 2020. Language models are open knowledge graphs. arXiv preprint arXiv:2010.11967 . · arXiv
- Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 . · arXiv
- Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885 . · arXiv
- Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. 2020. Revisiting few-sample BERT fine-tuning. CoRR , abs/2006.05987. · arXiv
- Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021a. Calibrate before use: Improving few-shot performance of language models. CoRR , abs/2102.09690. · arXiv
- Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021b. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690 . · arXiv
- Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [mask]: Learning vs. learning to recall. ArXiv , abs/2104.05240. · arXiv
Cited by
Sorted by citations
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Chatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355 · see citing works
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919 · see citing works
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
PTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2105.11259 · Cited by 478 · see citing works
“Moreover, human-picked prompts have achieved promising results on few-class classification tasks such as sentiment classification and natural language inference (Schick and Schütze, 2021; Liu et al., 2021).”
BackgroundInfluentialPersonalized Prompt Learning for Explainable Recommendation
Li, L., Zhang, Y., & Chen, L. (2023) · ACM Transactions on Information Systems · doi:10.1145/3580488 · Cited by 172 · see citing works
Large Language Models as Optimizers
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023) · arXiv · doi:10.48550/arxiv.2309.03409 · Cited by 151 · see citing works
OpenPrompt: An Open-source Framework for Prompt-learning
Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H.-T., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2111.01998 · Cited by 60 · see citing works
“A series of studies of prompt-learning (Liu et al., 2021a) have been proposed to investigate the strategies of constructing templates (Schick and Sch¨utze, 2021; Gao et al., 2021; Liu et al., 2021b), verbalizers (Hu et al., 2021), optimization (Lester et al., 2021), and application (Li and Liang,…”
BackgroundPetals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2023.acl-demo.54 · Cited by 52 · see citing works
ChatGPT in Academia: University Students’ Attitudes Towards the use of ChatGPT and Plagiarism
Fajt, B., & Schiller, E. (2025) · Journal of Academic Ethics · doi:10.1007/s10805-025-09603-5 · Cited by 47 · see citing works
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Borzunov, A., Ryabinin, M., Chumachenko, A., Baranchuk, D., Dettmers, T., Belkada, Y., Samygin, P., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2312.08361 · Cited by 35 · see citing works
Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution
Fernando, C., Banarse, D., Michalewski, H., Osindero, S., & Rocktäschel, T. (2023) · arXiv · doi:10.48550/arxiv.2309.16797 · Cited by 31 · see citing works
Petals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Ryabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2209.01188 · Cited by 6 · see citing works
ConceptFormer: Towards Efficient Use of Knowledge-Graph Embeddings in Large Language Models
Barmettler, J., Bernstein, A., & Rossetto, L. (2025) · arXiv · doi:10.48550/arxiv.2504.07624
- Authors
- Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J.
- Affiliations
- Tsinghua University (CN)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Yang, Zhilin
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 5
- Pages
- 208-215
- ISSN
- 2666-6510 (ISSN-L)
- arXiv
- 2103.10385
- OpenAlex
- W3139080614
- Cited by
- 908
- References
- 0
- Citations by year
- 2026: 78 · 2025: 251 · 2024: 257 · 2023: 189 · 2022: 88 · 2021: 43 · 2020: 2
- FWCI
- 73.06
- Type
- journal article
- Language
- it
- Open access
- gold
- doi.org
- Licence
- Locations
- AI Open (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- natural language understanding, pre-trained language models, prompt tuning, few-shot learning, training stability, lamas, SuperGLUE
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
