Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Shin et al. learn a logistic classifier using the contextualized representation of the [Z] token as input and select the top-k tokens that achieve the highest probability score.

“Shin et al. [125] learn a logistic classifier using the contextualized representation of the [Z] token as input.In the search step, they select the top-k tokens that achieve the highest probability score using the learned logistic classifier in the first step.”

• cites [125] 125. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 7699-7715. https://doi.org/10.18653/v1/2021.emnlp-main.608 sourceDisputed

  The “Dynamic Prompt Creation” section describes generating outputs with a language model, using beam search and a nextTokens function to filter valid continuations. This is a different search mechanism from ranking top-k tokens with a logistic classifier trained on a [Z] representation; the cited paper describes no such classifier. (LLM, conf=0.83)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Jiang et al. observed that the probabilities of pre-trained LMs (e.g., BART, T5, and GPT-2) on QA tasks are well calibrated.

“Jiang et al. [51] observed the probabilities of pre-trained LMs (e.g., BART, T5, and GPT-2) on QA tasks are well calibrated.”

• cites [51] 51. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? On the calibration of language models for question answering. Trans. Assoc. Comput. Ling. 9 (09 2021), 962-977. https://doi.org/10.1162/tacl_a_00407 sourceDisputed

  The Conclusion says that LM-based QA models tend to be poorly calibrated in their probability estimates, the opposite of the citing sentence. Section 5.5 also reports that BART and GPT-2 have lower calibration performance than T5; it does not describe these models as well calibrated. (LLM, conf=0.95)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Guo et al. use reinforcement learning to generate prompts to control the text generation process.

“Guo et al. [36] use reinforcement learning [132] to generate prompts to control the text generation process.”

• cites [36] 36. Han Guo, Bowen Tan, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. 2021. Text generation with efficient (soft) Q-learning. arXiv:2106.07704. Retrieved from https://arxiv.org/abs/2106.07704. sourceNot checked — no full text of the source
• cites [132] 132. Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE 3.0: Large-scale knowledge enhanced pre- training for language understanding and generation. arXiv:2107.02137. Retrieved from https://arxiv.org/abs/2107. 02137. sourceCould not verify

  The cited paper describes ERNIE 3.0’s language-modeling pre-training in §3.2.1 (p. 5) and its use of prompts for evaluation in §4.3.1 (p. 13). Neither passage addresses reinforcement learning to generate prompts or Guo et al.’s method; searches for those terms found no passage that does. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S2 (checkable).

Creating and experimenting with prompts is an art that takes time and experience, particularly for complicated tasks such as semantic parsing.

“Creating and experimenting with these prompts is an art that takes time and experience, particularly for some complicated tasks such as semantic parsing [124]”

• cites [124] 124. Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot se- mantic parsers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), sourcePartly supported

  The Dynamic Prompt Creation section describes designing and selecting example-based prompts for semantic parsing. It does not say that creating and experimenting with prompts is an art or requires substantial time and experience; the Introduction instead emphasizes rapid prototyping. Page numbers are unavailable in our copy. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: Creating and experimenting with prompts is an art that takes time and experience.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Davison et al. investigate knowledge base completion and design a template for an input head-relation-tail triple using LMs, scoring filled prompts with a unidirectional LM and selecting the one with highest LM probability.

“Davison et al. [19] investigate the task of knowledge base completion and design a template for an input (head-relation-tail triple) using LMs.They first handcraft a set of templates as potential candidates and fill the input and answer slots to form a filled prompt.They then use a unidirectional LM to score those filled prompts, selecting the one with the highest LM probability.”

• cites [19] 19. Joe Davison, Joshua Feldman, and Alexander M. Rush. 2019. Commonsense knowledge mining from pretrained mod- els. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP'19), Kentaro Inui, Jing Jiang, Vincent Ng, and Xi- aojun Wan (Eds.). Association for Computational Linguistics, 1173-1178. https://doi.org/10.18653/v1/D19-1109 sourceSupported

May combine two claims.

Shin et al. automatically search for template tokens using downstream application training samples and demonstrate strong performance in prompting scenarios.

“Shin et al. [125] automatically search for template tokens using downstream application training samples and demonstrates strong performance in prompting scenarios.”

• cites [125] 125. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 7699-7715. https://doi.org/10.18653/v1/2021.emnlp-main.608 sourcePartly supported

  The Dynamic Prompt Creation section says Shin et al. automatically select relevant training examples for GPT-3 prompts, not that they search for template tokens. The Overnight results section reports performance comparable to a state-of-the-art method on some domains with substantially less training data, supporting the performance part of the claim. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Jiang et al.'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.

“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”

• cites [52] 52. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Trans. Assoc. Comput. Ling. 8 (2020), 423-438. https://doi.org/10.1162/tacl_a_00324 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Schick and Schütze and Schick et al. find tokens containing at least two alphabetic characters that are frequent in a large unlabeled dataset.

“Schick and Schütze [117] and Schick et al. [115] find tokens containing at least two alphabetic characters that are frequent in a large unlabeled dataset.”

• cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 sourceSupported
• cites [115] 115. Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifying words that can serve as la- bels for few-shot text classification. In Proceedings of the 28th International Conference on Computational Linguistics (COLING'20), Donia Scott, Núria Bel, and Chengqing Zong (Eds.). International Committee on Computational Lin- guistics, 5569-5578. https://doi.org/10.18653/v1/2020.coling-main.488 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

P-tuning represents prompt embeddings as the output of a BiLSTM to account for interaction between prompt tokens.

“To account for interaction between prompt tokens, they represent prompt embeddings as the output of a BiLSTM [35].”

• cites [77] 77. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv:2103.10385. Retrieved from https://arxiv.org/abs/2103.10385. sourcePartly supported

  The Introduction (p. 1) says P-Tuning uses an LSTM or MLP prompt encoder to model dependencies between continuous prompt embeddings. Section 2.3 (p. 3) describes mapping trainable prompt embeddings to model inputs, but our copy does not specify a bidirectional LSTM; it also presents MLP and identity mappings as alternatives. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [35] 35. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 6645-6649. https://doi.org/10.1109/ICASSP.2013.6638947 sourcePartly supported

  Section 2 (p. 2) explains that bidirectional LSTM processes a sequence in both directions to access context, and Section 4 (p. 4) reports its use in the paper’s speech-recognition networks. The paper does not discuss P-tuning, prompt embeddings, or interactions between prompt tokens, so it supports the BiLSTM method but not its stated application. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P-tuning introduces task-related anchor tokens within the template for further improvement, and these anchor tokens are not tuned during training.

“P-tuning also introduces the use of task-related anchor tokens (such as”

• cites [77] 77. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv:2103.10385. Retrieved from https://arxiv.org/abs/2103.10385. sourcePartly supported

  Section 2.2 (p. 2) says P-Tuning concatenates discrete prompts with continuous prompts and that this combination performs better. The Introduction (p. 1) says the continuous prompts are updated by backpropagation, but our copy does not call the discrete tokens “anchor tokens” or explicitly establish that their embeddings are never tuned during training; the method also permits finetuning the language model. (LLM, conf=0.80)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Longer phrasal or sentential answers are often used in language generation tasks and in multiple-choice question answering.

“Longer phrasal or sentential answers are often used in language generation tasks [105] and in other tasks such as multiple-choice question answering (where the scores of multiple phrases are compared against each-other; Khashabi et al. [55]).”

• cites [105] 105. Alec Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsu- pervised multitask learners.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [55] 55. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020. Association for Computational Linguistics, 1896-1907. https://doi.org/10.18653/v1/ 2020.findings-emnlp.171 sourcePartly supported

  Section 3.1 describes multiple-choice QA as choosing among candidate answers, and Table 2 reports multi-token average answer lengths for several multiple-choice datasets. The cited paper therefore supports the use of phrasal answers in multiple-choice QA, but the passages do not establish the parenthetical claim that scores of multiple phrases are compared. The language-generation part is attached to a different citation. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Prompt ensembling is connected to ensembling methods that are used to combine together multiple systems, which have a long history in machine learning.

“Prompt ensembling is connected to ensembling methods that are used to combine together multiple systems, which have a long history in machine learning [24,133,153].”

• cites [24] 24. Kevin Duh, Katsuhito Sudoh, Xianchao Wu, Hajime Tsukada, and Masaaki Nagata. 2011. Generalized minimum bayes risk system combination. In Proceedings of the 5th International Joint Conference on Natural Language Processing. 1356-1360.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [133] 133. Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [153] 153. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT'21), Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (Eds.). Association for Com- putational Linguistics, 5017-5033. https://doi.org/10.18653/v1/2021.naacl-main.398 sourceCould not verify

  The Method Prompt Data-driven? section describes individual prompt methods, and §2.3 describes optimizing a prompt for a relation. Searches for “prompt ensembling,” combining predictions across prompts, and “ensembling” found no passage connecting prompt ensembling to methods for combining multiple systems or discussing their history in machine learning. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Hambardzumyan et al. introduce the use of a continuous template whose shape follows a manual prompt template.

“Similarly, Hambardzumyan et al. [40] introduce the use of a continuous template whose shape follows a manual prompt template.”

• cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 sourcePartly supported

  Section 3 (WARP) describes a continuous prompt: trainable embeddings are inserted into predefined template positions. The paper also describes initializing prompts from manually designed Prompt-Verbalizer Patterns, but treats this as an option, not a requirement; Section 6.2 says WARP succeeds without manual initialization. The claim overstates the extent to which the continuous template follows a manual prompt template. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Answer paraphrasing methods start with an initial answer space Z and then use paraphrasing to expand this answer space to broaden its coverage.

“These methods start with an initial answer space Z and then use paraphrasing to expand this answer space to broaden its coverage [51].”

• cites [51] 51. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? On the calibration of language models for question answering. Trans. Assoc. Comput. Ling. 9 (09 2021), 962-977. https://doi.org/10.1162/tacl_a_00407 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Cui et al. manually design lists such as person, location, and so on, for named entity recognition tasks.

“Cui et al. [17] manually design lists such as”

• cites [17] 17. Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021. Template-based named entity recognition using BART. arXiv:2106.01760 [cs.CL]. Retrieved from https://arxiv.org/abs/2106.01760. sourcePartly supported

  Section 4.1 (page unavailable) says Cui et al. manually create templates, map entity labels such as LOC to words such as “location,” and form a list of templates. The Introduction gives “person” and “location” as template values. The paper does not say they manually designed the list of entity categories themselves; it describes those categories as pre-defined. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Qin and Eisner propose to learn a mixture of soft templates for each input where the weights and parameters for each template are jointly learned using training samples.

“Qin and Eisner [103] propose to learn a mixture of soft templates for each input where the weights and parameters for each template are jointly learned using training samples.”

• cites [103] 103. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Pro- ceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 5203-5212. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Jiang et al. specifically use a back-translation method to generate a list of multiple paraphrased answers.

“This paraphrasing can be performed using any method, but Jiang et al. [51] specifically use a back-translation method, first translating into another language then back to generate a list of multiple paraphrased answers.”

• cites [51] 51. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? On the calibration of language models for question answering. Trans. Assoc. Comput. Ling. 9 (09 2021), 962-977. https://doi.org/10.1162/tacl_a_00407 sourceSupported

May combine two claims.

Hambardzumyan et al. assign a virtual token for each class label and optimize the token embedding for each class together with prompt token embeddings.

“Hambardzumyan et al. [40] assign a virtual token for each class label and optimize the token embedding for each class together with prompt token embeddings.”

• cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Zhong et al. first define a template using a discrete search method such as AutoPrompt, initialize virtual tokens based on this discovered prompt, and then fine-tune the embeddings to increase task accuracy.

“Zhong et al. [152] first define a template using a discrete search method such as AutoPrompt [125]'s, initialize virtual tokens based on this discovered prompt, and then fine-tune the embeddings to increase task accuracy.”

• cites [152] 152. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML'21), Proceedings of Machine Learning Research, Vol. 139, Marina Meila and Tong Zhang (Eds.). PMLR, 12697-12706. http:// proceedings.mlr.press/v139/zhao21c.html. sourceCould not verify

  The Background and Experimental Setup (p. 2) describes prompt templates, but not the attributed sequence of discrete search, virtual-token initialization, and embedding fine-tuning. The Discussion (p. 7) says the authors use a fixed language model without fine-tuning. Searches for Zhong, virtual tokens, AutoPrompt-based initialization, and embedding fine-tuning found no passage addressing that sequence. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [125] 125. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 7699-7715. https://doi.org/10.18653/v1/2021.emnlp-main.608 sourceCould not verify

  The “Dynamic Prompt Creation” section describes selecting examples for GPT-3 prompts and, separately, fine-tuning GPT-2 XL or BART. Searches for Zhong, AutoPrompt, virtual tokens, and embedding initialization found no passage describing the attributed method. Our copy therefore does not address the claim about Zhong et al. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Li and Liang observe that continuous prefix-based learning is more sensitive to different initialization in low-data settings than the use of discrete prompts with real words.

“Experimentally, Li and Liang [71] observe that such continuous prefix-based learning is more sensitive to different initialization in low-data settings than the use of discrete prompts with real words.”

• cites [71] 71. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP'21), Volume 1: Long Papers, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, 4582-4597. https://doi.org/10.18653/v1/2021.acl- long.353 sourcePartly supported

  Section 7.4 (Initialization; page unavailable) reports that prefix initialization has a large impact in low-data settings: random initialization gives low performance and high variance, while initialization from real-word activations improves generation. It does not compare that sensitivity with the use of discrete prompts made of real words; those are a different method, discussed in Sections 2 and 7.2. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Prefix Tuning prepends a sequence of continuous task-specific vectors to the input while keeping the LM parameters frozen.

“Prefix Tuning [71] is a method that prepends a sequence of continuous task-specific vectors to the input, while keeping the LM parameters frozen.”

• cites [71] 71. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP'21), Volume 1: Long Papers, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, 4582-4597. https://doi.org/10.18653/v1/2021.acl- long.353 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Jiang et al. first filter their prompts by selecting K prompts that achieve the highest accuracy on the training set and then use the average log probabilities obtained from the top K prompts to calculate the probability for a single token at [Z] position when performing factual probing tasks.

“Jiang et al. [52] first filter their prompts by selecting K prompts that achieve the highest accuracy on the training set and then use the average log probabilities obtained from the top K prompts to calculate the probability for a single token at [Z] position when performing factual probing tasks.”

• cites [52] 52. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Trans. Assoc. Comput. Ling. 8 (2020), 423-438. https://doi.org/10.1162/tacl_a_00324 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Yuan et al. formulate text generation evaluation as a text generation problem and take the average of the final generation scores obtained using different prompts.

“When performing text generation evaluation, Yuan et al. [147] formulates this task as a text generation problem and take the average of the final generation scores obtained using different prompts.”

• cites [147] 147. Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of knowledge-enhanced text generation (unpublished). https://doi.org/10.1145/3512467 sourceCould not verify

  The survey’s A.1 Evaluation Metrics section (p. 40) describes BLEU, ROUGE, and Distinct, while its benchmark discussion (p. 29) concerns dataset selection. Searches for Yuan, prompt-based evaluation, and averaging generation scores found no passage in our copy addressing the attributed method. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

May combine two claims.

Since the answer tokens are optimized directly in the embedding space, they do not make use of the embeddings learned by the LM and instead learn an embedding from scratch for each label.

“Since the answer tokens are optimized directly in the embedding space, they do not make use of the embeddings learned by the LM and instead learn an embedding from scratch for each label.”

• cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 sourcePartly supported

  Section 3 (WARP; page unavailable) says the verbalizer embeddings are optimized directly in continuous embedding space, with one trainable embedding per class, and that the model's word-embedding decoder is omitted. But Appendix B (Learned Tokens; page unavailable) says verbalizer embeddings can be initialized from existing verbalizer-token embeddings, so the claim that each label embedding is learned from scratch without using LM-learned embeddings is too absolute. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In many cases, the answer space Z is the space of all tokens, fixed-length spans, or token sequences, and it is most common to directly map answer z to the final output y using the identity mapping.

“In many cases, the answer space Z is the space of all tokens [100], fixed-length spans [50], or token sequences [105].In these cases, it is most common to directly map answer z to the final output y using the identity mapping.”

• cites [100] 100. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP'19). As- sociation for Computational Linguistics, 2463-2473. https://doi.org/10.18653/v1/D19-1250 sourceSupported
• cites [50] 50. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'20). Association for Computational Linguistics, 5943-5959. https://doi.org/ 10.18653/v1/2020.emnlp-main.479 sourceSupported
• cites [105] 105. Alec Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsu- pervised multitask learners.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Liu et al. propose P-tuning, where continuous prompts are learned by inserting trainable variables into the embedded input.

“Liu et al. [77] propose”

• cites [77] 77. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv:2103.10385. Retrieved from https://arxiv.org/abs/2103.10385. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Lester et al. prepend the input sequence with special tokens to form a template and tune the embeddings of these tokens directly.

“Similarly, Lester et al. [67] prepend the input sequence with special tokens to form a template and tune the embeddings of these tokens directly.”

• cites [67] 67. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 3045-3059. https://doi.org/10.18653/v1/2021.emnlp-main.243 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

With regards to multiple-choice question answering, it is common to use an LM to calculate the probability of an output among multiple choices, with Zweig et al. being an early example.

“With regards to multiple-choice question answering, it is common to use an LM to calculate the probability of an output among multiple choices, with Zweig et al. [155] being an early example.”

• cites [155] 155. Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021. Enhancing factual consistency of abstractive summarization. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 718-733. https://doi.org/10.18653/v1/2021.naacl-main.58 sourceCould not verify

  Section 3.2.4 describes token probabilities for summary generation, not selecting among multiple-choice answers. Searches for Zweig and language-model scoring of answer choices found no passage addressing the attributed example or practice in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Schick and Schütze try a simple average when using an ensemble model to annotate an unlabeled dataset.

“Schick and Schütze [117] also try a simple average when using an ensemble model to annotate an unlabeled dataset.”

• cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 sourceSupported

May combine two claims.

Prompting methods are particularly useful in few-shot learning because there are generally not enough training examples to fully specify the desired behavior.

“Prompting methods are particularly useful in the latter case [9,32,117], as there are generally not enough training examples to fully specify the desired behavior, and thus using a prompt to push the model in the right direction is particularly effective.”

• cites [9] 9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranz sourcePartly supported

  The Introduction (p. 5) reports that adding a natural-language task description improves performance in a few-shot example, and Evaluation (p. 10) says the authors sometimes use prompts alongside demonstrations. The paper does not establish the broader explanation that few-shot examples are generally insufficient to fully specify the desired behavior. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). sourceNot checked — no full text of the source
• cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 sourceSupported

May combine two claims.

Promptless fine-tuning may overfit or not learn stably on small datasets.

“This is a simple, powerful, and widely used method, but it may overfit or not learn stably on small datasets [21].”

• cites [21] 21. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv:2002.06305. Retrieved from https://arxiv.org/abs/2002.06305. sourcePartly supported

  The Figure 4 caption (Section 5, p. 6) says training is unstable on smaller datasets, and Section 5 (p. 6) reports that some fine-tuning runs fail to converge. These passages support the instability part of the claim, but our copy does not establish that the models overfit small datasets. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Large-scale pre-training allows LMs to handle linguistic phenomena such as analogies, negations, semantic role sensitivity, semantic similarity, cant understanding, and rare word understanding.

“Besides factual knowledge, large-scale pre-training also allows LMs to handle linguistic phenomena such as analogies [9], negations [25], semantic role sensitivity [25], semantic similarity [131], cant understanding [131], and rare word understanding [116].”

• cites [9] 9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranz sourcePartly supported

  The citation to Brown et al. is attached to rare word understanding, not the other listed phenomena. Section 3.9.5 (pp. 26, 29) reports that GPT-3 plausibly used newly defined, nonexistent words in sentences. It does not test understanding of rare existing words or inferring a word’s meaning from its usage. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [25] 25. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Trans. Assoc. Comput. Ling. 8 (2020), 34-48. sourcePartly supported

  The Discussion (p. 11) reports that BERT generally distinguishes role reversals, though less sensitively than humans, supporting a limited claim about semantic role sensitivity. But the Introduction (p. 1) says BERT fails to show generalizable understanding of negation, so this source does not support the claim that pre-training allows models to handle negation. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [131] 131. Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv:1608.05203. Retrieved from https://arxiv.org/abs/1608.05203. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Schick and Schütze set the weight for each prompt proportional to the accuracy on the training set before training.

“Schick and Schütze [117] and Schick and Schütze [120] set the weight for each prompt proportional to the accuracy on the training set before training.”

• cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 sourceSupported
• cites [120] 120. Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learn- ers. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2339-2352. https://doi.org/10.18653/v1/ 2021.naacl-main.185 sourceSupported

May combine two claims.

For text classification tasks, most previous work has used cloze prompts, and both prompt template engineering and prompt answer engineering have been explored extensively.

“For text classification tasks, most previous work has used cloze prompts, and both prompt template engineering [32,40,67] and prompt answer engineering [32,115,117] have been explored extensively.”

• cites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). sourceNot checked — no full text of the source
• cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 sourceSupported
• cites [67] 67. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 3045-3059. https://doi.org/10.18653/v1/2021.emnlp-main.243 sourcePartly supported

  Section 2 describes prompt design and the selection of prompt tokens by manual or search methods; Section 2.1 discusses initializing classification prompts with output-class embeddings, drawing a comparison with verbalizers. These passages support that prompt design and class-label choices have been explored, but they do not establish that most previous text-classification work used cloze prompts or that both forms of engineering were explored extensively. (LLM, conf=0.77)

  Experimental — accuracy not yet measured
• cites [115] 115. Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifying words that can serve as la- bels for few-shot text classification. In Proceedings of the 28th International Conference on Computational Linguistics (COLING'20), Donia Scott, Núria Bel, and Chengqing Zong (Eds.). International Committee on Computational Lin- guistics, 5569-5578. https://doi.org/10.18653/v1/2020.coling-main.488 sourceSupported
• cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

This practice is arguably not true zero-shot learning with respect to the downstream task.

“As noted by Perez et al. [96], this is arguably not true zero-shot learning with respect to the downstream task.”

• cites [96] 96. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS'21), Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 11054-11070. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Jiang et al. learn the weight for each prompt by maximizing the probability of the target output over training data.

“For example, Jiang et al. [52] learn the weight for each prompt by maximizing the probability of the target output over training data.”

• cites [52] 52. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Trans. Assoc. Comput. Ling. 8 (2020), 423-438. https://doi.org/10.1162/tacl_a_00324 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

For classification tasks, majority voting can be used to combine the results from different prompts.

“For classification tasks, majority voting can also be used to combine the results from different prompts [40,67].”

• cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 sourceCould not verify

  Section 3 (WARP) describes learning a prompt to produce a classification prediction, and Section 4.2 (Results) compares prompt configurations. Neither says that predictions from different prompts are combined by majority voting. Searches for majority voting, ensembling, and combining predictions across prompts found no supporting passage; the mention of predicting the 'majority label' in Section 4.1 (Tasks) concerns a single-label baseline, not a vote across prompts. Page numbers were unavailable in our copy. (LLM, conf=0.73)

  Experimental — accuracy not yet measured
• cites [67] 67. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 3045-3059. https://doi.org/10.18653/v1/2021.emnlp-main.243 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Qin and Eisner use the same approach except that the weight for each prompt is optimized together with soft prompt parameters.

“Qin and Eisner [103] use the same approach except that the weight for each prompt is optimized together with soft prompt parameters.”

• cites [103] 103. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Pro- ceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 5203-5212. sourceSupported

May combine two claims.

Schick and Schütze train a separate model for each manually created template-answer pair and use the ensemble of them to annotate an unlabeled dataset, then the final model is trained to distill the knowledge from the annotated dataset.

“To incorporate this idea, Schick and Schütze [117] and Schick and Schütze [118,120] train a separate model for each manually created template-answer pair and use the ensemble of them to annotate an unlabeled dataset.Then the final model is trained to distill the knowledge from the annotated dataset.”

• cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 sourceSupported
• cites [118] 118. Timo Schick and Hinrich Schütze. 2021. Few-shot text generation with natural language instructions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 390-402. https://doi.org/10.18653/v1/2021.emnlp-main.32 sourcePartly supported

  Section 5.2, “Combining Instructions,” describes a model for each pattern–decoder-prefix pair, generating and scoring outputs for unlabeled examples, then training a final model on those outputs. But §5.3, “Preventing Overfitting,” says the method can instead train one model jointly on all instructions, so the claim that they train a separate model for every pair does not describe the approach without qualification. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [120] 120. Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learn- ers. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2339-2352. https://doi.org/10.18653/v1/ 2021.naacl-main.185 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Qin and Eisner introduce a data-dependent weighting strategy where the probability of the input appearing in that prompt is considered in weighting different prompts.

“Besides, Qin and Eisner [103] also introduce a data-dependent weighting strategy where the probability of the input appearing in that prompt is considered in weighting different prompts as well.”

• cites [103] 103. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Pro- ceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 5203-5212. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Logan IV et al. observe that prompt engineering can be reduced by allowing for a combination of prompt answer engineering and partial LM fine-tuning.

“Logan IV et al. [48] more recently observe that the prompt engineering can be reduced by allowing for a combination of prompt answer engineering and partial LM fine-tuning.”

• cites [48] 48. Robert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022. Cutting down on prompts and parameters: Simple few-shot learning with language models. In Findings of the Association for Computational Linguistics: ACL 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, 2824-2835. sourcePartly supported

  Section 4 shows that null prompts and random verbalizer tokens can reduce manual prompt and answer design, but those experiments finetuned all masked-LM parameters. The paper also combines null prompts with bias-only finetuning (BitFit), as reported in Table 2, but does not show the null-prompt-plus-random-verbalizer combination with partial finetuning. Thus the claimed combination is only partly supported. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

For text pair classification tasks, cloze prompts are commonly used.

“Similarly to text classification tasks, for text pair classification tasks, cloze prompts are commonly used [117,120].”

• cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 sourcePartly supported

  Section 3, “Pattern-Exploiting Training,” defines cloze-question patterns for textual inference and gives an example that classifies a pair of sentences. This supports their use for text-pair classification, but the paper does not establish that they are commonly used across such tasks. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [120] 120. Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learn- ers. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2339-2352. https://doi.org/10.18653/v1/ 2021.naacl-main.185 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Probabilities from pre-trained LMs on QA tasks are not very predictive of whether the model is correct or not.

“Jiang et al. [51] take a closer look at such prompt-based QA systems using sequence-to-sequence pre-trained LMs (T5, BART, and GPT2) and observe that probabilities from these pre-trained LMs on QA tasks are not very predictive of whether the model is correct or not.”

• cites [51] 51. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? On the calibration of language models for question answering. Trans. Assoc. Comput. Ling. 9 (09 2021), 962-977. https://doi.org/10.1162/tacl_a_00407 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Pre-trained LMs can generate datasets given certain instructions.

“Schick and Schütze [119] propose to use pre-trained LMs to generate datasets given certain instructions.”

• cites [119] 119. Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 6943-6951. https:// doi.org/10.18653/v1/2021.emnlp-main.555 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Guidance signals for controlled generation can also be keywords, relation triples, or highlighted phrases or sentences to plan the content of generated texts.

“It could also be keywords [112], relation triples [154] or even highlighted phrases or sentences [34,78] to plan the content of generated texts.”

• cites [112] 112. Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, and Junji Tomita. 2020. Abstractive summarization with combination of pre-trained sequence-to-sequence and saliency models. arXiv:2003.13028. Retrieved from https://arxiv.org/abs/ 2003.13028. sourceNot checked — no full text of the source
• cites [154] 154. Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. 2002. Ensembling neural networks: Many could be better than all. Artif. Intell. 137, 1-2 (2002), 239-263. sourceNot checked — no full text of the source
• cites [34] 34. David Grangier and Michael Auli. 2018. QuickEdit: Editing text & translations by crossing words out. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, 272-282. https://doi.org/10.18653/ v1/N18-1025 sourcePartly supported

  The Abstract and §3 describe user-marked tokens in an existing sentence as guidance for generating a revised sentence. They do not describe highlighting whole phrases or sentences to plan the content of a generated text. The keywords and relation triples have separate citations and are not attributed to this paper. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [78] 78. Yixin Liu, Zi-Yi Dou, and Pengfei Liu. 2021. RefSum: Refactoring neural summarization. In Proceedings of the Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 1437-1448. https://doi.org/10.18653/v1/2021.naacl-main.113 sourceCould not verify

  The Introduction and §4.1 describe Refactor selecting document sentences or choosing among candidate summaries, not using highlighted phrases or sentences as guidance to plan generated text. Searches for highlighted phrases and guided generation found no passage in our copy supporting the part attributed to this citation; the other citations attach to keywords and relation triples. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Knowing to pay attention to important information is a key step when extracting useful information from objects such as long text sequences, images, or knowledge bases.

“Knowing to pay attention to the important information is a key step when extracting useful information from objects such as long text sequences [75,129], images [130,149], or knowledge bases [23,146]).”

• cites [75] 75. Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016. Neural machine translation with super- vised attention. In Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers (COLING'16). The COLING 2016 Organizing Committee, 3093-3102. sourceNot checked — no full text of the source
• cites [129] 129. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1631-1642. sourcePartly supported

  Section 3, “Stanford Sentiment Treebank” (page unavailable), says classifiers can use a few strongly sentiment-bearing words in longer documents, supporting the importance of identifying useful information in text. It does not establish that learning to pay attention is a key step in extracting information from long text sequences; its own experiments focus on phrases and sentences. The citations attached to images and knowledge bases are separate from the ones attached to text. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [130] 130. Ekta Sood, Simon Tannert, Philipp Mueller, and Andreas Bulling. 2020. Improving natural language processing tasks with human gaze-guided neural attention. In Advances in Neural Information Processing Systems, Vol. 33. 6327-6341. sourceNot checked — no full text of the source
• cites [149] 149. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with extracted gap- sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning (ICML'20). PMLR, 11328-11339. http://proceedings.mlr.press/v119/zhang20ae.html. sourceSupported
• cites [23] 23. Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A general framework for guided neural abstractive summarization. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 4830-4842. https://doi.org/10.18653/v1/2021.naacl-main.384 sourceSupported
• cites [146] 146. Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woo-Myoung Park. 2021. GPT3Mix: Leveraging large-scale language models for text augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 2225-2239. https://doi.org/10.18653/v1/2021.findings-emnlp.192 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Data augmentation introduces additional training samples in an explicit way while prompts can be regarded as highly-condensed training samples.

“Data augmentation introduce additional training samples in an explicit way while prompts can be regarded as highly-condensed training samples [65].”

• cites [65] 65. Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? In Proceedings of the Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2627-2636. https://doi.org/10.18653/v1/2021.naacl-main.208 sourcePartly supported

  The Introduction reports that task-targeted prompts provide a measurable advantage equivalent to additional training data, supporting the comparison of prompts to condensed training samples. The paper does not establish the sentence’s separate claim about data augmentation explicitly adding training samples; its Related Work section only mentions additional pseudo-labeled data in another method. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The idea of unifying multiple NLP tasks into a QA framework has been further explored in information extraction and text classification.

“Later, this idea has been further explored in information extraction [70,142] and text classification [12].”

• cites [70] 70. Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 5849-5859. https://doi.org/10.18653/v1/2020.acl-main.519 sourceSupported
• cites [142] 142. Colin Wei, Sang Michael Xie, and Tengyu Ma. 2021. Why do pretrained language models help in downstream tasks? An analysis of head and prompt tuning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS'21), Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 16158-16170. sourceCould not verify

  The Abstract (p. 1) and Formulations and notations (p. 3) describe downstream classification using head and prompt tuning, but do not describe recasting text classification as question answering. A search for QA reformulation found no passage in our copy addressing that attributed idea. (LLM, conf=0.73)

  Experimental — accuracy not yet measured
• cites [12] 12. Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and Jiwei Li. 2020. Description based text classification with reinforce- ment learning. In Proceedings of the International Conference on Machine Learning. PMLR, 1371-1382.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Guidance signals for controlled generation can be style tokens, length specifications, domain tags, or other pieces of information used to control generated text.

“Specifically, the guidance signals could be style tokens [27,123], length specifications [56], domain tags [14], or any variety of other pieces of information used to control of the generated text.”

• cites [27] 27. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation. Association for Computational Linguistics, 45-54. sourceSupported
• cites [123] 123. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with mono- lingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 86-96. https://doi.org/10.18653/v1/P16-1009 sourceCould not verify

  The citation is attached specifically to “domain tags.” Section 4.2.2 and the Conclusion discuss adapting translation models to a domain using in-domain data, but the passages found for domain tags and domain indicators do not describe tags as guidance signals for controlling generated text. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [56] 56. Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In Proceedings of the Conference on Empirical Methods in Natural Language Pro- cessing. Association for Computational Linguistics, 1328-1338. https://doi.org/10.18653/v1/D16-1140 sourceSupported
• cites [14] 14. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 385-391. https://doi.org/10.18653/v1/P17-2061 sourceSupported

May combine two claims.

Pre-trained embeddings and LMs can perform simple operations such as addition and subtraction when the number of digits is small but fail when the numbers are larger.

“Within the context of pre-trained LMs, researchers have found that pre-trained embeddings and LMs can perform simple operations such as addition and subtraction when the number of digits is small but fail when the numbers are larger [9,88,139].”

• cites [9] 9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranz sourcePartly supported

  Section 3.9.1 (p. 22) reports that GPT-3 performs addition and subtraction well with small numbers and that accuracy decreases as digit count increases. It does not test pre-trained embeddings, and GPT-3 still answers some four- and five-digit problems correctly, so 'fail when the numbers are larger' overstates the result. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [88] 88. Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. 2019. Exploring numeracy in word em- beddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 3374-3380. https://doi.org/10.18653/v1/P19-1329 sourceCould not verify

  Section 2 (Analysis Framework) tests number magnitude and numeration, and Section 4.1 reports how embeddings perform on those tests. Neither passage assesses addition or subtraction, or compares arithmetic performance by number of digits; searches for those operations found no such result. The copy also does not test pre-trained language models on arithmetic. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [139] 139. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP'19), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 2153-2162. https://doi.org/ 10.18653/v1/D19-1221 sourceCould not verify

  The Sentiment Analysis and Natural Language Inference sections discuss models using pre-trained embeddings, but report task accuracy rather than arithmetic ability. Searches for addition, subtraction, and number of digits found no passage addressing whether performance changes with number size. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Supervised attention aims to provide explicit supervision over the attention of models based on the fact that completely data-driven attention can overfit to some artifacts.

“Supervised attention [76] aims to provide explicit supervision over the attention of models based on the fact that completely data-driven attention can overfit to some artifacts [73].”

• cites [76] 76. Shulin Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2017. Exploiting argument information to improve event detection via supervised attention mechanisms. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1789-1798. sourceSupported
• cites [73] 73. Chenxi Liu, Junhua Mao, Fei Sha, and Alan Yuille. 2017. Attention correctness in neural image captioning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

References [61,83] are earlier works that attempt to unify multiple NLP tasks into a QA framework.

“References [61,83] are earlier works that attempt to unify multiple NLP tasks into a QA framework.”

• cites [61] 61. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In Proceedings of the International Conference on Machine Learning. PMLR, 1378-1387. sourceSupported
• cites [83] 83. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv:1806.08730. Retrieved from https://arxiv.org/abs/1806.08730. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Supervised attention methods tried to use additional loss functions to learn to predict gold attention on a manually labeled corpus.

“To solve this problem, supervised attention methods tried to use additional loss functions to learn to predict gold attention on a manually labeled corpus [31,49,102].”

• cites [31] 31. Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong. 2017. Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision. 1811-1820. sourceSupported
• cites [49] 49. Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. 2015. Salicon: Saliency in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1072-1080. sourceNot checked — no full text of the source
• cites [102] 102. Tingting Qiao, Jianfeng Dong, and Duanqing Xu. 2018. Exploring human-like attention supervision in visual question answering. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. sourcePartly supported

  The “Attention supervision for VQA” section (p. 5) describes an additional attention-supervision loss combined with the classification loss. But the supervised VQA model uses attention maps generated automatically by a network trained on human-labeled data, rather than gold attention manually labeled for the corpus on which that model is trained (p. 4). (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

Prompts selected under tuned few-shot learning scenario generalize well across models of similar sizes, while prompts selected under true few-shot learning scenario do not generalize as effectively among models with similar sizes.

“Reference [96] show that prompts selected under tuned few-shot learning scenario (where one has a larger validation set to choose prompts) generalize well across models of similar sizes while prompts selected under true few-shot learning scenario (where one only has a few training samples) do not generalize as effectively as the former setting among models with similar sizes.”

• cites [96] 96. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS'21), Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 11054-11070. sourceSupported

May combine two claims.

Current works typically select answers before selecting templates, but Hambardzumyan et al. have demonstrated the initial potential of simultaneously learning both.

“Current works typically select answers before select template [32,125], but Hambardzumyan et al. [40] have demonstrated the initial potential of simultaneously learning both.”

• cites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). sourceNot checked — no full text of the source
• cites [125] 125. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 7699-7715. https://doi.org/10.18653/v1/2021.emnlp-main.608 sourceCould not verify

  The citation to this paper is attached to the claim that current works select answers before templates, not to the separate claim about Hambardzumyan et al. The Dynamic Prompt Creation section discusses selecting examples for a prompt, and Additional information discusses English templates, but neither addresses an answer-before-template selection order. Searches for that order and for answer–template selection found no supporting passage in our copy. (LLM, conf=0.72)

  Experimental — accuracy not yet measured
• cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 sourceSupported

May combine two claims.

Empirically, for token or span prediction tasks (e.g., NER), prompt decomposition can be considered, while for span relation prediction tasks (e.g., entity coreference), prompt composition would be a better choice.

“Empirically, for those token [81] or span [30] prediction tasks (e.g., NER), prompt decomposition can be considered, while for those span relation prediction [66] tasks (e.g., entity coreference), prompts composition would be a better choice.”

• cites [81] 81. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1064-1074. https://doi.org/10.18653/v1/P16-1101 sourceSupported
• cites [30] 30. Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021. SpanNER: Named entity re-/recognition as span prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP'21), Volume 1: Long Papers, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, 7183-7195. https://doi.org/10.18653/v1/ 2021.acl-long.558 sourceSupported
• cites [66] 66. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 188-197. https://doi.org/10.18653/v1/D17-1018 sourceSupported

May combine two claims.

Almost all works use prompt learning for text generation relying solely on a single answer, with only a few exceptions.

“So far, almost all works use prompt learning for text generation relying solely on a single answer, with only a few exceptions [52].”

• cites [52] 52. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Trans. Assoc. Comput. Ling. 8 (2020), 423-438. https://doi.org/10.1162/tacl_a_00324 sourceCould not verify

  The Introduction (p. 1) describes prompts used to generate answers or predict missing words, while §4.2 (p. 4) describes this paper’s method for combining multiple prompts. Neither passage establishes that almost all prompt-learning work on text generation relies on a single answer, or that there are only a few exceptions. Searches for those claims and for prior single-prompt work found no supporting passage. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Even during the pre-train and fine-tune paradigm, there was some use of pre-trained LMs exemplified by word2vec and GloVe, but they were used for only a limited portion of the final model parameters.

“Even during this stage, there was some use of pre-trained LMs exemplified by word2vec[85,86] and GloVe[95], but they were used for only a limited portion of the final model parameters.”

• cites [85] 85. Tomas Mikolov, Kai Chen, G. Corrado, and J. Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR'13). sourcePartly supported

  Section 1.2 (p. 2) says this paper focuses on learning word vectors separately from a full neural language model and notes that word vectors can be used in NLP applications. Section 7 (p. 11) also reports that the authors published vectors for others to use. The paper does not establish that such vectors occupied only a limited portion of a downstream model’s final parameters; the separate GloVe citation is not judged here. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [86] 86. Tomas Mikolov, Ilya Sutskever, Kai Chen, G. Corrado, and J. Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the Conference and Workshop on Neural Information Processing Systems (NIPS'13). sourcePartly supported

  The Introduction (p. 1) and Conclusion (p. 8) identify this work as training Skip-gram word representations, supporting the citation as an example of word2vec. The paper does not describe word2vec as a pre-trained language model or establish that its representations occupied only a limited portion of a final model’s parameters. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [95] 95. Jeffrey Pennington, R. Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'14). sourceNot checked — no full text of the source

May combine two claims.

Existing prompt augmentation methods are limited by the input length; feeding too many demonstrations to input is infeasible.

“Existing prompt augmentation methods are limited by the input length, i.e., feeding too many demonstrations to input is infeasible.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Prompt sharing, where prompt learning is applied to multiple tasks, domains, or languages, has not been explored so far.

“So far this field has not been explored.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

There are few to no systematic comparisons of the benefits brought by prompt-based learning for different pre-trained LMs.

“So far, there are few to no systematic comparisons of the benefits brought by prompt-based learning for different pre-trained LMs.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

For text generation tasks, the study of prompt ensemble learning has not been performed so far, probably because ensemble learning in text generation itself is relatively complicated.

“For text generation tasks, the study of prompt ensemble learning has not been performed so far, probably because ensemble learning in text generation itself is relatively complicated.”

Probably not a substantial claim: may not pass S3 (load-bearing).

How to select ensemble-worthy prompts is under-explored.

“In addition, how to select ensemble-worthy prompts is also under-explored.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There are a fairly wide variety of methods for tuning parameters of prompts, LMs, or both.

“As discussed in Section 6, there are a fairly wide variety of methods for tuning parameters of prompts, LMs, or both.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Multiple continuous prompts are typically learned by using different initializations or different random seeds.

“Multiple continuous prompts are typically learned by using different initializations or different random seeds.”

Probably not a substantial claim: may not pass S3 (load-bearing).

How to distill the knowledge from different prompts remains underexplored.

“How to distill the knowledge from different prompts remains underexplored.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All the above considerations refer to the application of prompt in a single task, domain or language.

“All the above considerations refer to the application of prompt in a single task, domain or language.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Moving beyond simple structures to more complicated varieties of structure is largely unexplored.

“However, moving beyond this to more complicated varieties of structure is largely unexplored, and a potentially interesting research area.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Most existing works about prompt-based learning revolve around either text classification or generation-based tasks.

“Most existing works about prompt-based learning revolve around either text classification or generation-based tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The question of whether pre-training methods effective for the pre-train and fine-tune paradigm are applicable as-is to prompting, or whether pre-training methods can be re-thought to improve prompting, has not been covered extensively by the literature.

“This is an important research question that has not been covered extensively by the literature.”

Classified as probably not a substantial claim.

The field still lacks a systematic understanding of the tradeoffs between different tuning strategies.

“However, given the nascent stage of this research field, we still lack a systematic understanding of the tradeoffs between these methods.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Prompt paraphrasing approaches take an existing seed prompt, paraphrase it into a set of other candidate prompts, and then select the one that achieves the highest training accuracy on the target task.

“Paraphrasing-based approaches take in an existing seed prompt (e.g., manually constructed or mined), paraphrase it into a set of other candidate prompts, and then selects the one that achieves the highest training accuracy on the target task.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Gao et al. introduced the seq2seq pre-trained LM T5 into the template search process.

“Gao et al. [32] introduce the seq2seq pre-trained LM T5 into the template search process.”

• cites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Ben-David et al. propose a domain adaptation algorithm that trains T5 to generate unique domain relevant features for each input.

“Ben-David et al. [5] propose a domain adaptation algorithm that trains T5 to generate unique domain relevant features (DRFs) (a set of keywords that characterize domain information) for each input.”

• cites [5] 5. Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022. PADA: Example-based prompt learning for on-the-fly adapta- tion to unseen domains. Trans. Assoc. Comput. Linguist. 10 (4 2022), 414-433. https://doi.org/10.1162/tacl_a_00468 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For tasks with multiple inputs such as text pair classification, prompt templates must contain space for two inputs [X1] and [X2] or more.

“Finally, for some tasks regarding multiple inputs such as text pair classification, prompt templates must contain space for two inputs, [X1] and [X2], or more.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Manually crafting templates is intuitive and does allow solving various tasks with some degree of accuracy.

“While the strategy of manually crafting templates is intuitive and does allow solving various tasks with some degree of accuracy”

May combine two claims.

Choosing a proper prompt template has a large effect not only on accuracy but also on which task the model performs.

“Given that the prompt specifies the task, choosing a proper prompt template has a large effect not only on the accuracy but also on which task the model performs in the first place.”

• AssumptionRests on an assumption: The prompt specifies the task.

May combine two claims.

Continuous prompts remove the constraint that the embeddings of template words be the embeddings of natural language words and the restriction that the template is parameterized by the pre-trained LM's parameters.

“continuous prompts remove two constraints: (1) relax the constraint that the embeddings of template words be the embeddings of natural language (e.g., English) words and (2) remove the restriction that the template is parameterized by the pre-trained LM's parameters.”

Classified as probably not a substantial claim.

Prompt-based learning is based on language models that model the probability of text directly.

“prompt-based learning is based on language models that model the probability of text directly.”

Probably not a substantial claim: may not pass S4 (based).

The prompting framework allows a language model to be pre-trained on massive amounts of raw text.

“It allows the language model to be pre-trained on massive amounts of raw text”

Probably not a substantial claim: may not pass S3 (load-bearing).

Template words are not necessarily composed of natural language tokens; they could be virtual words or continuous vectors.

“In many cases, these template words are not necessarily composed of natural language tokens; they could be virtual words (e.g., represented by numeric ids) that would be embedded in a continuous space later, and some prompting methods even generate continuous vectors directly”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Prompt-based learning is a new paradigm in natural language processing.

“This article surveys and organizes research works in a new paradigm in natural language processing, which we dub”

Probably not a substantial claim: may not pass S3 (load-bearing).

For tasks solved using masked LMs, cloze prompts are a good fit.

“For tasks that are solved using masked LMs, cloze prompts are a good fit, as they very closely match the form of the pre-training task.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The number of [X] slots and the number of [Z] slots can be flexibly changed for the need of tasks at hand.

“The number of [X] slots and the number of [Z] slots can be flexibly changed for the need of tasks at hand.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Full text reconstruction models are more versatile and can be used with either cloze or prefix prompts.

“Full text reconstruction models are more versatile and can be used with either cloze or prefix prompts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Zhong et al. found that initializing with manual templates can provide a better starting point for the search process.

“This work found that initializing with manual templates can provide a better starting point for the search process.”

• cites [152] 152. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML'21), Proceedings of Machine Learning Research, Vol. 139, Marina Meila and Tong Zhang (Eds.). PMLR, 12697-12706. http:// proceedings.mlr.press/v139/zhao21c.html.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Chen et al. automatically decompose each relation label into its constituent words and use them as an answer.

“When performing relation extraction, Chen et al. [13] automatically decompose each relation label into its constituent words and use them as an answer.”

• cites [13] 13. Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. KnowPrompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference (WWW'22), Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (Eds.). ACM, 2778-2788. https://doi.org/10.1145/3485447. 3511998 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The resulting model from Tsimpoukelli et al. can perform few-shot learning for vision-language tasks such as visual question answering.

“They show that the resulting model can perform few-shot learning for vision-language tasks such as visual question answering, and so on.”

• cites [135] 135. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. arXiv:1806.02847. Retrieved from http://arxiv.org/abs/1806.02847. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Weighted averaging can be used for prompt ensembling where each prompt is associated with a weight.

“To account for this, some works also explore to use of weighted averages for prompt ensembling where each prompt is associated with a weight.”

Probably not a substantial claim: may not pass S3 (load-bearing).

PTR inserts several virtual tokens whose embeddings can be tuned together with the pre-trained LMs parameters using training samples.

“To enhance the representation ability of the resulting template, they also insert several virtual tokens whose embeddings can be tuned together with the pre-trained LMs parameters using training samples.”

• cites [41] 41. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. PTR: Prompt Tuning with Rules for Text Classification. arXiv:2105.11259 [cs.CL]. Retrieved from https://arxiv.org/abs/2105.11259. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

Manually created answers may be sub-optimal for getting the LM to achieve ideal prediction performance.

“As with manually created prompts, it is possible that manually created answers are sub-optimal for getting the LM to achieve ideal prediction performance.”

• AssumptionRests on an assumption: Manually created answers may be sub-optimal for getting the LM to achieve ideal prediction performance.

Probably not a substantial claim: may not pass S3 (load-bearing).

Yin et al. manually design lists of words relating to relevant topics, emotions, or other aspects of the input text to be classified.

“Yin et al. [144] manually design lists of words relating to relevant topics (”

• cites [144] 144. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Gen- eralized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS'19), Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 5754-5764.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Prompt ensembling can leverage the complementary advantages of different prompts, alleviate the cost of prompt engineering, and stabilize performance on downstream tasks.

“This sort of prompt ensembling can (1) leverage the complementary advantages of different prompts, (2) alleviate the cost of prompt engineering, since choosing one best-performing prompt is challenging, and (3) stabilize performance on downstream tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Han et al. propose prompt tuning with rules (PTR), which uses manually crafted sub-templates to compose a complete template using logic rules.

“Han et al. [41] propose prompt tuning with rules (PTR), which uses manually crafted sub-templates to compose a complete template using logic rules.”

• cites [41] 41. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. PTR: Prompt Tuning with Rules for Text Classification. arXiv:2105.11259 [cs.CL]. Retrieved from https://arxiv.org/abs/2105.11259. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Gao et al. first construct a pruned search space Z by selecting top-k vocabulary words based on their generation probability at the [Z] position determined by training samples, then further prune by zero-shot accuracy, then fine-tune the LM with fixed templates and every answer mapping and select the best label word based on development set accuracy.

“Gao et al. [32] first construct a pruned search space Z by selecting top-k vocabulary words based on their generation probability at the [Z] position determined by training samples.Then the search space is further pruned down by only selecting a subset of words within Z based on their zero-shot accuracy on the training samples.In the search step, they fine-tune the LM with fixed templates together”

• cites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The key to prompting for classification-based tasks is reformulating it as an appropriate prompt.

“The key to prompting for classification-based tasks is reformulating it as an appropriate prompt.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In tuning-free prompting, because prompts are the only method that provide the task specification, heavy engineering is necessary to achieve high accuracy.

“Because prompts are the only method that provide the task specification, heavy engineering is necessary to achieve high accuracy.”

May combine two claims.

Fixed-LM prompt tuning can retain knowledge in LMs and is suitable in few-shot scenarios.

“Similarly to tuning-free prompting, it can retain knowledge in LMs and is suitable in few-shot scenarios.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In relation extraction, the larger label space results in more difficulty in prompt answer engineering.

“Chen et al. [13] first explored the application of fixed-prompt LM Tuning in relation extraction and discuss two major challenges that hinder the direct inheritance of prompting methodology from classification tasks: (1) The larger label space (e.g., 80 in relation extraction vs. 2 in binary sentiment classification) results in more difficulty in prompt answer engineering.”

• cites [13] 13. Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. KnowPrompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference (WWW'22), Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (Eds.). ACM, 2778-2788. https://doi.org/10.1145/3485447. 3511998 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Tuning all the LM parameters allows the model to fit to larger training datasets.

“Tuning all the LM parameters allows the model to fit to larger training datasets.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

An ensemble of deep learning models can typically improve the performance, and this superior performance can be distilled into a single model using knowledge distillation.

“An ensemble of deep learning models can typically improve the performance, and this superior performance can be distilled into a single model using knowledge distillation [2].”

• cites [2] 2. Zeyuan Allen-Zhu and Yuanzhi Li. 2020. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. arXiv:2012.09816. Retrieved from https://arxiv.org/abs/2012.09816. sourceNot checked — no full text of the source

May combine two claims.

While effective in few-shot scenarios, fixed-LM prompt tuning has limited representation power in large-data settings.

“While effective in few-shot scenarios, representation power is limited in large-data settings.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In prompt+LM tuning, the addition of the prompt can provide additional bootstrapping at the start of model training.

“Notably, this setting is very similar to the standard pre-train and fine-tune paradigm, but the addition of the prompt can provide additional bootstrapping at the start of model training.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Fixed-LM prompt tuning is not applicable in zero-shot scenarios.

“Not applicable in zero-shot scenarios.”

Classified as probably not a substantial claim.

It is not easy to predict all the span types at the same time due to the large number of spans.

“It is not easy to predict all the span types at the same time due to the large number of spans, so different prompts for each span can be created and predicted separately.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Prompt+LM tuning may overfit to small datasets.

“May overfit to small datasets.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Prompt+LM tuning is the most expressive method and likely suitable for high-data settings.

“This is the most expressive method, likely suitable for high-data settings.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LMs fine-tuned on one downstream task may not be effective on another one.

“LMs fine-tuned on one downstream task may not be effective on another one.”

May combine two claims.

In the in-context learning setting, providing many answered prompts can be slow at test time and thus cannot easily use large training datasets.

“In particular in the in-context learning setting, providing many answered prompts can be slow at test time and thus cannot easily use large training datasets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Gao et al. use a similar ensemble method on their automatically generated templates.

“Gao et al. [32] use a similar ensemble method on their automatically generated templates.”

• cites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). sourceNot checked — no full text of the source

May combine two claims.

For text pair classification tasks, researchers mainly focus on template search in the few-shot learning setting and the answer space Z is usually manually preselected from the vocabulary.

“Regarding prompt engineering, researchers mainly focus on the template search in the few-shot learning setting and the answer space Z is usually manually preselected from the vocabulary.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Input-dependent prompts are a common setting and effective in controlled text generation.

“However, this is a common setting and effective in the controlled text generation, which may provide valuable direction for the future work on prompt learning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Unlike PTR or AdaPrompt, argRanker's new formulation does not use the entirety of the pre-trained LM's architecture (e.g., the word prediction layer of BERT).

“Different from works such as PTR, or AdaPrompt, argRanker's new formulation does not use the entirety of the pre-trained LM's architecture (e.g., the word prediction layer of the BERT model).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Controlled generation aims to incorporate various types of guidance beyond the input text into the generation model.

“Controlled generation aims to incorporate various types of guidance beyond the input text into the generation model [146].”

• cites [146] 146. Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woo-Myoung Park. 2021. GPT3Mix: Leveraging large-scale language models for text augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 2225-2239. https://doi.org/10.18653/v1/2021.findings-emnlp.192 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The main motivation for using prompts for text generation is to specify the task itself and better utilize the pre-trained LM.

“In contrast, the main motivation for using prompts for text generation is to specify the task itself and better utilize the pre-trained LM.”

May combine two claims.

Both prompting and controlled generation add extra information to the input text for better generation, and these additional signals are often learnable parameters.

“Both add extra information to the input text for better generation, and these additional signals are (often) learnable parameters.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Controlled generation targets generation of a particular type of text while prompt learning uses prompts to specify the task itself.

“Controlled generation targets generation of a particular type of text while prompt learning uses prompts to specify the task itself.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There are several commonalities between prompt-based learning and query reformulation, including that both aim to make better use of existing knowledge bases by asking the right questions and that the knowledge bases are usually a black-box.

“There are several commonalities between prompt-based learning and query reformulation, for example (1) both aim to make better use of some existing knowledge bases by asking a right questions and (2) the knowledge bases are usually a black-box, not available to the users, so researchers must learn how to probe it optimally based on solely questions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The application of prompt-based learning in tagging task has not been fully explored.

“Overall, the application of prompt-based learning in tagging task has not been fully explored.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Research works on supervised attention usually target salient information from an image or text, while prompt learning aims to utilize relevant knowledge from the pre-trained LM.

“Research works on supervised attention usually target at salient information from an image or text, while prompt learning aims to utilize relevant knowledge from the pre-trained LM.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Prompt augmentation is a specific subset of few-shot learning.

“Prompt augmentation is a specific subset of few-shot learning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Opitz's argRanker method conceptualizes argumentative relation classification as a ranking problem for two sentences connected by manually defined phrases.

“Opitz [93]'s argRanker method conceptualizes argumentative relation classification as a ranking problem for two sentences that are connected by manually defined phrases.”

• cites [93] 93. Juri Opitz. 2019. Argumentative relation classification as plausibility ranking. arXiv:1909.09031. Retrieved from https://arxiv.org/abs/1909.09031. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Many prompting methods are a type of controllable generation, where the prompt is usually used to specify the task itself.

“In a way, many of the prompting methods described here are a type of controllable generation, where the prompt is usually used to specify the task itself.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

One key point of prompting methods is how to better use the knowledge in pre-trained LMs, and this was not covered extensively in previous works advocating for QA formulations.

“However, one of the key points of prompting methods is how to better use the knowledge in pre-trained LMs, and these were not covered extensively on previous works advocating for QA formulations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In prompt ensembling, multiple predictions result from different prompt variants.

“In prompt ensembling, multiple predictions result from different prompt variants.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Additional information introduced in larger-context learning is not necessarily labeled data.

“Additional information introduced in larger-context learning is not necessarily the labeled data.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

QA-based formulations aim to solve different tasks through question answering, while prompting additionally targets full use of pre-trained LMs.

“QA-based formulations aim to solve different tasks through question answering, while prompting additionally targets full use of pre-trained LMs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Compared to previous few-shot learning methods, prompt augmentation directly prepends several labeled samples to the currently processed sample to elicit knowledge from pre-trained LMs even without any parameter tuning.

“Compared to previous methods, prompt augmentation directly prepends several labeled samples to the currently processed sample to elicit knowledge from pre-trained LMs even without any parameter tuning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Controlled generation methods do not necessarily require a pre-trained LM.

“They do not necessarily require a pre-trained LM.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In prompt ensembling methods, the space and time complexity increase as more prompts are considered.

“In prompt ensembling methods, the space and time complexity increase as we consider more prompts.”

May combine two claims.

Wei et al. showed that soft-prompt tuning can relax the non-degeneracy assumptions needed for downstream recovery, making it easier to extract task-specific information.

“Wei et al. [141] showed that soft-prompt tuning can relax the non-degeneracy assumptions (the generation probability of each token is linearly independent) needed for downstream recovery (i.e., recover the ground-truth labels of the downstream task.).making it easier to extract task-specific information.”

• cites [141] 141. Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM Comput. Surv. 53, 3 (2020), 1-34. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For text generation tasks, qualified answers can be semantically equivalent but syntactically diverse.

“For text generation tasks, qualified answers can be semantically equivalent but syntactically diverse.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Aghajanyan et al. present structured prompts based on hyper text markup language for more fine-grained web text generation.

“Aghajanyan et al. [1] present structured prompts based on hyper text markup language for more fine-grained web text generation.”

• cites [1] 1. Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. HTLM: Hyper-text pre-training and prompting of language models. arXiv:2107.06955. Retrieved from https:// arxiv.org/abs/2107.06955. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

When there are too many classes, selecting an appropriate answer space becomes a difficult combinatorial optimization problem.

“When there are too many classes, how to select an appropriate answer space becomes a difficult combinatorial optimization problem.”

Classified as probably not a substantial claim.

Even with a calibrated probability distribution, one must be careful when assuming a single gold answer for an input.

“Even though we have a calibrated probability distribution, we also need to be careful when we assume a single gold answer for an input.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

If the gold answer is

“For example, if we consider the gold answer to be”

May combine two claims.

The calibration method of Zhao et al. has two drawbacks: it comes with the overhead of finding proper context-free input, and the probability distribution of the underlying pre-trained LM is still not calibrated.

“However, this method has two drawbacks: (1) it comes with the overhead of finding proper context-free input (e.g., whether to use”

• cites [151] 151. Yue Zhang and Joakim Nivre. 2011. Transition-based dependency parsing with rich non-local features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Associa- tion for Computational Linguistics, 188-193. https://www.aclweb.org/anthology/P11-2033.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Prompt-based learning has shown significant potential among different tasks and scenarios, but several challenges remain.

“Although prompt-based learning has shown significant potential among different tasks and scenarios, several challenges remain, some of which we detail below.”

May combine two claims.

The pre-train and fine-tune paradigm is less conducive to architectural exploration, because unsupervised pre-training allows models to learn with fewer structural priors, and pre-training is time-consuming, making experimentation with structural variants costly.

“This paradigm is less conducive to architectural exploration, because (i) unsupervised pre-training allows models to learn with fewer structural priors, and (ii) as pre-training of models is time-consuming, experimenting with structural variants is costly.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The number of template tokens does not need to be pre-specified, since T5 can decode multiple tokens at a masked position.

“The number of template tokens do not need to be pre-specified, since T5 can decode multiple tokens at a masked position.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Prompt learning and supervised attention share the idea of extracting salient information with clues that need to be provided separately.

“In this respect, prompt learning and supervised attention share ideas that both aim to extract salient information with some clues, which need to be provided separately.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Applications of prompt-based learning to information extraction and text analysis tasks have been discussed less, largely because the design of prompts is less straightforward.

“Applications to information extraction and text analysis tasks have been discussed less, largely because the design of prompts is less straightforward.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Much of the success of the prompting paradigm is built on top of pre-trained LMs that were developed for the pre-train and fine-tune paradigm, such as BERT.

“Notably, much of the success of the prompting paradigm is built on top of pre-trained LMs that were developed for the pre-train and fine-tune paradigm, such as BERT.”