Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

In the paper

“Shin et al. [125] learn a logistic classifier using the contextualized representation of the [Z] token as input.In the search step, they select the top-k tokens that achieve the highest probability score using the learned logistic classifier in the first step.”

What the sources say

• Disputed

  The “Dynamic Prompt Creation” section describes generating outputs with a language model, using beam search and a nextTokens function to filter valid continuations. This is a different search mechanism from ranking top-k tokens with a logistic classifier trained on a [Z] representation; the cited paper describes no such classifier. (LLM, conf=0.83)

  Checked by a reasoning model — may be wrong.

  cites [125] 125. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 7699-7715. https://doi.org/10.18653/v1/2021.emnlp-main.608 sourceExperimental — accuracy not yet measured

In the paper

“Jiang et al. [51] observed the probabilities of pre-trained LMs (e.g., BART, T5, and GPT-2) on QA tasks are well calibrated.”

What the sources say

• Disputed

  The Conclusion says that LM-based QA models tend to be poorly calibrated in their probability estimates, the opposite of the citing sentence. Section 5.5 also reports that BART and GPT-2 have lower calibration performance than T5; it does not describe these models as well calibrated. (LLM, conf=0.95)

  Checked by a reasoning model — may be wrong.

  cites [51] 51. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? On the calibration of language models for question answering. Trans. Assoc. Comput. Ling. 9 (09 2021), 962-977. https://doi.org/10.1162/tacl_a_00407 sourceExperimental — accuracy not yet measured

In the paper

“Guo et al. [36] use reinforcement learning [132] to generate prompts to control the text generation process.”

What the sources say

• ?Not checked — no full text of the sourcecites [36] 36. Han Guo, Bowen Tan, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. 2021. Text generation with efficient (soft) Q-learning. arXiv:2106.07704. Retrieved from https://arxiv.org/abs/2106.07704. source
• ?Could not verify

  The cited paper describes ERNIE 3.0’s language-modeling pre-training in §3.2.1 (p. 5) and its use of prompts for evaluation in §4.3.1 (p. 13). Neither passage addresses reinforcement learning to generate prompts or Guo et al.’s method; searches for those terms found no passage that does. (LLM, conf=0.76)

  cites [132] 132. Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE 3.0: Large-scale knowledge enhanced pre- training for language understanding and generation. arXiv:2107.02137. Retrieved from https://arxiv.org/abs/2107. 02137. sourceExperimental — accuracy not yet measured

In the paper

“Creating and experimenting with these prompts is an art that takes time and experience, particularly for some complicated tasks such as semantic parsing [124]”

What the sources say

• Partly supported

  The Dynamic Prompt Creation section describes designing and selecting example-based prompts for semantic parsing. It does not say that creating and experimenting with prompts is an art or requires substantial time and experience; the Introduction instead emphasizes rapid prototyping. Page numbers are unavailable in our copy. (LLM, conf=0.78)

  cites [124] 124. Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot se- mantic parsers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), sourceExperimental — accuracy not yet measured

In the paper

“Davison et al. [19] investigate the task of knowledge base completion and design a template for an input (head-relation-tail triple) using LMs.They first handcraft a set of templates as potential candidates and fill the input and answer slots to form a filled prompt.They then use a unidirectional LM to score those filled prompts, selecting the one with the highest LM probability.”

What the sources say

• Supported

  Section 2.1 describes generating candidate sentences from head–relation–tail triples using handcrafted templates, then selecting the candidate with the highest log-likelihood under a pretrained unidirectional language model. Section 3 identifies commonsense knowledge base completion as an evaluation task. (LLM, conf=0.94)

  cites [19] 19. Joe Davison, Joshua Feldman, and Alexander M. Rush. 2019. Commonsense knowledge mining from pretrained mod- els. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP'19), Kentaro Inui, Jing Jiang, Vincent Ng, and Xi- aojun Wan (Eds.). Association for Computational Linguistics, 1173-1178. https://doi.org/10.18653/v1/D19-1109 source

In the paper

“Shin et al. [125] automatically search for template tokens using downstream application training samples and demonstrates strong performance in prompting scenarios.”

What the sources say

• Partly supported

  The Dynamic Prompt Creation section says Shin et al. automatically select relevant training examples for GPT-3 prompts, not that they search for template tokens. The Overnight results section reports performance comparable to a state-of-the-art method on some domains with substantially less training data, supporting the performance part of the claim. (LLM, conf=0.87)

  cites [125] 125. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 7699-7715. https://doi.org/10.18653/v1/2021.emnlp-main.608 sourceExperimental — accuracy not yet measured

In the paper

“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”

What the sources say

• Supported

  Section 3.1 (p. 3) describes automatically extracting prompts from Wikipedia sentences containing subject–object pairs, using either middle words or dependency paths. Section 5.1 (p. 5) identifies this method as “Mine.” The claim’s inputs x and outputs y correspond to the paper’s subjects and objects. (LLM, conf=0.91)

  cites [52] 52. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Trans. Assoc. Comput. Ling. 8 (2020), 423-438. https://doi.org/10.1162/tacl_a_00324 source

In the paper

“Schick and Schütze [117] and Schick et al. [115] find tokens containing at least two alphabetic characters that are frequent in a large unlabeled dataset.”

What the sources say

• Supported

  The Additional information section says the authors restrict their search to tokens with at least two alphabetic characters and retain the 10,000 most frequent in D. The Dataset Details section identifies D as a set of unlabeled examples; page numbers are not provided in our copy. (LLM, conf=0.94)

  cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 source
• Supported

  Section 4.1 (Verbalization Candidates) says the authors retain tokens corresponding to real words with at least two alphabetic characters, then select the 10,000 most frequent in the task’s unlabeled data. The cited paper therefore supports the attributed token-selection procedure. (LLM, conf=0.93)

  cites [115] 115. Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifying words that can serve as la- bels for few-shot text classification. In Proceedings of the 28th International Conference on Computational Linguistics (COLING'20), Donia Scott, Núria Bel, and Chengqing Zong (Eds.). International Committee on Computational Lin- guistics, 5569-5578. https://doi.org/10.18653/v1/2020.coling-main.488 source

In the paper

“To account for interaction between prompt tokens, they represent prompt embeddings as the output of a BiLSTM [35].”

What the sources say

• Partly supported

  The Introduction (p. 1) says P-Tuning uses an LSTM or MLP prompt encoder to model dependencies between continuous prompt embeddings. Section 2.3 (p. 3) describes mapping trainable prompt embeddings to model inputs, but our copy does not specify a bidirectional LSTM; it also presents MLP and identity mappings as alternatives. (LLM, conf=0.85)

  cites [77] 77. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv:2103.10385. Retrieved from https://arxiv.org/abs/2103.10385. sourceExperimental — accuracy not yet measured
• Partly supported

  Section 2 (p. 2) explains that bidirectional LSTM processes a sequence in both directions to access context, and Section 4 (p. 4) reports its use in the paper’s speech-recognition networks. The paper does not discuss P-tuning, prompt embeddings, or interactions between prompt tokens, so it supports the BiLSTM method but not its stated application. (LLM, conf=0.86)

  cites [35] 35. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 6645-6649. https://doi.org/10.1109/ICASSP.2013.6638947 sourceExperimental — accuracy not yet measured

In the paper

“P-tuning also introduces the use of task-related anchor tokens (such as”

What the sources say

• Partly supported

  Section 2.2 (p. 2) says P-Tuning concatenates discrete prompts with continuous prompts and that this combination performs better. The Introduction (p. 1) says the continuous prompts are updated by backpropagation, but our copy does not call the discrete tokens “anchor tokens” or explicitly establish that their embeddings are never tuned during training; the method also permits finetuning the language model. (LLM, conf=0.80)

  cites [77] 77. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv:2103.10385. Retrieved from https://arxiv.org/abs/2103.10385. sourceExperimental — accuracy not yet measured

In the paper

“Longer phrasal or sentential answers are often used in language generation tasks [105] and in other tasks such as multiple-choice question answering (where the scores of multiple phrases are compared against each-other; Khashabi et al. [55]).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [105] 105. Alec Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsu- pervised multitask learners.
• Partly supported

  Section 3.1 describes multiple-choice QA as choosing among candidate answers, and Table 2 reports multi-token average answer lengths for several multiple-choice datasets. The cited paper therefore supports the use of phrasal answers in multiple-choice QA, but the passages do not establish the parenthetical claim that scores of multiple phrases are compared. The language-generation part is attached to a different citation. (LLM, conf=0.83)

  cites [55] 55. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020. Association for Computational Linguistics, 1896-1907. https://doi.org/10.18653/v1/ 2020.findings-emnlp.171 sourceExperimental — accuracy not yet measured

In the paper

“Prompt ensembling is connected to ensembling methods that are used to combine together multiple systems, which have a long history in machine learning [24,133,153].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [24] 24. Kevin Duh, Katsuhito Sudoh, Xianchao Wu, Hajime Tsukada, and Masaaki Nagata. 2011. Generalized minimum bayes risk system combination. In Proceedings of the 5th International Joint Conference on Natural Language Processing. 1356-1360.
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [133] 133. Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press.
• ?Could not verify

  The Method Prompt Data-driven? section describes individual prompt methods, and §2.3 describes optimizing a prompt for a relation. Searches for “prompt ensembling,” combining predictions across prompts, and “ensembling” found no passage connecting prompt ensembling to methods for combining multiple systems or discussing their history in machine learning. (LLM, conf=0.75)

  cites [153] 153. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT'21), Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (Eds.). Association for Com- putational Linguistics, 5017-5033. https://doi.org/10.18653/v1/2021.naacl-main.398 sourceExperimental — accuracy not yet measured

In the paper

“Similarly, Hambardzumyan et al. [40] introduce the use of a continuous template whose shape follows a manual prompt template.”

What the sources say

• Partly supported

  Section 3 (WARP) describes a continuous prompt: trainable embeddings are inserted into predefined template positions. The paper also describes initializing prompts from manually designed Prompt-Verbalizer Patterns, but treats this as an option, not a requirement; Section 6.2 says WARP succeeds without manual initialization. The claim overstates the extent to which the continuous template follows a manual prompt template. (LLM, conf=0.84)

  cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 sourceExperimental — accuracy not yet measured

In the paper

“These methods start with an initial answer space Z and then use paraphrasing to expand this answer space to broaden its coverage [51].”

What the sources say

• Supported

  Section 4.3 describes starting with candidate outputs in I(X) and generating several paraphrases of each. It then sums probabilities across those expressions to reduce sensitivity to wording. The claim uses Z rather than the paper’s I(X), but accurately describes the expansion of answer expressions. (LLM, conf=0.88)

  cites [51] 51. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? On the calibration of language models for question answering. Trans. Assoc. Comput. Ling. 9 (09 2021), 962-977. https://doi.org/10.1162/tacl_a_00407 source

In the paper

“Cui et al. [17] manually design lists such as”

What the sources say

• Partly supported

  Section 4.1 (page unavailable) says Cui et al. manually create templates, map entity labels such as LOC to words such as “location,” and form a list of templates. The Introduction gives “person” and “location” as template values. The paper does not say they manually designed the list of entity categories themselves; it describes those categories as pre-defined. (LLM, conf=0.82)

  cites [17] 17. Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021. Template-based named entity recognition using BART. arXiv:2106.01760 [cs.CL]. Retrieved from https://arxiv.org/abs/2106.01760. sourceExperimental — accuracy not yet measured

In the paper

“Qin and Eisner [103] propose to learn a mixture of soft templates for each input where the weights and parameters for each template are jointly learned using training samples.”

What the sources say

• Supported

  Section 3.3 defines a mixture of soft prompts used to predict an output for an input. Section 3.5 says the prompts and their mixture weights are jointly optimized on training examples; §3.4 also proposes allowing weights to depend on the input. (LLM, conf=0.90)

  cites [103] 103. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Pro- ceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 5203-5212. source

In the paper

“This paraphrasing can be performed using any method, but Jiang et al. [51] specifically use a back-translation method, first translating into another language then back to generate a list of multiple paraphrased answers.”

What the sources say

• Supported

  Section 4.3 says the authors paraphrase each candidate answer by translating it into another language and back to generate a set of paraphrases. The Implementation Details specify English–German back-translation and the use of multiple unique paraphrases. (LLM, conf=0.95)

  cites [51] 51. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? On the calibration of language models for question answering. Trans. Assoc. Comput. Ling. 9 (09 2021), 962-977. https://doi.org/10.1162/tacl_a_00407 source

In the paper

“Hambardzumyan et al. [40] assign a virtual token for each class label and optimize the token embedding for each class together with prompt token embeddings.”

What the sources say

• Supported

  Section 3 (WARP) assigns verbalizer tokens [V1] through [VC] to classes 1 through C and says their embeddings and the prompt-token embeddings are the only trainable parameters. Appendix B (Learned Tokens) likewise states that there is one trainable verbalizer token per output class. (LLM, conf=0.94)

  cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 source

In the paper

“Zhong et al. [152] first define a template using a discrete search method such as AutoPrompt [125]'s, initialize virtual tokens based on this discovered prompt, and then fine-tune the embeddings to increase task accuracy.”

What the sources say

• ?Could not verify

  The Background and Experimental Setup (p. 2) describes prompt templates, but not the attributed sequence of discrete search, virtual-token initialization, and embedding fine-tuning. The Discussion (p. 7) says the authors use a fixed language model without fine-tuning. Searches for Zhong, virtual tokens, AutoPrompt-based initialization, and embedding fine-tuning found no passage addressing that sequence. (LLM, conf=0.76)

  cites [152] 152. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML'21), Proceedings of Machine Learning Research, Vol. 139, Marina Meila and Tong Zhang (Eds.). PMLR, 12697-12706. http:// proceedings.mlr.press/v139/zhao21c.html. sourceExperimental — accuracy not yet measured
• ?Could not verify

  The “Dynamic Prompt Creation” section describes selecting examples for GPT-3 prompts and, separately, fine-tuning GPT-2 XL or BART. Searches for Zhong, AutoPrompt, virtual tokens, and embedding initialization found no passage describing the attributed method. Our copy therefore does not address the claim about Zhong et al. (LLM, conf=0.74)

  cites [125] 125. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 7699-7715. https://doi.org/10.18653/v1/2021.emnlp-main.608 sourceExperimental — accuracy not yet measured

In the paper

“Experimentally, Li and Liang [71] observe that such continuous prefix-based learning is more sensitive to different initialization in low-data settings than the use of discrete prompts with real words.”

What the sources say

• Partly supported

  Section 7.4 (Initialization; page unavailable) reports that prefix initialization has a large impact in low-data settings: random initialization gives low performance and high variance, while initialization from real-word activations improves generation. It does not compare that sensitivity with the use of discrete prompts made of real words; those are a different method, discussed in Sections 2 and 7.2. (LLM, conf=0.88)

  cites [71] 71. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP'21), Volume 1: Long Papers, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, 4582-4597. https://doi.org/10.18653/v1/2021.acl- long.353 sourceExperimental — accuracy not yet measured

In the paper

“Prefix Tuning [71] is a method that prepends a sequence of continuous task-specific vectors to the input, while keeping the LM parameters frozen.”

What the sources say

• Supported

  The Introduction states that prefix-tuning prepends continuous task-specific vectors to the input and optimizes only the prefix. Section 4.2 (Method) confirms that the language-model parameters are fixed while the prefix parameters are trainable. (LLM, conf=0.95)

  cites [71] 71. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP'21), Volume 1: Long Papers, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, 4582-4597. https://doi.org/10.18653/v1/2021.acl- long.353 source

In the paper

“Jiang et al. [52] first filter their prompts by selecting K prompts that achieve the highest accuracy on the training set and then use the average log probabilities obtained from the top K prompts to calculate the probability for a single token at [Z] position when performing factual probing tasks.”

What the sources say

• Supported

  Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  cites [52] 52. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Trans. Assoc. Comput. Ling. 8 (2020), 423-438. https://doi.org/10.1162/tacl_a_00324 source

In the paper

“When performing text generation evaluation, Yuan et al. [147] formulates this task as a text generation problem and take the average of the final generation scores obtained using different prompts.”

What the sources say

• ?Could not verify

  The survey’s A.1 Evaluation Metrics section (p. 40) describes BLEU, ROUGE, and Distinct, while its benchmark discussion (p. 29) concerns dataset selection. Searches for Yuan, prompt-based evaluation, and averaging generation scores found no passage in our copy addressing the attributed method. (LLM, conf=0.75)

  cites [147] 147. Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of knowledge-enhanced text generation (unpublished). https://doi.org/10.1145/3512467 sourceExperimental — accuracy not yet measured

In the paper

“Since the answer tokens are optimized directly in the embedding space, they do not make use of the embeddings learned by the LM and instead learn an embedding from scratch for each label.”

What the sources say

• Partly supported

  Section 3 (WARP; page unavailable) says the verbalizer embeddings are optimized directly in continuous embedding space, with one trainable embedding per class, and that the model's word-embedding decoder is omitted. But Appendix B (Learned Tokens; page unavailable) says verbalizer embeddings can be initialized from existing verbalizer-token embeddings, so the claim that each label embedding is learned from scratch without using LM-learned embeddings is too absolute. (LLM, conf=0.88)

  cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 sourceExperimental — accuracy not yet measured

In the paper

“In many cases, the answer space Z is the space of all tokens [100], fixed-length spans [50], or token sequences [105].In these cases, it is most common to directly map answer z to the final output y using the identity mapping.”

What the sources say

• Supported

  The citation to Petroni et al. is attached to the token-answer example, not to fixed-length spans or token sequences. Section 4.5 (page unavailable) says the LAMA probe considers only single-token objects as prediction targets, and Section 4.2 describes generating and ranking tokens as answers. (LLM, conf=0.88)

  cites [100] 100. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP'19). As- sociation for Computational Linguistics, 2463-2473. https://doi.org/10.18653/v1/D19-1250 source
• Supported

  Section 4, “Multi-token Decoding,” describes predicting multi-token entities as sequences of tokens. Section 4.2, “Final Prediction,” says the selected prediction is the one with the highest confidence, and Section 3.4 evaluates that prediction against valid objects. The citation is attached to “token sequences”; the other answer-space examples have separate citations. (LLM, conf=0.86)

  cites [50] 50. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'20). Association for Computational Linguistics, 5943-5959. https://doi.org/ 10.18653/v1/2020.emnlp-main.479 source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [105] 105. Alec Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsu- pervised multitask learners.

In the paper

“Liu et al. [77] propose”

What the sources say

• Supported

  The Introduction (p. 1) says the authors propose P-Tuning, which concatenates trainable continuous prompt embeddings with discrete prompt tokens and updates them by backpropagation. Section 2.2 (p. 2) describes the continuous prompt embeddings and their optimization, supporting the paraphrase about learned variables in the embedded input. (LLM, conf=0.93)

  cites [77] 77. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv:2103.10385. Retrieved from https://arxiv.org/abs/2103.10385. source

In the paper

“Similarly, Lester et al. [67] prepend the input sequence with special tokens to form a template and tune the embeddings of these tokens directly.”

What the sources say

• Supported

  Section 2 describes prompt tuning as prepending a fixed prompt of special tokens to the input and updating only their embeddings. It specifies that the prompt embeddings are concatenated before the embedded input and that only the prompt parameters are trained. (LLM, conf=0.94)

  cites [67] 67. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 3045-3059. https://doi.org/10.18653/v1/2021.emnlp-main.243 source

In the paper

“With regards to multiple-choice question answering, it is common to use an LM to calculate the probability of an output among multiple choices, with Zweig et al. [155] being an early example.”

What the sources say

• ?Could not verify

  Section 3.2.4 describes token probabilities for summary generation, not selecting among multiple-choice answers. Searches for Zweig and language-model scoring of answer choices found no passage addressing the attributed example or practice in our copy. (LLM, conf=0.76)

  cites [155] 155. Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021. Enhancing factual consistency of abstractive summarization. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 718-733. https://doi.org/10.18653/v1/2021.naacl-main.58 sourceExperimental — accuracy not yet measured

In the paper

“Schick and Schütze [117] also try a simple average when using an ensemble model to annotate an unlabeled dataset.”

What the sources say

• Supported

  Section 3.3, “Combining PVPs,” says the authors use an ensemble to annotate examples from the unlabeled dataset D. They test uniform weighting, setting every model’s weight to 1, which makes the combined score a simple average. (LLM, conf=0.94)

  cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 source

In the paper

“Prompting methods are particularly useful in the latter case [9,32,117], as there are generally not enough training examples to fully specify the desired behavior, and thus using a prompt to push the model in the right direction is particularly effective.”

What the sources say

• Partly supported

  The Introduction (p. 5) reports that adding a natural-language task description improves performance in a few-shot example, and Evaluation (p. 10) says the authors sometimes use prompts alongside demonstrations. The paper does not establish the broader explanation that few-shot examples are generally insufficient to fully specify the desired behavior. (LLM, conf=0.82)

  cites [9] 9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranz sourceExperimental — accuracy not yet measured
• ?Not checked — no full text of the sourcecites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). source
• Supported

  The Introduction explains that a few examples can leave the task ambiguous and that a task description makes it much easier to solve. It then reports that natural-language patterns used in few-shot learning substantially outperform supervised baselines. Page numbers are not available in our copy. (LLM, conf=0.93)

  cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 source

In the paper

“This is a simple, powerful, and widely used method, but it may overfit or not learn stably on small datasets [21].”

What the sources say

• Partly supported

  The Figure 4 caption (Section 5, p. 6) says training is unstable on smaller datasets, and Section 5 (p. 6) reports that some fine-tuning runs fail to converge. These passages support the instability part of the claim, but our copy does not establish that the models overfit small datasets. (LLM, conf=0.86)

  cites [21] 21. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv:2002.06305. Retrieved from https://arxiv.org/abs/2002.06305. sourceExperimental — accuracy not yet measured

In the paper

“Besides factual knowledge, large-scale pre-training also allows LMs to handle linguistic phenomena such as analogies [9], negations [25], semantic role sensitivity [25], semantic similarity [131], cant understanding [131], and rare word understanding [116].”

What the sources say

• Partly supported

  The citation to Brown et al. is attached to rare word understanding, not the other listed phenomena. Section 3.9.5 (pp. 26, 29) reports that GPT-3 plausibly used newly defined, nonexistent words in sentences. It does not test understanding of rare existing words or inferring a word’s meaning from its usage. (LLM, conf=0.84)

  cites [9] 9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranz sourceExperimental — accuracy not yet measured
• Partly supported

  The Discussion (p. 11) reports that BERT generally distinguishes role reversals, though less sensitively than humans, supporting a limited claim about semantic role sensitivity. But the Introduction (p. 1) says BERT fails to show generalizable understanding of negation, so this source does not support the claim that pre-training allows models to handle negation. (LLM, conf=0.88)

  cites [25] 25. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Trans. Assoc. Comput. Ling. 8 (2020), 34-48. sourceExperimental — accuracy not yet measured
• ?Not checked — no full text of the sourcecites [131] 131. Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv:1608.05203. Retrieved from https://arxiv.org/abs/1608.05203. source

In the paper

“Schick and Schütze [117] and Schick and Schütze [120] set the weight for each prompt proportional to the accuracy on the training set before training.”

What the sources say

• Supported

  Section 3.3, “Combining PVPs” (page not provided), says the weighted variant sets each prompt’s weight to its accuracy on the training set before training. Section 4, “Experiments,” says this weighted variant is used unless otherwise specified. (LLM, conf=0.95)

  cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 source
• Supported

  Section 3, “Pattern-Exploiting Training” (p. 3), says the weighting term for each pattern-verbalizer pair is proportional to its accuracy on the training set before training. This directly supports the claim’s description of prompt weighting. (LLM, conf=0.95)

  cites [120] 120. Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learn- ers. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2339-2352. https://doi.org/10.18653/v1/ 2021.naacl-main.185 source

In the paper

“For text classification tasks, most previous work has used cloze prompts, and both prompt template engineering [32,40,67] and prompt answer engineering [32,115,117] have been explored extensively.”

What the sources say

• ?Not checked — no full text of the sourcecites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). source
• Supported

  The citation is attached to prompt-template and prompt-answer engineering. Section 3 (WARP) describes jointly learning prompt and verbalizer embeddings for masked-token predictions, and Section 4.1 (Tasks) applies the method to text classification. The paper therefore exemplifies both approaches; the uncited claim that most previous work used cloze prompts is not part of this citation's scope. (LLM, conf=0.88)

  cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 source
• Partly supported

  Section 2 describes prompt design and the selection of prompt tokens by manual or search methods; Section 2.1 discusses initializing classification prompts with output-class embeddings, drawing a comparison with verbalizers. These passages support that prompt design and class-label choices have been explored, but they do not establish that most previous text-classification work used cloze prompts or that both forms of engineering were explored extensively. (LLM, conf=0.77)

  cites [67] 67. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 3045-3059. https://doi.org/10.18653/v1/2021.emnlp-main.243 sourceExperimental — accuracy not yet measured
• Supported

  The citation is attached to prompt answer engineering. Section 4, “Likelihood Ratio Verbalizer Search,” develops a method for automatically choosing the words that map a cloze prompt’s predictions to classification labels; Section 5 compares it with manual and other automatic verbalizers. The paper is therefore an example of work exploring prompt answers, though it alone does not establish how extensive the field is. (LLM, conf=0.90)

  cites [115] 115. Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifying words that can serve as la- bels for few-shot text classification. In Proceedings of the 28th International Conference on Computational Linguistics (COLING'20), Donia Scott, Núria Bel, and Chengqing Zong (Eds.). International Committee on Computational Lin- guistics, 5569-5578. https://doi.org/10.18653/v1/2020.coling-main.488 source
• Supported

  The citations to this paper attach to prompt template and prompt answer engineering, rather than to the preceding claim about most previous work. Section 4.1 defines multiple cloze patterns and their verbalizers for text-classification tasks; the Additional information section also explores automatic verbalizer search. These support the two cited examples of engineering, though the paper is not evidence for the uncited 'most previous work' assertion. (LLM, conf=0.88)

  cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 source

In the paper

“As noted by Perez et al. [96], this is arguably not true zero-shot learning with respect to the downstream task.”

What the sources say

• Supported

  The Conclusion (p. 11) says that tuning prompts and label names using ImageNet training examples uses data unavailable in true zero-shot learning. It also describes choosing prompts through manual evaluation as effectively leveraging human supervision. These passages support the qualified claim that such a practice is not true zero-shot learning with respect to the downstream task. (LLM, conf=0.88)

  cites [96] 96. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS'21), Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 11054-11070. source

In the paper

“For example, Jiang et al. [52] learn the weight for each prompt by maximizing the probability of the target output over training data.”

What the sources say

• Supported

  Section 4.3 (p. 4) describes an optimized ensemble that assigns weights to candidate prompts. It states that the weight parameters are optimized to maximize the probability of gold-standard objects over training data. (LLM, conf=0.94)

  cites [52] 52. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Trans. Assoc. Comput. Ling. 8 (2020), 423-438. https://doi.org/10.1162/tacl_a_00324 source

In the paper

“For classification tasks, majority voting can also be used to combine the results from different prompts [40,67].”

What the sources say

• ?Could not verify

  Section 3 (WARP) describes learning a prompt to produce a classification prediction, and Section 4.2 (Results) compares prompt configurations. Neither says that predictions from different prompts are combined by majority voting. Searches for majority voting, ensembling, and combining predictions across prompts found no supporting passage; the mention of predicting the 'majority label' in Section 4.1 (Tasks) concerns a single-label baseline, not a vote across prompts. Page numbers were unavailable in our copy. (LLM, conf=0.73)

  cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 sourceExperimental — accuracy not yet measured
• Supported

  Section 6, “Prompt Ensembling,” says the authors trained five prompts for each SuperGLUE task and used simple majority voting to produce ensemble predictions. This directly supports combining results from different prompts for classification tasks. (LLM, conf=0.94)

  cites [67] 67. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 3045-3059. https://doi.org/10.18653/v1/2021.emnlp-main.243 source

In the paper

“Qin and Eisner [103] use the same approach except that the weight for each prompt is optimized together with soft prompt parameters.”

What the sources say

• Supported

  Section 3.5, “Training Objective” (page unavailable), explicitly says the authors jointly optimize the soft prompts and their mixture weights. This supports the attributed claim that each prompt’s weight is optimized together with its soft-prompt parameters. (LLM, conf=0.94)

  cites [103] 103. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Pro- ceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 5203-5212. source

In the paper

“To incorporate this idea, Schick and Schütze [117] and Schick and Schütze [118,120] train a separate model for each manually created template-answer pair and use the ensemble of them to annotate an unlabeled dataset.Then the final model is trained to distill the knowledge from the annotated dataset.”

What the sources say

• Supported

  Section 3.3 describes fine-tuning a separate model for each pattern–verbalizer pair, using their ensemble to annotate unlabeled examples, and training a final classifier on the resulting soft-labeled dataset. The conclusion describes the pairs as defined for the task, consistent with the claim that they are manually created. (LLM, conf=0.94)

  cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 source
• Partly supported

  Section 5.2, “Combining Instructions,” describes a model for each pattern–decoder-prefix pair, generating and scoring outputs for unlabeled examples, then training a final model on those outputs. But §5.3, “Preventing Overfitting,” says the method can instead train one model jointly on all instructions, so the claim that they train a separate model for every pair does not describe the approach without qualification. (LLM, conf=0.86)

  cites [118] 118. Timo Schick and Hinrich Schütze. 2021. Few-shot text generation with natural language instructions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 390-402. https://doi.org/10.18653/v1/2021.emnlp-main.32 sourceExperimental — accuracy not yet measured
• Supported

  Section 3 (pp. 2–3) defines pattern-verbalizer pairs—the paper’s term for template-answer pairs—and describes fine-tuning models for each pair, using their ensemble to soft-label unlabeled examples, and training a final classifier on those labels. Section 4.1 (p. 4) gives the patterns and verbalizers the authors constructed for their tasks. (LLM, conf=0.93)

  cites [120] 120. Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learn- ers. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2339-2352. https://doi.org/10.18653/v1/ 2021.naacl-main.185 source

In the paper

“Besides, Qin and Eisner [103] also introduce a data-dependent weighting strategy where the probability of the input appearing in that prompt is considered in weighting different prompts as well.”

What the sources say

• Supported

  Section 3.4, “Data-Dependent Mixture Modeling” (page not provided), replaces prompt weights p(t | r) with p(t | r, x). It explains that the weighting incorporates p(x | t, r), reflecting how plausible the input x is in a prompt. (LLM, conf=0.94)

  cites [103] 103. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Pro- ceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 5203-5212. source

In the paper

“Logan IV et al. [48] more recently observe that the prompt engineering can be reduced by allowing for a combination of prompt answer engineering and partial LM fine-tuning.”

What the sources say

• Partly supported

  Section 4 shows that null prompts and random verbalizer tokens can reduce manual prompt and answer design, but those experiments finetuned all masked-LM parameters. The paper also combines null prompts with bias-only finetuning (BitFit), as reported in Table 2, but does not show the null-prompt-plus-random-verbalizer combination with partial finetuning. Thus the claimed combination is only partly supported. (LLM, conf=0.83)

  cites [48] 48. Robert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022. Cutting down on prompts and parameters: Simple few-shot learning with language models. In Findings of the Association for Computational Linguistics: ACL 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, 2824-2835. sourceExperimental — accuracy not yet measured

In the paper

“Similarly to text classification tasks, for text pair classification tasks, cloze prompts are commonly used [117,120].”

What the sources say

• Partly supported

  Section 3, “Pattern-Exploiting Training,” defines cloze-question patterns for textual inference and gives an example that classifies a pair of sentences. This supports their use for text-pair classification, but the paper does not establish that they are commonly used across such tasks. (LLM, conf=0.82)

  cites [117] 117. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-questions for few-shot text classification and natural lan- guage inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL'21), Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Compu- tational Linguistics, 255-269. https://doi.org/10.18653/v1/2021.eacl-main.20 sourceExperimental — accuracy not yet measured
• Supported

  Section 4.1 (p. 4) applies cloze-style pattern-verbalizer pairs to CB and RTE, both text-pair entailment tasks. Figure 2 (p. 2) likewise shows a pair of texts converted into a cloze question for entailment classification. These passages support the citing sentence’s broad description of cloze prompts being used for text-pair classification. (LLM, conf=0.87)

  cites [120] 120. Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learn- ers. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2339-2352. https://doi.org/10.18653/v1/ 2021.naacl-main.185 source

In the paper

“Jiang et al. [51] take a closer look at such prompt-based QA systems using sequence-to-sequence pre-trained LMs (T5, BART, and GPT2) and observe that probabilities from these pre-trained LMs on QA tasks are not very predictive of whether the model is correct or not.”

What the sources say

• Supported

  The Background on Calibration section says normalized probability estimates for LM-based question answering were barely correlated with whether outputs were correct. The Introduction identifies T5, BART, and GPT-2 as models examined across QA datasets. (LLM, conf=0.94)

  cites [51] 51. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? On the calibration of language models for question answering. Trans. Assoc. Comput. Ling. 9 (09 2021), 962-977. https://doi.org/10.1162/tacl_a_00407 source

In the paper

“Schick and Schütze [119] propose to use pre-trained LMs to generate datasets given certain instructions.”

What the sources say

• Supported

  The Introduction states that the authors introduce DINO, a method for generating labeled datasets by providing pretrained language models with instructions. The Conclusion specifies that the instructions are in natural language. (LLM, conf=0.95)

  cites [119] 119. Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'21), Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 6943-6951. https:// doi.org/10.18653/v1/2021.emnlp-main.555 source

In the paper

“It could also be keywords [112], relation triples [154] or even highlighted phrases or sentences [34,78] to plan the content of generated texts.”

What the sources say

• ?Not checked — no full text of the sourcecites [112] 112. Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, and Junji Tomita. 2020. Abstractive summarization with combination of pre-trained sequence-to-sequence and saliency models. arXiv:2003.13028. Retrieved from https://arxiv.org/abs/ 2003.13028. source
• ?Not checked — no full text of the sourcecites [154] 154. Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. 2002. Ensembling neural networks: Many could be better than all. Artif. Intell. 137, 1-2 (2002), 239-263. source
• Partly supported

  The Abstract and §3 describe user-marked tokens in an existing sentence as guidance for generating a revised sentence. They do not describe highlighting whole phrases or sentences to plan the content of a generated text. The keywords and relation triples have separate citations and are not attributed to this paper. (LLM, conf=0.84)

  cites [34] 34. David Grangier and Michael Auli. 2018. QuickEdit: Editing text & translations by crossing words out. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, 272-282. https://doi.org/10.18653/ v1/N18-1025 sourceExperimental — accuracy not yet measured
• ?Could not verify

  The Introduction and §4.1 describe Refactor selecting document sentences or choosing among candidate summaries, not using highlighted phrases or sentences as guidance to plan generated text. Searches for highlighted phrases and guided generation found no passage in our copy supporting the part attributed to this citation; the other citations attach to keywords and relation triples. (LLM, conf=0.74)

  cites [78] 78. Yixin Liu, Zi-Yi Dou, and Pengfei Liu. 2021. RefSum: Refactoring neural summarization. In Proceedings of the Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 1437-1448. https://doi.org/10.18653/v1/2021.naacl-main.113 sourceExperimental — accuracy not yet measured

In the paper

“Knowing to pay attention to the important information is a key step when extracting useful information from objects such as long text sequences [75,129], images [130,149], or knowledge bases [23,146]).”

What the sources say

• ?Not checked — no full text of the sourcecites [75] 75. Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016. Neural machine translation with super- vised attention. In Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers (COLING'16). The COLING 2016 Organizing Committee, 3093-3102. source
• Partly supported

  Section 3, “Stanford Sentiment Treebank” (page unavailable), says classifiers can use a few strongly sentiment-bearing words in longer documents, supporting the importance of identifying useful information in text. It does not establish that learning to pay attention is a key step in extracting information from long text sequences; its own experiments focus on phrases and sentences. The citations attached to images and knowledge bases are separate from the ones attached to text. (LLM, conf=0.78)

  cites [129] 129. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1631-1642. sourceExperimental — accuracy not yet measured
• ?Not checked — no full text of the sourcecites [130] 130. Ekta Sood, Simon Tannert, Philipp Mueller, and Andreas Bulling. 2020. Improving natural language processing tasks with human gaze-guided neural attention. In Advances in Neural Information Processing Systems, Vol. 33. 6327-6341. source
• Supported

  The citation is attached to the long-text-sequences part of the sentence, not the images or knowledge-bases examples. In §1, the paper says a good summary covers the input’s principal information; in §3.1, it selects important or principal sentences from documents, and §6.1.2 reports that this selection works best for downstream summarization. (LLM, conf=0.88)

  cites [149] 149. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with extracted gap- sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning (ICML'20). PMLR, 11328-11339. http://proceedings.mlr.press/v119/zhang20ae.html. source
• Supported

  The citation supports the long-text portion of the sentence, not the separately cited claims about images or knowledge bases. In the Introduction and §3.2 (page numbers not supplied), GSum describes selecting informative guidance from source documents so a summarization model attends to relevant information. (LLM, conf=0.90)

  cites [23] 23. Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A general framework for guided neural abstractive summarization. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 4830-4842. https://doi.org/10.18653/v1/2021.naacl-main.384 source
• ?Not checked — no full text of the sourcecites [146] 146. Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woo-Myoung Park. 2021. GPT3Mix: Leveraging large-scale language models for text augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 2225-2239. https://doi.org/10.18653/v1/2021.findings-emnlp.192 source

In the paper

“Data augmentation introduce additional training samples in an explicit way while prompts can be regarded as highly-condensed training samples [65].”

What the sources say

• Partly supported

  The Introduction reports that task-targeted prompts provide a measurable advantage equivalent to additional training data, supporting the comparison of prompts to condensed training samples. The paper does not establish the sentence’s separate claim about data augmentation explicitly adding training samples; its Related Work section only mentions additional pseudo-labeled data in another method. (LLM, conf=0.84)

  cites [65] 65. Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? In Proceedings of the Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2627-2636. https://doi.org/10.18653/v1/2021.naacl-main.208 sourceExperimental — accuracy not yet measured

In the paper

“Later, this idea has been further explored in information extraction [70,142] and text classification [12].”

What the sources say

• Supported

  The Introduction describes reformulating named entity recognition, an information-extraction task, as question answering. This supports the information-extraction part attributed to this paper; the separate text-classification citation is not being evaluated. (LLM, conf=0.93)

  cites [70] 70. Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 5849-5859. https://doi.org/10.18653/v1/2020.acl-main.519 source
• ?Could not verify

  The Abstract (p. 1) and Formulations and notations (p. 3) describe downstream classification using head and prompt tuning, but do not describe recasting text classification as question answering. A search for QA reformulation found no passage in our copy addressing that attributed idea. (LLM, conf=0.73)

  cites [142] 142. Colin Wei, Sang Michael Xie, and Tengyu Ma. 2021. Why do pretrained language models help in downstream tasks? An analysis of head and prompt tuning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS'21), Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 16158-16170. sourceExperimental — accuracy not yet measured
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [12] 12. Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and Jiwei Li. 2020. Description based text classification with reinforce- ment learning. In Proceedings of the International Conference on Machine Learning. PMLR, 1371-1382.

In the paper

“Specifically, the guidance signals could be style tokens [27,123], length specifications [56], domain tags [14], or any variety of other pieces of information used to control of the generated text.”

What the sources say

• Supported

  The citation to this paper is attached to “length specifications,” not the other examples in the sentence. Section 2.2, Length-Constrained Summarization (page not supplied), describes conditioning generated text on a desired length using a length-marker token. (LLM, conf=0.94)

  cites [27] 27. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation. Association for Computational Linguistics, 45-54. source
• ?Could not verify

  The citation is attached specifically to “domain tags.” Section 4.2.2 and the Conclusion discuss adapting translation models to a domain using in-domain data, but the passages found for domain tags and domain indicators do not describe tags as guidance signals for controlling generated text. (LLM, conf=0.76)

  cites [123] 123. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with mono- lingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 86-96. https://doi.org/10.18653/v1/P16-1009 sourceExperimental — accuracy not yet measured
• Supported

  The citation to this paper, [56], is attached specifically to “length specifications,” not to style tokens or domain tags. The Introduction says the proposed models receive the desired output length as input, and Section 6.3 reports that their outputs concentrate near that length. (LLM, conf=0.94)

  cites [56] 56. Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In Proceedings of the Conference on Empirical Methods in Natural Language Pro- cessing. Association for Computational Linguistics, 1328-1338. https://doi.org/10.18653/v1/D16-1140 source
• Supported

  The citation is attached to “domain tags,” not to the other examples of guidance signals. Section 3.2 says a domain tag is appended to source sentences to prime the NMT decoder to generate sentences for a specific domain. (LLM, conf=0.94)

  cites [14] 14. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, 385-391. https://doi.org/10.18653/v1/P17-2061 source

In the paper

“Within the context of pre-trained LMs, researchers have found that pre-trained embeddings and LMs can perform simple operations such as addition and subtraction when the number of digits is small but fail when the numbers are larger [9,88,139].”

What the sources say

• Partly supported

  Section 3.9.1 (p. 22) reports that GPT-3 performs addition and subtraction well with small numbers and that accuracy decreases as digit count increases. It does not test pre-trained embeddings, and GPT-3 still answers some four- and five-digit problems correctly, so 'fail when the numbers are larger' overstates the result. (LLM, conf=0.88)

  cites [9] 9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranz sourceExperimental — accuracy not yet measured
• ?Could not verify

  Section 2 (Analysis Framework) tests number magnitude and numeration, and Section 4.1 reports how embeddings perform on those tests. Neither passage assesses addition or subtraction, or compares arithmetic performance by number of digits; searches for those operations found no such result. The copy also does not test pre-trained language models on arithmetic. (LLM, conf=0.76)

  cites [88] 88. Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. 2019. Exploring numeracy in word em- beddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 3374-3380. https://doi.org/10.18653/v1/P19-1329 sourceExperimental — accuracy not yet measured
• ?Could not verify

  The Sentiment Analysis and Natural Language Inference sections discuss models using pre-trained embeddings, but report task accuracy rather than arithmetic ability. Searches for addition, subtraction, and number of digits found no passage addressing whether performance changes with number size. (LLM, conf=0.76)

  cites [139] 139. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP'19), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 2153-2162. https://doi.org/ 10.18653/v1/D19-1221 sourceExperimental — accuracy not yet measured

In the paper

“Supervised attention [76] aims to provide explicit supervision over the attention of models based on the fact that completely data-driven attention can overfit to some artifacts [73].”

What the sources say

• Supported

  Section 3.2.2 says the model constructs gold attention vectors from annotated arguments and uses them to supervise the attention mechanism. The statement about data-driven attention overfitting to artifacts is attached to a different citation, [73], so it is not attributed to this source. (LLM, conf=0.93)

  cites [76] 76. Shulin Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2017. Exploiting argument information to improve event detection via supervised attention mechanisms. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1789-1798. source
• Supported

  The citation to this paper [76] supports the description of supervised attention: the Introduction (p. 2) says the authors propose a model with explicit supervision of attention maps, and the Supervised Attention Model section (p. 3) explains how they introduce it. The claim about data-driven attention overfitting to artifacts has a separate citation [73] and is not attributed to this source. (LLM, conf=0.93)

  cites [73] 73. Chenxi Liu, Junhua Mao, Fei Sha, and Alan Yuille. 2017. Attention correctness in neural image captioning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence. source

In the paper

“References [61,83] are earlier works that attempt to unify multiple NLP tasks into a QA framework.”

What the sources say

• Supported

  The Introduction (p. 1) says that most NLP tasks can be cast as question-answering problems and proposes the DMN as a framework for several task types. The Experiments (p. 5) test it on question answering, part-of-speech tagging, and sentiment analysis. The Conclusion (p. 8) calls its single architecture a first step toward a joint model, which supports the claim that this work attempts to unify tasks rather than having already done so. (LLM, conf=0.93)

  cites [61] 61. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In Proceedings of the International Conference on Machine Learning. PMLR, 1378-1387. source
• Supported

  The Abstract (p. 1) introduces a challenge spanning ten NLP tasks and states that the authors cast all of them as question answering over a context. This supports identifying the paper as a work that unifies multiple NLP tasks in a QA framework. (LLM, conf=0.94)

  cites [83] 83. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv:1806.08730. Retrieved from https://arxiv.org/abs/1806.08730. source

In the paper

“To solve this problem, supervised attention methods tried to use additional loss functions to learn to predict gold attention on a manually labeled corpus [31,49,102].”

What the sources say

• Supported

  Section 3.1.1 (p. 6) says the authors train an attention network with a cross-entropy loss to match ground-truth attention derived from segmentations in their VQS dataset. The Introduction (p. 2) describes participants selecting the segmentations that answer each question, supporting the description of manually labeled supervision. (LLM, conf=0.93)

  cites [31] 31. Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong. 2017. Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision. 1811-1820. source
• ?Not checked — no full text of the sourcecites [49] 49. Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. 2015. Salicon: Saliency in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1072-1080. source
• Partly supported

  The “Attention supervision for VQA” section (p. 5) describes an additional attention-supervision loss combined with the classification loss. But the supervised VQA model uses attention maps generated automatically by a network trained on human-labeled data, rather than gold attention manually labeled for the corpus on which that model is trained (p. 4). (LLM, conf=0.88)

  cites [102] 102. Tingting Qiao, Jianfeng Dong, and Duanqing Xu. 2018. Exploring human-like attention supervision in visual question answering. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. sourceExperimental — accuracy not yet measured

In the paper

“Reference [96] show that prompts selected under tuned few-shot learning scenario (where one has a larger validation set to choose prompts) generalize well across models of similar sizes while prompts selected under true few-shot learning scenario (where one only has a few training samples) do not generalize as effectively as the former setting among models with similar sizes.”

What the sources say

• Supported

  Section 3.7 (p. 8) says prompts chosen using test accuracy generalize reasonably well across models of similar sizes. It reports weaker transfer for prompts chosen by cross-validation and a further degradation of the size-specific transfer pattern for minimum description length, supporting the comparison in the citing sentence. (LLM, conf=0.91)

  cites [96] 96. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS'21), Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 11054-11070. source

In the paper

“Current works typically select answers before select template [32,125], but Hambardzumyan et al. [40] have demonstrated the initial potential of simultaneously learning both.”

What the sources say

• ?Not checked — no full text of the sourcecites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). source
• ?Could not verify

  The citation to this paper is attached to the claim that current works select answers before templates, not to the separate claim about Hambardzumyan et al. The Dynamic Prompt Creation section discusses selecting examples for a prompt, and Additional information discusses English templates, but neither addresses an answer-before-template selection order. Searches for that order and for answer–template selection found no supporting passage in our copy. (LLM, conf=0.72)

  cites [125] 125. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 7699-7715. https://doi.org/10.18653/v1/2021.emnlp-main.608 sourceExperimental — accuracy not yet measured
• Supported

  Section 3 (WARP) defines a single optimization over both prompt embeddings, which form the template, and verbalizer embeddings, which represent class answers. The comparison with AutoPrompt in §6.2 also reports results from this learned-prompt approach. The separate claim about what current works typically do is cited to other sources, not Hambardzumyan et al. (LLM, conf=0.91)

  cites [40] 40. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial reprogramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 4921-4933. https://doi.org/10.18653/v1/2021.acl-long.381 source

In the paper

“Empirically, for those token [81] or span [30] prediction tasks (e.g., NER), prompt decomposition can be considered, while for those span relation prediction [66] tasks (e.g., entity coreference), prompts composition would be a better choice.”

What the sources say

• Supported

  The citation is attached to “span prediction tasks,” with NER as the example, not to the citing authors’ recommendation about prompt composition or decomposition. The source identifies NER as a sequence-labeling task (Introduction) and uses entity tags to identify named entities (Section 4.1 Data Sets). (LLM, conf=0.88)

  cites [81] 81. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1064-1074. https://doi.org/10.18653/v1/P16-1101 source
• Supported

  The citation [30] is attached to “span prediction tasks,” with NER given as the example. The Introduction (§1) explicitly describes NER as a span-level prediction task. The recommendation about prompt decomposition is the citing authors’ own framing, not a claim attributed to this source. (LLM, conf=0.92)

  cites [30] 30. Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021. SpanNER: Named entity re-/recognition as span prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP'21), Volume 1: Long Papers, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, 7183-7195. https://doi.org/10.18653/v1/ 2021.acl-long.558 source
• Supported

  The citation follows “span relation prediction,” and the Task section describes coreference resolution as predicting antecedents that represent links between text spans. This supports the cited paper as an example of that task type; the sentence’s prompt-composition recommendation is not attributed to this citation. (LLM, conf=0.92)

  cites [66] 66. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 188-197. https://doi.org/10.18653/v1/D17-1018 source

In the paper

“So far, almost all works use prompt learning for text generation relying solely on a single answer, with only a few exceptions [52].”

What the sources say

• ?Could not verify

  The Introduction (p. 1) describes prompts used to generate answers or predict missing words, while §4.2 (p. 4) describes this paper’s method for combining multiple prompts. Neither passage establishes that almost all prompt-learning work on text generation relies on a single answer, or that there are only a few exceptions. Searches for those claims and for prior single-prompt work found no supporting passage. (LLM, conf=0.72)

  cites [52] 52. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Trans. Assoc. Comput. Ling. 8 (2020), 423-438. https://doi.org/10.1162/tacl_a_00324 sourceExperimental — accuracy not yet measured

In the paper

“Even during this stage, there was some use of pre-trained LMs exemplified by word2vec[85,86] and GloVe[95], but they were used for only a limited portion of the final model parameters.”

What the sources say

• Partly supported

  Section 1.2 (p. 2) says this paper focuses on learning word vectors separately from a full neural language model and notes that word vectors can be used in NLP applications. Section 7 (p. 11) also reports that the authors published vectors for others to use. The paper does not establish that such vectors occupied only a limited portion of a downstream model’s final parameters; the separate GloVe citation is not judged here. (LLM, conf=0.82)

  cites [85] 85. Tomas Mikolov, Kai Chen, G. Corrado, and J. Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR'13). sourceExperimental — accuracy not yet measured
• Partly supported

  The Introduction (p. 1) and Conclusion (p. 8) identify this work as training Skip-gram word representations, supporting the citation as an example of word2vec. The paper does not describe word2vec as a pre-trained language model or establish that its representations occupied only a limited portion of a final model’s parameters. (LLM, conf=0.82)

  cites [86] 86. Tomas Mikolov, Ilya Sutskever, Kai Chen, G. Corrado, and J. Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the Conference and Workshop on Neural Information Processing Systems (NIPS'13). sourceExperimental — accuracy not yet measured
• ?Not checked — no full text of the sourcecites [95] 95. Jeffrey Pennington, R. Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'14). source

In the paper

“Existing prompt augmentation methods are limited by the input length, i.e., feeding too many demonstrations to input is infeasible.”

In the paper

“So far this field has not been explored.”

In the paper

“So far, there are few to no systematic comparisons of the benefits brought by prompt-based learning for different pre-trained LMs.”

In the paper

“For text generation tasks, the study of prompt ensemble learning has not been performed so far, probably because ensemble learning in text generation itself is relatively complicated.”

In the paper

“In addition, how to select ensemble-worthy prompts is also under-explored.”

In the paper

“As discussed in Section 6, there are a fairly wide variety of methods for tuning parameters of prompts, LMs, or both.”

In the paper

“Multiple continuous prompts are typically learned by using different initializations or different random seeds.”

In the paper

“How to distill the knowledge from different prompts remains underexplored.”

In the paper

“All the above considerations refer to the application of prompt in a single task, domain or language.”

In the paper

“However, moving beyond this to more complicated varieties of structure is largely unexplored, and a potentially interesting research area.”

In the paper

“Most existing works about prompt-based learning revolve around either text classification or generation-based tasks.”

In the paper

“This is an important research question that has not been covered extensively by the literature.”

In the paper

“However, given the nascent stage of this research field, we still lack a systematic understanding of the tradeoffs between these methods.”

In the paper

“Paraphrasing-based approaches take in an existing seed prompt (e.g., manually constructed or mined), paraphrase it into a set of other candidate prompts, and then selects the one that achieves the highest training accuracy on the target task.”

In the paper

“Gao et al. [32] introduce the seq2seq pre-trained LM T5 into the template search process.”

What the sources say

• ?Not checked — no full text of the sourcecites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). source

In the paper

“Ben-David et al. [5] propose a domain adaptation algorithm that trains T5 to generate unique domain relevant features (DRFs) (a set of keywords that characterize domain information) for each input.”

What the sources say

• ?Not checked — no full text of the sourcecites [5] 5. Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022. PADA: Example-based prompt learning for on-the-fly adapta- tion to unseen domains. Trans. Assoc. Comput. Linguist. 10 (4 2022), 414-433. https://doi.org/10.1162/tacl_a_00468 source

In the paper

“Finally, for some tasks regarding multiple inputs such as text pair classification, prompt templates must contain space for two inputs, [X1] and [X2], or more.”

In the paper

“While the strategy of manually crafting templates is intuitive and does allow solving various tasks with some degree of accuracy”

In the paper

“Given that the prompt specifies the task, choosing a proper prompt template has a large effect not only on the accuracy but also on which task the model performs in the first place.”

In the paper

“continuous prompts remove two constraints: (1) relax the constraint that the embeddings of template words be the embeddings of natural language (e.g., English) words and (2) remove the restriction that the template is parameterized by the pre-trained LM's parameters.”

In the paper

“prompt-based learning is based on language models that model the probability of text directly.”

In the paper

“It allows the language model to be pre-trained on massive amounts of raw text”

In the paper

“In many cases, these template words are not necessarily composed of natural language tokens; they could be virtual words (e.g., represented by numeric ids) that would be embedded in a continuous space later, and some prompting methods even generate continuous vectors directly”

In the paper

“This article surveys and organizes research works in a new paradigm in natural language processing, which we dub”

In the paper

“For tasks that are solved using masked LMs, cloze prompts are a good fit, as they very closely match the form of the pre-training task.”

In the paper

“The number of [X] slots and the number of [Z] slots can be flexibly changed for the need of tasks at hand.”

In the paper

“Full text reconstruction models are more versatile and can be used with either cloze or prefix prompts.”

In the paper

“This work found that initializing with manual templates can provide a better starting point for the search process.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [152] 152. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML'21), Proceedings of Machine Learning Research, Vol. 139, Marina Meila and Tong Zhang (Eds.). PMLR, 12697-12706. http:// proceedings.mlr.press/v139/zhao21c.html.

In the paper

“When performing relation extraction, Chen et al. [13] automatically decompose each relation label into its constituent words and use them as an answer.”

What the sources say

• ?Not checked — no full text of the sourcecites [13] 13. Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. KnowPrompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference (WWW'22), Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (Eds.). ACM, 2778-2788. https://doi.org/10.1145/3485447. 3511998 source

In the paper

“They show that the resulting model can perform few-shot learning for vision-language tasks such as visual question answering, and so on.”

What the sources say

• ?Not checked — no full text of the sourcecites [135] 135. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. arXiv:1806.02847. Retrieved from http://arxiv.org/abs/1806.02847. source

In the paper

“To account for this, some works also explore to use of weighted averages for prompt ensembling where each prompt is associated with a weight.”

In the paper

“To enhance the representation ability of the resulting template, they also insert several virtual tokens whose embeddings can be tuned together with the pre-trained LMs parameters using training samples.”

What the sources say

• ?Not checked — no full text of the sourcecites [41] 41. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. PTR: Prompt Tuning with Rules for Text Classification. arXiv:2105.11259 [cs.CL]. Retrieved from https://arxiv.org/abs/2105.11259. source

In the paper

“As with manually created prompts, it is possible that manually created answers are sub-optimal for getting the LM to achieve ideal prediction performance.”

In the paper

“Yin et al. [144] manually design lists of words relating to relevant topics (”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [144] 144. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Gen- eralized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS'19), Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 5754-5764.

In the paper

“This sort of prompt ensembling can (1) leverage the complementary advantages of different prompts, (2) alleviate the cost of prompt engineering, since choosing one best-performing prompt is challenging, and (3) stabilize performance on downstream tasks.”

In the paper

“Han et al. [41] propose prompt tuning with rules (PTR), which uses manually crafted sub-templates to compose a complete template using logic rules.”

What the sources say

• ?Not checked — no full text of the sourcecites [41] 41. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. PTR: Prompt Tuning with Rules for Text Classification. arXiv:2105.11259 [cs.CL]. Retrieved from https://arxiv.org/abs/2105.11259. source

In the paper

“Gao et al. [32] first construct a pruned search space Z by selecting top-k vocabulary words based on their generation probability at the [Z] position determined by training samples.Then the search space is further pruned down by only selecting a subset of words within Z based on their zero-shot accuracy on the training samples.In the search step, they fine-tune the LM with fixed templates together”

What the sources say

• ?Not checked — no full text of the sourcecites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). source

In the paper

“The key to prompting for classification-based tasks is reformulating it as an appropriate prompt.”

In the paper

“Because prompts are the only method that provide the task specification, heavy engineering is necessary to achieve high accuracy.”

In the paper

“Similarly to tuning-free prompting, it can retain knowledge in LMs and is suitable in few-shot scenarios.”

In the paper

“Chen et al. [13] first explored the application of fixed-prompt LM Tuning in relation extraction and discuss two major challenges that hinder the direct inheritance of prompting methodology from classification tasks: (1) The larger label space (e.g., 80 in relation extraction vs. 2 in binary sentiment classification) results in more difficulty in prompt answer engineering.”

What the sources say

• ?Not checked — no full text of the sourcecites [13] 13. Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. KnowPrompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference (WWW'22), Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (Eds.). ACM, 2778-2788. https://doi.org/10.1145/3485447. 3511998 source

In the paper

“Tuning all the LM parameters allows the model to fit to larger training datasets.”

In the paper

“An ensemble of deep learning models can typically improve the performance, and this superior performance can be distilled into a single model using knowledge distillation [2].”

What the sources say

• ?Not checked — no full text of the sourcecites [2] 2. Zeyuan Allen-Zhu and Yuanzhi Li. 2020. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. arXiv:2012.09816. Retrieved from https://arxiv.org/abs/2012.09816. source

In the paper

“While effective in few-shot scenarios, representation power is limited in large-data settings.”

In the paper

“Notably, this setting is very similar to the standard pre-train and fine-tune paradigm, but the addition of the prompt can provide additional bootstrapping at the start of model training.”

In the paper

“Not applicable in zero-shot scenarios.”

In the paper

“It is not easy to predict all the span types at the same time due to the large number of spans, so different prompts for each span can be created and predicted separately.”

In the paper

“May overfit to small datasets.”

In the paper

“This is the most expressive method, likely suitable for high-data settings.”

In the paper

“LMs fine-tuned on one downstream task may not be effective on another one.”

In the paper

“In particular in the in-context learning setting, providing many answered prompts can be slow at test time and thus cannot easily use large training datasets.”

In the paper

“Gao et al. [32] use a similar ensemble method on their automatically generated templates.”

What the sources say

• ?Not checked — no full text of the sourcecites [32] 32. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL'21). source

In the paper

“Regarding prompt engineering, researchers mainly focus on the template search in the few-shot learning setting and the answer space Z is usually manually preselected from the vocabulary.”

In the paper

“However, this is a common setting and effective in the controlled text generation, which may provide valuable direction for the future work on prompt learning.”

In the paper

“Different from works such as PTR, or AdaPrompt, argRanker's new formulation does not use the entirety of the pre-trained LM's architecture (e.g., the word prediction layer of the BERT model).”

In the paper

“Controlled generation aims to incorporate various types of guidance beyond the input text into the generation model [146].”

What the sources say

• ?Not checked — no full text of the sourcecites [146] 146. Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woo-Myoung Park. 2021. GPT3Mix: Leveraging large-scale language models for text augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 2225-2239. https://doi.org/10.18653/v1/2021.findings-emnlp.192 source

In the paper

“In contrast, the main motivation for using prompts for text generation is to specify the task itself and better utilize the pre-trained LM.”

In the paper

“Both add extra information to the input text for better generation, and these additional signals are (often) learnable parameters.”

In the paper

“Controlled generation targets generation of a particular type of text while prompt learning uses prompts to specify the task itself.”

In the paper

“There are several commonalities between prompt-based learning and query reformulation, for example (1) both aim to make better use of some existing knowledge bases by asking a right questions and (2) the knowledge bases are usually a black-box, not available to the users, so researchers must learn how to probe it optimally based on solely questions.”

In the paper

“Overall, the application of prompt-based learning in tagging task has not been fully explored.”

In the paper

“Research works on supervised attention usually target at salient information from an image or text, while prompt learning aims to utilize relevant knowledge from the pre-trained LM.”

In the paper

“Prompt augmentation is a specific subset of few-shot learning.”

In the paper

“Opitz [93]'s argRanker method conceptualizes argumentative relation classification as a ranking problem for two sentences that are connected by manually defined phrases.”

What the sources say

• ?Not checked — no full text of the sourcecites [93] 93. Juri Opitz. 2019. Argumentative relation classification as plausibility ranking. arXiv:1909.09031. Retrieved from https://arxiv.org/abs/1909.09031. source

In the paper

“In a way, many of the prompting methods described here are a type of controllable generation, where the prompt is usually used to specify the task itself.”

In the paper

“However, one of the key points of prompting methods is how to better use the knowledge in pre-trained LMs, and these were not covered extensively on previous works advocating for QA formulations.”

In the paper

“In prompt ensembling, multiple predictions result from different prompt variants.”

In the paper

“Additional information introduced in larger-context learning is not necessarily the labeled data.”

In the paper

“QA-based formulations aim to solve different tasks through question answering, while prompting additionally targets full use of pre-trained LMs.”

In the paper

“Compared to previous methods, prompt augmentation directly prepends several labeled samples to the currently processed sample to elicit knowledge from pre-trained LMs even without any parameter tuning.”

In the paper

“They do not necessarily require a pre-trained LM.”

In the paper

“In prompt ensembling methods, the space and time complexity increase as we consider more prompts.”

In the paper

“Wei et al. [141] showed that soft-prompt tuning can relax the non-degeneracy assumptions (the generation probability of each token is linearly independent) needed for downstream recovery (i.e., recover the ground-truth labels of the downstream task.).making it easier to extract task-specific information.”

What the sources say

• ?Not checked — no full text of the sourcecites [141] 141. Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM Comput. Surv. 53, 3 (2020), 1-34. source

In the paper

“For text generation tasks, qualified answers can be semantically equivalent but syntactically diverse.”

In the paper

“Aghajanyan et al. [1] present structured prompts based on hyper text markup language for more fine-grained web text generation.”

What the sources say

• ?Not checked — no full text of the sourcecites [1] 1. Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. HTLM: Hyper-text pre-training and prompting of language models. arXiv:2107.06955. Retrieved from https:// arxiv.org/abs/2107.06955. source

In the paper

“When there are too many classes, how to select an appropriate answer space becomes a difficult combinatorial optimization problem.”

In the paper

“Even though we have a calibrated probability distribution, we also need to be careful when we assume a single gold answer for an input.”

In the paper

“For example, if we consider the gold answer to be”

In the paper

“However, this method has two drawbacks: (1) it comes with the overhead of finding proper context-free input (e.g., whether to use”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [151] 151. Yue Zhang and Joakim Nivre. 2011. Transition-based dependency parsing with rich non-local features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Associa- tion for Computational Linguistics, 188-193. https://www.aclweb.org/anthology/P11-2033.

In the paper

“Although prompt-based learning has shown significant potential among different tasks and scenarios, several challenges remain, some of which we detail below.”

In the paper

“This paradigm is less conducive to architectural exploration, because (i) unsupervised pre-training allows models to learn with fewer structural priors, and (ii) as pre-training of models is time-consuming, experimenting with structural variants is costly.”

In the paper

“The number of template tokens do not need to be pre-specified, since T5 can decode multiple tokens at a masked position.”

In the paper

“In this respect, prompt learning and supervised attention share ideas that both aim to extract salient information with some clues, which need to be provided separately.”

In the paper

“Applications to information extraction and text analysis tasks have been discussed less, largely because the design of prompts is less straightforward.”

In the paper

“Notably, much of the success of the prompting paradigm is built on top of pre-trained LMs that were developed for the pre-train and fine-tune paradigm, such as BERT.”