CoChat
Sign Up
Cited by 180Topic ModelingNatural Language Processing TechniquesSentiment Analysis and Opinion Mining

Large Language Models for Data Annotation and Synthesis: A Survey

Open access
Tan, Zhen, Li, Dawei, Wang, Song, Beigi, Alimohammad, Jiang, Bohan, Bhattacharjee, Amrita, Karami, Mansooreh, Li, Jundong +2 more·2024·Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2024.emnlp-main.54
Open this paper in CoChat — free accountOpen PDF

Abstract

Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, Huan Liu. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.emnlp-main.54

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Using other VLMs to synthetically generate annotations is a popular approach, but it is likely to perpetuate inaccuracies or biases that the model already has, rather than capturing important nuances.

    “Using other VLMs to synthetically generate annotations is a popular approach [84, 119], but it is likely to perpetuate inaccuracies or biases that the model already has (see distribution shift [110]), rather than capturing important nuances.”

    From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [119]

    Section 3 describes LLM-generated annotation as an area of significant interest, supporting the approach cited in the first clause. But the Introduction says the survey focuses on pure language models and does not extensively cover multimodal models, so it does not establish that using VLMs specifically is popular. The later claim about perpetuating inaccuracies or biases has a separate citation in the citing sentence. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 253

  1. Sujan Reddy A. 2022. Automating human evaluation of dialogue systems. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 229-234, Hybrid: Seattle, Washington + On- line. Association for Computational Linguistics. · DOI
  2. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298-306.
  3. Arkadeep Acharya, Brijraj Singh, and Naoyuki Onoe. 2023. Llm based generation of item-description for recommendation system. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 1204-1207.
  4. Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models are zero-shot clinical information extractors. arXiv preprint arXiv:2205.12689. · arXiv
  5. Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2024. Can llms replace manual annotation of software engineering artifacts? arXiv preprint arXiv:2408.05534. · arXiv
  6. Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Reza Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard Bara- niuk. 2023. Self-consuming generative models go mad. ArXiv, abs/2307.01850. · arXiv
  7. Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Ko- robeynikova, and Fabrizio Gilardi. 2023. Open- source large language models outperform crowd workers and approach chatgpt in text-annotation tasks. arXiv preprint arXiv:2307.02179. · arXiv
  8. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2023. Learn- ing from mistakes makes llm better reasoner. arXiv preprint arXiv:2310.20689. · arXiv
  9. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. 2021. Efficient large scale language mod- eling with mixtures of experts. arXiv preprint arXiv:2112.10684. · arXiv
  10. Razvan Azamfirei, Sapna R Kudchadkar, and James Fackler. 2023. Large language models and the perils of their hallucinations. Critical Care, 27(1):1-2.
  11. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. · arXiv
  12. Nishant Balepur, Shramay Palta, and Rachel Rudinger. 2023. It's not easy being wrong: Evaluating process of elimination reasoning in large language models. arXiv preprint arXiv:2311.07532. · arXiv
  13. Alimohammad Beigi, Zhen Tan, Nivedh Mudiam, Canyu Chen, Kai Shu, and Huan Liu. 2024. Model attribution in machine-generated disinformation: A domain generalization approach with supervised con- trastive learning. arXiv preprint arXiv:2407.21264. · arXiv
  14. Thales Bertaglia, Stefan Huber, Catalina Goanta, Gerasi- mos Spanakis, and Adriana Iamnitchi. 2023. Closing the loop: Testing chatgpt to generate model expla- nations to improve human labelling of sponsored content on social media. In World Conference on Explainable Artificial Intelligence, pages 198-213. Springer.
  15. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gersten- berger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Pi- otr Nyczyk, et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682-17690.
  16. Amrita Bhattacharjee, Raha Moraffah, Joshua Gar- land, and Huan Liu. 2024. Zero-shot llm-guided counterfactual generation for text. arXiv preprint arXiv:2405.04793. · arXiv
  17. Zhen Bi, Jing Chen, Yinuo Jiang, Feiyu Xiong, Wei Guo, Huajun Chen, and Ningyu Zhang. 2024. Codekgc: Code language model for generative knowledge graph construction. ACM Transactions on Asian and Low-Resource Language Information Process- ing, 23(3):1-16.
  18. Ning Bian, Peilin Liu, Xianpei Han, Hongyu Lin, Yao- jie Lu, Ben He, and Le Sun. 2023. A drop of ink may make a million think: The spread of false in- formation in large language models. arXiv preprint arXiv:2305.04812. · arXiv
  19. Julia Bonn, Harish Tayyar Madabushi, Jena D Hwang, and Claire Bonial. 2024. Adjudicating llms as prop- bank annotators. LREC-COLING 2024, page 112.
  20. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. · arXiv
  21. Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. 2023. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on robot learn- ing, pages 287-318.
  22. PMLR.
  23. Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data cre- ation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094. · arXiv
  24. Manav Chaudhary, Harshit Gupta, and Vasudeva Varma. 2024. Brainstorm@ irel at smm4h 2024: Leveraging translation and topical embeddings for annotation de- tection in tweets. arXiv preprint arXiv:2405.11192. · arXiv
  25. Canyu Chen and Kai Shu. Can llm-generated misinfor- mation be detected? In The Twelfth International Conference on Learning Representations.
  26. Canyu Chen and Kai Shu. 2023. Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788. · arXiv
  27. Kai Chen, Chunwei Wang, Kuo Yang, Jianhua Han, Lanqing Hong, Fei Mi, Hang Xu, Zhengying Liu, Wenyong Huang, Zhenguo Li, et al. 2023a. Gain- ing wisdom from setbacks: Aligning large lan- guage models via mistake analysis. arXiv preprint arXiv:2310.10477. · arXiv
  28. Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur. 2023b. Places: Prompting language models for social conversation synthesis. In Findings of the Association for Compu- tational Linguistics: EACL 2023, pages 844-868.
  29. Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Andy Rosenbaum, Seokhwan Kim, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur. 2022. Weakly supervised data augmentation through prompting for dialogue understanding. In NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research.
  30. Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Ken- neth Heafield. 2023c. Iterative translation refine- ment with large language models. arXiv preprint arXiv:2306.03856. · arXiv
  31. Ruirui Chen, Chengwei Qin, Weifeng Jiang, and Dongkyu Choi. 2024a. Is a large language model a good annotator for event extraction? In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17772-17780.
  32. Wei-Lin Chen, Cheng-Kuang Wu, Yun-Nung Chen, and Hsin-Hsi Chen. 2023d. Self-icl: Zero-shot in-context learning with self-generated demonstrations. In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 15651- 15662.
  33. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2023e. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. Transactions on Machine Learning Research.
  34. Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Ke- fan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023f. Uni- versal self-consistency for large language model gen- eration. arXiv preprint arXiv:2311.17311. · arXiv
  35. Yunkai Chen, Qimeng Wang, Shiwei Wu, Yan Gao, Tong Xu, and Yao Hu. 2024b. Tomgpt: Reliable text-only training approach for cost-effective multi- modal large language model. ACM Transactions on Knowledge Discovery from Data.
  36. Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. 2023g. Disco: Dis- tilling counterfactuals with large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5514-5528.
  37. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024c. Self-play fine-tuning con- verts weak language models to strong language mod- els. arXiv preprint arXiv:2401.01335. · arXiv
  38. Lu Cheng, Kush R Varshney, and Huan Liu. 2021. So- cially responsible ai algorithms: Issues, purposes, and challenges. Journal of Artificial Intelligence Re- search, 71:1137-1181.
  39. Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, and Nan Du. 2024. Self-playing adversarial language game enhances llm reasoning. arXiv preprint arXiv:2404.10642. · arXiv
  40. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023b. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6.
  41. Zhixuan Chu, Yan Wang, Longfei Li, Zhibo Wang, Zhan Qin, and Kui Ren. 2024a. A causal explainable guardrails for large language models. arXiv preprint arXiv:2405.04160. · arXiv
  42. Zhumin Chu, Qingyao Ai, Yiteng Tu, Haitao Li, and Yiqun Liu. 2024b. Pre: A peer review based large language model evaluator. arXiv preprint arXiv:2401.15641. · arXiv
  43. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. · arXiv
  44. Roi Cohen, May Hamri, Mor Geva, and Amir Glober- son. 2023. Lm vs lm: Detecting factual errors via cross examination. arXiv preprint arXiv:2305.13281. · arXiv
  45. Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092. · arXiv
  46. Amit Das, Zheng Zhang, Fatemeh Jamshidi, Vinija Jain, Aman Chadha, Nilanjana Raychawdhary, Mary Sandage, Lauramarie Pope, Gerry Dozier, and Cheryl Seals. 2024. Investigating annotator bias in large language models for hate speech detection. arXiv preprint arXiv:2406.11109. · arXiv
  47. Yihe Deng, Weitong Zhang, Zixiang Chen, and Quan- quan Gu. 2023. Rephrase and respond: Let large language models ask better questions for themselves. arXiv preprint arXiv:2311.04205. · arXiv
  48. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805. · arXiv
  49. Linyi Ding, Sizhe Zhou, Jinfeng Xiao, and Ji- awei Han. 2024. Automated construction of theme-specific knowledge graphs. arXiv preprint arXiv:2404.19146. · arXiv
  50. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767. · arXiv
  51. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenen- baum, and Igor Mordatch. 2023a. Improving fac- tuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325. · arXiv
  52. Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Co- las, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. 2023b. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, pages 8657-8677.
  53. PMLR.
  54. Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions? arXiv preprint arXiv:2010.11982. · arXiv
  55. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130. · arXiv
  56. Yunlong Feng, Yang Xu, Libo Qin, Yasheng Wang, and Wanxiang Che. 2024. Improving language model rea- soning with self-motivated learning. arXiv preprint arXiv:2404.07017. · arXiv
  57. Zachary N Flamholz, Steven J Biller, and Libusha Kelly. 2024. Large language models improve annotation of prokaryotic viral proteins. Nature Microbiology, 9(2):537-549.
  58. Johann Frei and Frank Kramer. 2023. Annotated dataset creation through large language models for non- english medical nlp. Journal of Biomedical Infor- matics, 145:104478.
  59. Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. 2023. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142. · arXiv
  60. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang, and Xiaojun Wan. 2023. Human-like sum- marization evaluation with chatgpt. arXiv preprint arXiv:2304.02554. · arXiv
  61. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 3816-3830.
  62. Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, An- drey Gromov, et al. 2024. Is model collapse in- evitable? breaking the curse of recursion by ac- cumulating real and synthetic data. arXiv preprint arXiv:2404.01413. · arXiv
  63. Arnav Gudibande, Eric Wallace, Charles Burton Snell, Xinyang Geng, Hao Liu, P. Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary llms. ArXiv, abs/2305.15717. · arXiv
  64. Caglar Gulcehre, Tom Le Paine, Srivatsan Srini- vasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. Reinforced self- training (rest) for language modeling. arXiv preprint arXiv:2308.08998. · arXiv
  65. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Ce- sar Teodoro Mendes, Allison Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero C. Kauffmann, Gus- tavo de Rosa, Olli Saarikivi, Adil Salim, S. Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuan-Fang Li. 2023. Textbooks are all you need. ArXiv, abs/2306.11644. · arXiv
  66. Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xi- aoying Zhang, Zhaoran Wang, and Yang Liu. 2024a. Human-instruction-free llm self-alignment with lim- ited samples. arXiv preprint arXiv:2401.06785. · arXiv
  67. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. 2024b. Direct language model alignment from online ai feed- back. arXiv preprint arXiv:2402.04792. · arXiv
  68. Himanshu Gupta, Kevin Scaria, Ujjwala Anan- theswaran, Shreyas Verma, Mihir Parmar, Saurabh Arjun Sawant, Swaroop Mishra, and Chitta Baral. 2023. Targen: Targeted data gener- ation with large language models. arXiv preprint arXiv:2310.17876. · arXiv
  69. Huy Ha, Pete Florence, and Shuran Song. 2023. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning, pages 3766-3777. PMLR.
  70. Endre Hamerlik, Marek Šuppa, Miroslav Blšták, Jozef Kubík, Martin Takáč, Marián Šimko, and Andrej Findor. 2024. Chatgpt as your n-th annotator: Exper- iments in leveraging large language models for social science text annotation in slovak language. In Pro- ceedings of the 4th Workshop on Computational Lin- guistics for the Political and Social Sciences: Long and short papers, pages 81-89.
  71. Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. Rea- soning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154-8173.
  72. Wei He, Shichun Liu, Jun Zhao, Yiwen Ding, Yi Lu, Zhi- heng Xi, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Self-demos: Eliciting out-of-demonstration generalizability in large language models. arXiv preprint arXiv:2404.00884. · arXiv
  73. Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854. · arXiv
  74. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 14852-14882.
  75. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor. arXiv preprint arXiv:2212.09689. · arXiv
  76. Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2023. Large language models are zero-shot rankers for recommender systems. arXiv preprint arXiv:2305.08845. · arXiv
  77. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301. · arXiv
  78. Baixiang Huang, Canyu Chen, and Kai Shu. 2024a. Authorship attribution in the era of llms: Prob- lems, methodologies, and challenges. arXiv preprint arXiv:2408.08946. · arXiv
  79. Baixiang Huang, Canyu Chen, and Kai Shu. 2024b. Can large language models identify authorship? arXiv preprint arXiv:2403.08213. · arXiv
  80. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051-1068.
  81. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, pages 9118-9147. PMLR.
  82. Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. 2023. Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprint arXiv:2301.01820. · arXiv
  83. Bohan Jiang, Zhen Tan, Ayushi Nirmal, and Huan Liu. 2024a. Disinformation detection: An evolving chal- lenge in the age of llms. In Proceedings of the 2024 SIAM International Conference on Data Mining (SDM), pages 427-435. SIAM.
  84. Chunyang Jiang, Chi-min Chan, Wei Xue, Qifeng Liu, and Yike Guo. 2024b. Importance weighting can help large language models self-improve. arXiv preprint arXiv:2408.09849. · arXiv
  85. Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. 2023. Exploiting asymmetry for syn- thetic training data generation: Synthie and the case of information extraction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1555-1574.
  86. Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. 2024. Knowledge- augmented reasoning distillation for small language models in knowledge-intensive tasks. Advances in Neural Information Processing Systems, 36.
  87. Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2022. Self-generated in-context learning: Leveraging auto- regressive language models as a demonstration gen- erator. arXiv preprint arXiv:2206.08082. · arXiv
  88. Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, et al. 2023a. Soda: Million-scale dialogue distillation with social com- monsense contextualization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12930-12949.
  89. Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Yoo, and Minjoon Seo. 2023b. Aligning large language models through synthetic feedback. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 13677-13700.
  90. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
  91. Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schuetze. Longform: Effective instruction tuning with reverse instructions. In ICLR 2024 Work- shop on Navigating and Addressing Data Problems for Foundation Models.
  92. Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. 2022. Reward design with language models. In The Eleventh International Conference on Learning Representations.
  93. Daniil Larionov, Artem Shelmanov, Elena Chistova, and Ivan Smirnov. 2019. Semantic role labeling with pre- trained language models for known and unknown predicates. In Proceedings of the International Con- ference on Recent Advances in Natural Language Processing (RANLP 2019), pages 619-628.
  94. Md Tahmid Rahman Laskar, Mizanur Rahman, Israt Jahan, Enamul Hoque, and Jimmy Huang. 2023. Can large language models fix data annotation errors? an empirical study using debatepedia for query-focused text summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10245-10255.
  95. Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen White, and Sujay Jauhar. 2023a. Making large language models better data creators. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15349-15360.
  96. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Car- bune, and Abhinav Rastogi. 2023b. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. · arXiv
  97. Kyungjae Lee, Dasol Hwang, Sunghyun Park, Young- soo Jang, and Moontae Lee. 2024a. Reinforcement learning from reflective feedback (rlrf): Aligning and improving llms via fine-grained self-reflection. arXiv preprint arXiv:2403.14238. · arXiv
  98. Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Min- joon Seo, Kang Min Yoo, and Youngjae Yu. 2024b. Aligning large language models by on-policy self- judgment. arXiv preprint arXiv:2402.11253. · arXiv
  99. Dawei Li, William Hogan, and Jingbo Shang. 2024a. Read: Improving relation extraction from an adver- sarial perspective. arXiv preprint arXiv:2404.02931. · arXiv
  100. Dawei Li, Yaxuan Li, Dheeraj Mekala, Shuyao Li, Xueqi Wang, William Hogan, Jingbo Shang, et al. 2023a. Dail: Data augmentation for in- context learning via self-paraphrase. arXiv preprint arXiv:2311.03319. · arXiv
  101. Dawei Li, Zhen Tan, Tianlong Chen, and Huan Liu. 2024b. Contextualization distillation from large lan- guage model for knowledge graph completion. arXiv preprint arXiv:2402.01729. · arXiv
  102. Dawei Li, Shu Yang, Zhen Tan, Jae Young Baik, Sunkwon Yun, Joseph Lee, Aaron Chacko, Bojian Hou, Duy Duong-Tran, Ying Ding, et al. 2024c. Dalk: Dynamic co-augmentation of llms and kg to answer alzheimer's disease questions with scientific literature. arXiv preprint arXiv:2405.04819. · arXiv
  103. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2024d. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36.
  104. Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. 2023b. Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 1487-1505.
  105. Renhao Li, Minghuan Tan, Derek F Wong, and Min Yang. 2024e. Coevol: Constructing better responses for instruction finetuning through multi-agent coop- eration. arXiv preprint arXiv:2406.07054. · arXiv
  106. Rui Li, Guoyin Wang, and Jiwei Li. 2023c. Are human- generated demonstrations necessary for in-context learning? arXiv preprint arXiv:2309.14681. · arXiv
  107. Ruosen Li, Teerth Patel, and Xinya Du. 2023d. Prd: Peer rank and discussion improve large lan- guage model based evaluations. arXiv preprint arXiv:2307.02762. · arXiv
  108. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2023e. Self-alignment with instruction back- translation. In The Twelfth International Conference on Learning Representations.
  109. Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, and Yunchao Wei. 2023f. Stablellava: Enhanced visual instruction tuning with synthesized image-dialogue data. arXiv preprint arXiv:2308.10253. · arXiv
  110. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957. · arXiv
  111. Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. 2023g. A survey on fairness in large language models. arXiv preprint arXiv:2308.10149. · arXiv
  112. Hao Liang, Linzhuang Sun, Jingxuan Wei, Xijie Huang, Linkun Sun, Bihui Yu, Conghui He, and Wen- tao Zhang. 2024a. Synth-empathy: Towards high- quality synthetic empathy data. arXiv preprint arXiv:2407.21669. · arXiv
  113. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118. · arXiv
  114. Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, et al. 2024b. I-sheep: Self-alignment of llm from scratch through an iter- ative self-enhancement paradigm. arXiv preprint arXiv:2408.08072. · arXiv
  115. Q Vera Liao and Jennifer Wortman Vaughan. 2023. Ai transparency in the age of llms: A human-centered research roadmap. arXiv preprint arXiv:2306.01941. · arXiv
  116. Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. 2023a. Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 47(8):1345-1365.
  117. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. · arXiv
  118. Yen-Ting Lin, Alexandros Papangelis, Seokhwan Kim, Sungjin Lee, Devamanyu Hazarika, Mahdi Namazi- far, Di Jin, Yang Liu, and Dilek Hakkani-Tur. 2023b. Selective in-context data augmentation for intent de- tection using pointwise v-information. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1463-1476.
  119. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158-167.
  120. Hanmeng Liu, Zhiyang Teng, Leyang Cui, Chaoli Zhang, Qiji Zhou, and Yue Zhang. 2023a. Logicot: Logical chain-of-thought instruction tuning. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  121. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. arXiv preprint arXiv:2304.08485. · arXiv
  122. Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. 2024a. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503. · arXiv
  123. Tengxiao Liu, Qipeng Guo, Yuqing Yang, Xiangkun Hu, Yue Zhang, Xipeng Qiu, and Zheng Zhang. 2023c. Plan, verify and switch: Integrated reasoning with diverse x-of-thoughts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2807-2822.
  124. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023d. Trust- worthy llms: a survey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2308.05374. · arXiv
  125. Zheng Liu, Hao Liang, Wentao Xiong, Qinhan Yu, Con- ghui He, Bin Cui, and Wentao Zhang. 2024b. Syn- thvlm: High-efficiency and high-quality synthetic data for vision language models. arXiv preprint arXiv:2407.20756. · arXiv
  126. Zhili Liu, Yunhao Gou, Kai Chen, Lanqing Hong, Ji- ahui Gao, Fei Mi, Yu Zhang, Zhenguo Li, Xin Jiang, Qun Liu, et al. 2024c. Mixture of insightful ex- perts (mote): The synergy of thought chains and expert mixtures in self-alignment. arXiv preprint arXiv:2405.00557. · arXiv
  127. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2023e. Dynamic llm-agent network: An llm- agent collaboration framework with agent team opti- mization. arXiv preprint arXiv:2310.02170. · arXiv
  128. Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Fei Mi, Baojun Wang, Weichao Wang, Lifeng Shang, and Qun Liu. 2023. Self: Language-driven self-evolution for large language model. arXiv preprint arXiv:2310.00533. · arXiv
  129. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian- guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wiz- ardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583. · arXiv
  130. Man Luo, Christopher J Warren, Lu Cheng, Haidar M Abdul-Muhsin, and Imon Banerjee. 2024a. Assess- ing empathy in large language models with real- world physician-patient interactions. arXiv preprint arXiv:2405.16402. · arXiv
  131. Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, et al. 2024b. Mmevol: Empowering multimodal large language models with evol-instruct. arXiv preprint arXiv:2409.05840. · arXiv
  132. Alisia Lupidi, Carlos Gemmell, Nicola Cancedda, Jane Dwivedi-Yu, Jason Weston, Jakob Foerster, Roberta Raileanu, and Maria Lomeli. 2024. Source2synth: Synthetic data generation and curation grounded in real data sources. arXiv preprint arXiv:2409.08239. · arXiv
  133. Chenkai Ma and Xinya Du. 2023. Poe: Process of elimi- nation for multiple choice reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4487-4496.
  134. Mingyu Derek Ma, Xiaoxuan Wang, Po-Nien Kung, P Jeffrey Brantingham, Nanyun Peng, and Wei Wang. 2024. Star: Boosting low-resource information ex- traction by structure-to-text data generation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18751-18759.
  135. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large lan- guage models. Advances in neural information pro- cessing systems, 36:21702-21720.
  136. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Pro- cessing Systems, 36.
  137. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint arXiv:2303.08896. · arXiv
  138. Margherita Martorana, Tobias Kuhn, Lise Stork, and Jacco van Ossenbruggen. 2024a. Text classifica- tion of column headers with a controlled vocabu- lary: leveraging llms for metadata enrichment. arXiv preprint arXiv:2403.00884. · arXiv
  139. Margherita Martorana, Tobias Kuhn, Lise Stork, and Jacco van Ossenbruggen. 2024b. Zero-shot topic classification of column headers: Leveraging llms for metadata enrichment. In Knowledge Graphs in the Age of Language Models and Neuro-Symbolic AI, pages 52-66. IOS Press.
  140. Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language mod- els: Towards zero-shot language understanding. Ad- vances in Neural Information Processing Systems, 35:462-477.
  141. Yu Meng, Martin Michalski, Jiaxin Huang, Yu Zhang, Tarek Abdelzaher, and Jiawei Han. 2023. Tun- ing language models as training data generators for augmentation-enhanced few-shot learning. In Inter- national Conference on Machine Learning, pages 24457-24477. PMLR.
  142. Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2021. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys.
  143. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. · arXiv
  144. Kun-Peng Ning, Shuo Yang, Yu-Yang Liu, Jia-Yu Yao, Zhen-Hui Liu, Yu Wang, Ming Pang, and Li Yuan. 2024. Peer-review-in-llms: Automatic evaluation method for llms in open-environment. arXiv preprint arXiv:2402.01830. · arXiv
  145. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730-27744.
  146. Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebas- tian Krause, and Aliaksei Severyn. 2024. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arXiv:2401.12086. · arXiv
  147. Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024. Automatically correcting large language models: Sur- veying the landscape of diverse automated correction strategies. Transactions of the Association for Com- putational Linguistics, 12:484-506.
  148. Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Yang Wang. 2023. On the risk of misinformation pollution with large language models. arXiv preprint arXiv:2305.13661. · arXiv
  149. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024a. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733. · arXiv
  150. Xianghe Pang, Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, and Siheng Chen. 2024b. Self-alignment of large language models via monopolylogue-based social scene simulation. arXiv preprint arXiv:2402.05699. · arXiv
  151. Jingyuan Qi, Zhiyang Xu, Ying Shen, Minqian Liu, Di Jin, Qifan Wang, and Lifu Huang. 2023. The art of socratic questioning: Recursive thinking with large language models. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 4177-4199.
  152. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. · arXiv
  153. Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou- Chakra, Ian Reid, and Niko Suenderhauf. 2023. Say- plan: Grounding large language models using 3d scene graphs for scalable robot task planning. In 7th Annual Conference on Robot Learning.
  154. Vikas Raunak, Amr Sharaf, Yiren Wang, Hany Awadalla, and Arul Menezes. 2023. Leveraging gpt- 4 for automatic translation post-editing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12009-12024.
  155. Francesco Ronzano and Jay Nanavati. 2024. Towards ontology-enhanced representation learning for large language models. arXiv preprint arXiv:2405.20527. · arXiv
  156. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950. · arXiv
  157. Maximilian Schmidt, Andrea Bartezzaghi, and Ngoc Thang Vu. 2024. Prompting-based synthetic data generation for few-shot question answering. In Proceedings of the 2024 Joint International Con- ference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13168-13178.
  158. Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, Jieming Zhu, and Xi Xiao. 2024. Pmg: Personalized multimodal generation with large language models. In Proceed- ings of the ACM on Web Conference 2024, pages 3833-3843.
  159. Luísa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. 2024. Llm see, llm do: Guiding data generation to tar- get non-differentiable objectives. arXiv preprint arXiv:2407.01490. · arXiv
  160. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Re- flexion: Language agents with verbal reinforcement learning. Advances in Neural Information Process- ing Systems, 36.
  161. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling reasoning capabilities into smaller language models. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, pages 7059-7073.
  162. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023. The curse of recursion: Training on generated data makes models forget. ArXiv, abs/2305.17493. · arXiv
  163. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023. Prog- prompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523-11530. IEEE.
  164. Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, and Yongbin Li. 2024a. Scal- ing data diversity for fine-tuning language models in human alignment. In Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation (LREC- COLING 2024), pages 14358-14369.
  165. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2023. Pref- erence ranking optimization for human alignment. arXiv preprint arXiv:2306.17492. · arXiv
  166. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2024b. Pref- erence ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990-18998.
  167. Renliang Sun, Mengyuan Liu, Shiping Yang, Rui Wang, Junqing He, and Jiaxing Zhang. 2024a. Fostering nat- ural conversation in large language models with nico: a natural interactive conversation dataset. ArXiv, abs/2408.09330. · arXiv
  168. Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023a. Is chatgpt good at search? investigating large lan- guage models as re-ranking agent. arXiv preprint arXiv:2304.09542. · arXiv
  169. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2024b. Principle-driven self- alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36.
  170. Zhen Tan, Lu Cheng, Song Wang, Yuan Bo, Jundong Li, and Huan Liu. 2023. Interpreting pretrained lan- guage models via concept bottlenecks. arXiv preprint arXiv:2311.05014. · arXiv
  171. Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. In Forty-first International Conference on Machine Learning.
  172. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. · arXiv
  173. Ramya Tekumalla and Juan M Banda. 2023. Lever- aging large language models and weak supervision for social media data annotation: an evaluation using covid-19 self-reported vaccination tweets. In Interna- tional Conference on Human-Computer Interaction, pages 356-366. Springer.
  174. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature Medicine, pages 1-11.
  175. Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. 2024a. Can llms learn from previous mistakes? investigating llms' errors to boost for reasoning. arXiv preprint arXiv:2403.20046. · arXiv
  176. Yongqi Tong, Sizhe Wang, Dawei Li, Yifan Wang, Simeng Han, Zi Lin, Chengsong Huang, Jiaxin Huang, and Jingbo Shang. 2024b. Optimizing lan- guage model's reasoning abilities with weak supervi- sion. arXiv preprint arXiv:2405.04086. · arXiv
  177. Yongqi Tong, Yifan Wang, Dawei Li, Sizhe Wang, Zi Lin, Simeng Han, and Jingbo Shang. 2023. Elimi- nating reasoning via inferring with planning: A new framework to guide llms' non-linear thinking. arXiv preprint arXiv:2310.12342. · arXiv
  178. Petter Törnberg. 2024. Best practices for text anno- tation with large language models. arXiv preprint arXiv:2402.05129. · arXiv
  179. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971. · arXiv
  180. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  181. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830. · arXiv
  182. Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2023. Revisiting relation extraction in the era of large language models. arXiv preprint arXiv:2305.05003. · arXiv
  183. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. · arXiv
  184. Danqing Wang and Lei Li. 2023. Learning from mis- takes via cooperative study assistant for large lan- guage models. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pages 10667-10685.
  185. Haoyu Wang, Guozheng Ma, Ziqiao Meng, Zeyu Qin, Li Shen, Zhong Zhang, Bingzhe Wu, Liu Liu, Yatao Bian, Tingyang Xu, et al. 2024a. Step-on-feet tun- ing: Scaling self-alignment of llms via bootstrapping. arXiv preprint arXiv:2402.07610. · arXiv
  186. Hongru Wang, Boyang Xue, Baohang Zhou, Tianhua Zhang, Cunxiang Wang, Guanhua Chen, Huimin Wang, and Kam-fai Wong. 2024b. Self-dc: When to retrieve and when to generate? self divide-and- conquer for compositional unknown questions. arXiv preprint arXiv:2402.13514. · arXiv
  187. Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shrid- har, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. 2023a. Gensim: Generating robotic simulation tasks via large language models. In The Twelfth International Conference on Learning Representations.
  188. PeiFeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. 2022a. Pinto: Faithful language rea- soning using prompt-generated rationales. In The Eleventh International Conference on Learning Rep- resentations.
  189. Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023b. Scott: Self-consistent chain-of-thought distillation. arXiv preprint arXiv:2305.01879. · arXiv
  190. Ruida Wang, Wangchunshu Zhou, and Mrinmaya Sachan. 2023c. Let's synthesize step by step: It- erative dataset synthesis with large language models by extrapolating errors from small models. In Find- ings of the Association for Computational Linguistics: EMNLP 2023, pages 11817-11831.
  191. Song Wang, Zhen Tan, Ruocheng Guo, and Jundong Li. 2023d. Noise-robust fine-tuning of pretrained lan- guage models via external guidance. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12528-12540.
  192. Song Wang, Peng Wang, Tong Zhou, Yushun Dong, Zhen Tan, and Jundong Li. 2024c. Ceb: Compo- sitional evaluation benchmark for fairness in large language models. arXiv preprint arXiv:2407.02408. · arXiv
  193. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  194. Yifan Wang, David Stevens, Pranay Shah, Wenwen Jiang, Miao Liu, Xu Chen, Robert Kuo, Na Li, Boy- ing Gong, Daniel Lee, et al. 2024d. Model-in-the- loop (milo): Accelerating multimodal ai data annota- tion with llms. arXiv preprint arXiv:2409.10702. · arXiv
  195. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023e. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484-13508.
  196. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xing- shan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023g. Aligning large lan- guage models with human: A survey. arXiv preprint arXiv:2307.12966. · arXiv
  197. Zifeng Wang, Chun-Liang Li, Vincent Perot, Long T Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, and Tomas Pfister. 2024e. Codeclm: Aligning language models with tailored synthetic data. arXiv preprint arXiv:2404.05875. · arXiv
  198. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Mar- tin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, et al. 2024f. Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv preprint arXiv:2401.04398. · arXiv
  199. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
  200. Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2024. Llmrec: Large language models with graph augmentation for recommendation. In Pro- ceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 806-815.

200 of 253 shown

Cited by

Sorted by citations

Related

Authors
Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhattacharjee, A., Karami, M., Li, J., Cheng, L., & Liu, H.
Affiliations
  • Arizona State University (US)
  • University of Virginia (US)
ORCID
Venue
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2024
Pages
930-957
DOI
10.18653/v1/2024.emnlp-main.54
OpenAlex
W4404783774
Cited by
178
References
0
Citations by year
2026: 76 · 2025: 90 · 2024: 12
FWCI
52.61
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Funders
  • National Science Foundation (2229461, 2312862)
  • U.S. Department of Homeland Security (17STQAC00001-08-00)
  • Cisco Systems
  • National Institutes of Health
  • Office of Naval Research (N00014-21-1-4002, N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
large language models, data annotation, synthetic data generation, data augmentation, text generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review