Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation
Open accessAbstract
Large Language Models (LLMs) have in recent years demonstrated impressive prowess in natural language generation.A common practice to improve generation diversity is to sample multiple outputs from the model.However, partly due to the inaccessibility of LLMs, there lacks a simple and robust way of selecting the best output from these stochastic samples.As a case study framed in the context of question generation, we propose two prompt-based approaches, namely round-trip and prompt-based score, to selecting high-quality questions from a set of LLM-generated candidates.Our method works without the need to modify the underlying model, nor does it rely on human-annotated references -both of which are realistic constraints for real-world deployment of LLMs.With automatic as well as human evaluations, we empirically demonstrate that our approach can effectively select questions of higher qualities than greedy generation. 1 * Equal contribution. 1 We open-source all code and annotated data on github.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pre-trained LLMs such as GPT-3 can be sensitive to the specific data points provided as few-shot examples.
“There are many recent works in the NLP community report that pre-trained LLMs such as GPT-3 can be sensitive to the specific data points provided as few-shot examples [30, 38, 63, 41].”
From GPT-3-Driven Pedagogical Agents to Train Children’s Curious Question-Asking Skills (2024) · cites this paper as [63]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · see citing works
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752 · see citing works
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.185 · Cited by 747 · see citing works
Learning to Ask: Neural Question Generation for Reading Comprehension
Du, X., Shao, J., & Cardie, C. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1123 · Cited by 696 · see citing works
Unsupervised Neural Machine Translation
Artetxe, M., Labaka, G., Agirre, E., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1710.11041 · Cited by 632 · see citing works
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610 · see citing works
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563 · see citing works
Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
Yin, W., Hay, J., & Roth, D. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1404 · Cited by 537 · see citing works
A Systematic Review of Automatic Question Generation for Educational Purposes
Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-019-00186-y · Cited by 514 · see citing works
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · see citing works
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374 · see citing works
BARTScore: Evaluating Generated Text as Text Generation
Yuan, W., Neubig, G., & Liu, P. (2021) · arXiv · doi:10.48550/arxiv.2106.11520 · Cited by 353 · see citing works
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-TrainingPreprint
Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Piao, S., Gao, J., Zhou, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12804 · Cited by 257 · see citing works
Self-citationSynthetic QA Corpora Generation with Roundtrip Consistency
Alberti, C., Andor, D., Pitler, E., Devlin, J., & Collins, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1620 · Cited by 208 · see citing works
Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Webson, A., & Pavlick, E. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.167 · Cited by 203 · see citing works
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177 · see citing works
Harvesting Paragraph-level Question-Answer Pairs from Wikipedia
Du, X., & Cardie, C. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1177 · Cited by 168 · see citing works
Cycle-Consistency for Robust Visual Question Answering
Shah, M., Chen, X., Rohrbach, M., & Parikh, D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00681 · Cited by 165 · see citing works
Machine Comprehension by Text-to-Text Neural Question Generation
Yuan, X., Wang, T., Gülçehre, Ç., Sordoni, A., Bachman, P., Zhang, S., Subramanian, S., & Trischler, A. (2017) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w17-2603 · Cited by 158 · see citing works
Self-citationShow Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · see citing works
Can language models learn from explanations in context?
Lampinen, A., Dasgupta, I., Chan, S., Mathewson, K., Tessler, M., Creswell, A., McClelland, J., Wang, J., & Hill, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.38 · Cited by 142 · see citing works
Xiao, D., Zhang, H., Li, Y., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/553 · Cited by 122 · see citing works
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
Zhang, S., & Bansal, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1253 · Cited by 115 · see citing works
Asking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus
Liu, B., Wei, H., Niu, D., Chen, H., & He, Y. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380270 · Cited by 89 · see citing works
Recent Advances in Neural Question GenerationPreprint
Pan, L., Lei, W., Chua, T., & Kan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08949 · Cited by 79 · see citing works
Memory-assisted prompt editing to improve GPT-3 after deployment
Madaan, A., Tandon, N., Clark, P., & Yang, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.183 · Cited by 60 · see citing works
Unsupervised FAQ Retrieval with Question Generation and BERT
Mass, Y., Carmeli, B., Roitman, H., & Konopnicki, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.74 · Cited by 43 · see citing works
Conversational agents for fostering curiosity-driven learning in children
Abdelghani, R., Oudeyer, P.-Y., Law, E., de Vulpillières, C., & Sauzéon, H. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102887 · Cited by 39 · see citing works
Self-citationAsking It All: Generating Contextualized Questions for any Semantic Role
Pyatkin, V., Roit, P., Michael, J., Goldberg, Y., Tsarfaty, R., & Dagan, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.108 · Cited by 26 · see citing works
Qi, P., Zhang, Y., & Manning, C. D. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.3 · Cited by 23 · see citing works
Xu, Y., Wang, D., Yu, M., Ritchie, D., Yao, B., Wu, T., Zhang, Z., Li, T. J.-J., Bradford, N., Sun, B., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.13947 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural ma- chine translation. In International Conference on Learning Representations.
- Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 610-623.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Cited by
Sorted by citations
Xiao, Z., Yuan, X., Liao, Q. V., Abdelghani, R., & Oudeyer, P. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581754.3584136 · Cited by 273 · see citing works
Self-citationGPT-3-Driven Pedagogical Agents to Train Children’s Curious Question-Asking Skills
Abdelghani, R., Wang, Y.-H., Yuan, X., Wang, T., Lucas, P., Sauzéon, H., & Oudeyer, P.-Y. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00340-7 · Cited by 129 · see citing works
Self-citationWhen Search Engine Services Meet Large Language Models: Visions and Challenges
Xiong, H., Bian, J., Li, Y., Li, X., Du, M., Wang, S., Yin, D., & Helal, S. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3451185 · Cited by 82 · see citing works
Large Language Models are Diverse Role-Players for Summarization Evaluation
Wu, N., Gong, M., Shou, L., Liang, S., & Jiang, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_54 · Cited by 66 · see citing works
Lee, J., Jung, W., & Baek, S. (2024) · Applied Sciences · doi:10.3390/app14052096 · Cited by 27 · see citing works
Kholodna, N., Julka, S., Khodadadi, M., Gumus, M. N., & Granitzer, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70381-2_25 · Cited by 32 · see citing works
Harnessing T5 Large Language Model for Enhanced PDF Text Comprehension and Q&A Generation
Chelliah, B. J., Hariharan, M., Prakash, A., Manoharan, B., & Senthilselvi, A. (2024) · doi:10.1109/iccds60734.2024.10560390 · Cited by 11 · see citing works
Gallmetzer, S., Sondoqah, M., Turri, E., Koester, L., Louwen, A., & Moser, D. (2025) · EPJ Photovoltaics · doi:10.1051/epjpv/2025010 · Cited by 7 · see citing works
Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank
Kumar, N. A., Fernandez, N., Wang, Z., & Lan, A. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.22 · Cited by 7 · see citing works
Alshamrani, S. (2024) · IEEE Access · doi:10.1109/access.2024.3434612 · Cited by 2 · see citing works
Manmathan, M. K. (2024) · Journal of Electrical Systems · doi:10.52783/jes.2030 · Cited by 2 · see citing works
Can Question Validation Criteria Improve the Quality of LLM-Generated Multiple-Choice Questions?
Morland, R., Jhajj, G., Yan, H., Lin, F. O., Mellick, J., Treu, R., Farrelly, G., Zariski, A., Al-Shamali, F. M., Wang, Z., et al. (2025) · IEEE Smart World Congress (SWC) · doi:10.1109/swc65939.2025.00073 · Cited by 1 · see citing works
Automatic Question Generation with Large Language Models: A Survey
de Oliveira, R. M., Hernández, A., Garbin, M. C., dos Reis, J. C., & Azevedo, R. (2026) · ACM Computing Surveys · doi:10.1145/3832773 · Cited by 1 · see citing works
Efficient Integration of ASR with Large Language Models to Enhance Video Search at Scale
Zhang, Q., Xiao, F., Li, T., Lin, L., Fang, H., Sun, H., Liu, R., Zhu, X., & Wang, J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715220 · Cited by 1 · see citing works
Ibarra, I. N. D. R. L. G. I., Sanpablo, A. I. P., Rodríguez, M. D., Californa, U. A. D. B., Arquer-Ruiz, M. D. C., Salle, U. L., Ramírez-Morales, A. O., Salle, U. L., Peñaloza, A. M., & Ibarra, I. N. D. R. L. G. I. (2023) · Revista Mexicana de Ingeniería Biomédica · doi:10.17488/rmib.44.4.11
RATHORE, A. S., & TĂNASE, S. (2024) · Romanian Cyber Security Journal · doi:10.54851/v6i1y202404
Sattabun, T., & Siriborvornratanakul, T. (2025) · Asia Pacific Information Technology Conference (APIT) · doi:10.1145/3726101.3726109
Teacher-Guided Code Generation With API Update
Sun, T., Wang, Z., Wang, W., Chen, R., & Pei, J. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11342663
Zhou, H., Li, L., Xu, X., & Yin, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5719-6_17
A Modular Multi-Agent Architecture for Automated Reading Comprehension Question Generation
Zhao, P., Blake, J., & Pyshkin, E. (2026) · International Conference on Information Technology and Education Technology (ITET) · doi:10.1109/itet70052.2026.11651012
The Evolution of Question Generation: Analyzing the Impact of Large Language Models
Bhakuni, P., Singh, G. B., Punia, A., Suyal, H., & Sharma, N. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70133
- Authors
- Yuan, X., Wang, T., Wang, Y.-H., Fine, E., Abdelghani, R., Sauzéon, H., & Oudeyer, P.-Y.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- Corresponding author
- Wang, Tong
- Venue
- Findings of the Association for Computational Linguistics: ACL 2023
- Published
- 2023
- Pages
- 12952-12965
- OpenAlex
- W4296965999
- Cited by
- 21
- References
- 42
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 7 · 2023: 5
- FWCI
- 3.72
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- output selection, large language models, question generation, generational diversity, automated assessment, human evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
