CoChat
Sign Up
PreprintCited by 0Topic ModelingAdversarial Robustness in Machine LearningNatural Language Processing Techniques

N-Critics: Self-Refinement of Large Language Models with Ensemble of Critics

Open access
Mousavi, Sajad, Gutiérrez, Ricardo Luna, Rengarajan, Desik, Gundecha, Vineet, Babu, Ashwin Ramesh, Naug, Avisek, Guillen, Antonio, Sarkar, Soumyendu·2023·arXiv
DOI 10.48550/arxiv.2310.18679arXiv 2310.18679
Open this paper in CoChat — free accountOpen PDF

Abstract

We propose a self-correction mechanism for Large Language Models (LLMs) to mitigate issues such as toxicity and fact hallucination. This method involves refining model outputs through an ensemble of critics and the model's own feedback. Drawing inspiration from human behavior, we explore whether LLMs can emulate the self-correction process observed in humans who often engage in self-reflection and seek input from others to refine their understanding of complex topics. Our approach is model-agnostic and can be applied across various domains to enhance trustworthiness by addressing fairness, bias, and robustness concerns. We consistently observe performance improvements in LLMs for reducing toxicity and correcting factual errors.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.18679

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Self-reflection and iterative refinement methods align with short-term, inside-trial memory.

    “Self-reflection and iterative refinement methods (Madaan et al., 2023; Mousavi et al., 2023; Han et al., 2025c) naturally align with short-term, inside-trial memory, as the agent repeatedly leverages intermediate reasoning traces or outcomes from prior attempts within the same task.”

    From Memory in the Age of AI Agents (2025) · cites this paper as Mousavi et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Research works printed but not linked: 52

  1. OpenAI, Gpt-4 technical report, 2023. arXiv:2303.08774. · arXiv
  2. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, N. Fiedel, Palm: Scaling language modeling with pathways, 2022. arXiv:2204.02311. · arXiv
  3. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, J.-R. Wen, A survey of large language models, 2023. arXiv:2303.18223. · arXiv
  4. A. Deshpande, V. Murahari, T. Rajpurohit, A. Kalyan, K. Narasimhan, Toxicity in chatgpt: Analyzing persona-assigned language models, 2023. arXiv:2304.05335. · arXiv
  5. S. Lin, J. Hilton, O. Evans, Truthfulqa: Measuring how models mimic human falsehoods, 2022. arXiv:2109.07958. · arXiv
  6. J. Maynez, S. Narayan, B. Bohnet, R. McDonald, On faithfulness and factuality in abstractive sum- marization, in: Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, Association for Computational Linguistics, Online, 2020, pp. 1906-1919. URL: https: //aclanthology.org/2020.acl-main.173. doi:10.18653/v1/2020.acl-main.173. · DOI · Paper page
  7. L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, W. Y. Wang, Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies, 2023. URL: http://arxiv.org/abs/2308.03188, arXiv:2308.03188 [cs. · arXiv
  8. W. Xu, X. Qian, M. Wang, L. Li, W. Y. Wang, SESCORE2: Learning text generation evaluation via synthesizing realistic mistakes, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 5166-5183. URL: https://aclanthology.org/2023.acl-long. 283. doi:10.18653/v1/2023.acl-long.283. · DOI
  9. Y. Liu, P. Liu, SimCLS: A simple framework for contrastive learning of abstractive summarization, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Association for Computational Linguistics, Online, 2021, pp. 1065-1072. URL: https: //aclanthology.org/2021.acl-short.135. doi:10.18653/v1/2021.acl-short.135. · DOI
  10. S. Li, D. Lei, P. Qin, W. Y. Wang, Deep reinforcement learning with distributional semantic rewards for abstractive summarization, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 6038-6044. URL: https://aclanthology.org/D19-1623. doi:10.18653/ v1/D19-1623. · DOI
  11. I. Jauregi Unanue, J. Parnell, M. Piccardi, BERTTune: Fine-tuning neural machine translation with BERTScore, in: Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing (Volume 2: Short Papers), Association for Computational Linguistics, Online, 2021, pp. 915-924. URL: https://aclanthology.org/2021.acl-short.115. doi:10.18653/v1/ 2021.acl-short.115. · DOI
  12. E. Zelikman, Y. Wu, J. Mu, N. Goodman, Star: Bootstrapping reasoning with reason- ing, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Ad- vances in Neural Information Processing Systems, volume 35, Curran Associates, Inc., 2022, pp. 15476-15488. URL: https://proceedings.neurips.cc/paper_files/paper/2022/ file/639a9a172c044fbb64175b5fad42e9a5-Paper-Conference.pdf.
  13. J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, J. Han, Large language models can self-improve, 2022. arXiv:2210.11610. · arXiv
  14. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, P. Clark, Self- Refine: Iterative Refinement with Self-Feedback, 2023. URL: http://arxiv.org/abs/2303. 17651, arXiv:2303.17651 [cs. · arXiv
  15. N. Shinn, F. Cassano, B. Labash, A. Gopinath, K. Narasimhan, S. Yao, Reflexion: Language agents with verbal reinforcement learning, 2023. arXiv:2303.11366. · arXiv
  16. K. Zhang, Z. Li, J. Li, G. Li, Z. Jin, Self-edit: Fault-aware code editor for code generation, 2023. arXiv:2305.04087. · arXiv
  17. L. Pan, A. Albalak, X. Wang, W. Y. Wang, Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning, 2023. arXiv:2305.12295. · arXiv
  18. W. Du, Z. M. Kim, V. Raheja, D. Kumar, D. Kang, Read, revise, repeat: A system demonstration for human-in-the-loop iterative text revision, in: Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 96-108. URL: https://aclanthology.org/2022.in2writing-1. 14. doi:10.18653/v1/2022.in2writing-1. · DOI
  19. W. Yu, Z. Zhang, Z. Liang, M. Jiang, A. Sabharwal, Improving language models via plug-and-play retrieval feedback, 2023. arXiv:2305.14002. · DOI
  20. Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan, W. Chen, CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, 2023. URL: http://arxiv.org/abs/2305. 11738, arXiv:2305.11738 [cs. · arXiv
  21. D. Paul, M. Ismayilzada, M. Peyrard, B. Borges, A. Bosselut, R. West, B. Faltings, Refiner: Reasoning feedback on intermediate representations, 2023. arXiv:2304.01904. · arXiv
  22. H. Le, Y. Wang, A. D. Gotmare, S. Savarese, S. C. H. Hoi, Coderl: Master- ing code generation through pretrained models and deep reinforcement learning, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Advances in Neural Information Processing Systems, volume 35, Curran Associates, Inc., 2022, pp. 21314-21328. URL: https://proceedings.neurips.cc/paper_files/paper/2022/ file/8636419dea1aa9fbd25fc4248e702da4-Paper-Conference.pdf.
  23. A. F. Akyurek, E. Akyurek, A. Kalyan, P. Clark, D. T. Wijaya, N. Tandon, RL4F: Generating natural language feedback with reinforcement learning for repairing model outputs, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 7716-7733. URL: https: //aclanthology.org/2023.acl-long.427. doi:10.18653/v1/2023.acl-long.427. · DOI
  24. S. Gehman, S. Gururangan, M. Sap, Y. Choi, N. A. Smith, Realtoxicityprompts: Evaluating neural toxic degeneration in language models, arXiv preprint arXiv:2009.11462 (2020). · arXiv
  25. S. Min, J. Michael, H. Hajishirzi, L. Zettlemoyer, AmbigQA: Answering ambiguous open-domain questions, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), Association for Computational Linguistics, Online, 2020, pp. 5783- 5797. URL: https://aclanthology.org/2020.emnlp-main.466. doi:10.18653/v1/2020. emnlp-main.466. · DOI
  26. M. Joshi, E. Choi, D. S. Weld, L. Zettlemoyer, Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Vancouver, Canada, 2017.
  27. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, C. D. Manning, Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. arXiv:1809.09600. · arXiv
  28. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, Advances in Neural Information Processing Systems 35 (2022) 27730-27744.
  29. W. Du, Z. M. Kim, V. Raheja, D. Kumar, D. Kang, Read, revise, repeat: A system demonstration for human-in-the-loop iterative text revision, arXiv preprint arXiv:2204.03685 (2022). · arXiv
  30. S. Sarkar, A. R. Babu, V. Gundecha, A. Guillen, S. Mousavi, R. Luna, S. Ghorbanpour, A. Naug, Rl- cam: Visual explanations for convolutional networks using reinforcement learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023a, pp. 3860-3868.
  31. S. Sarkar, A. R. Babu, S. Mousavi, V. Gundecha, S. Ghorbanpour, A. Shmakov, R. L. Gutierrez, A. Guillen, A. Naug, Robustness with black-box adversarial attack using reinforcement learning, in: AAAI 2023: Proceedings of the Workshop on Artificial Intelligence Safety 2023 (SafeAI 2023), volume 3381, https://ceur-ws.org/Vol-3381/8.pdf, 2023b.
  32. S. Sarkar, A. R. Babu, V. Gundecha, A. Guillen, S. Mousavi, R. Luna, S. Ghorbanpour, A. Naug, Robustness with query-efficient adversarial attack using reinforcement learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023c, pp. 2329-2336.
  33. S. Sarkar, S. Mousavi, A. R. Babu, V. Gundecha, S. Ghorbanpour, A. K. Shmakov, Measuring robustness with black-box adversarial attack using reinforcement learning, in: NeurIPS ML Safety Workshop, 2022. URL: https://openreview.net/forum?id=Lj8fj0ECPv.
  34. S. Sarkar, V. Gundecha, A. Shmakov, S. Ghorbanpour, A. R. Babu, P. Faraboschi, M. Cocho, A. Pichard, J. Fievez, Multi-objective reinforcement learning controller for multi-generator industrial wave energy converter, in: NeurIPs Tackling Climate Change with Machine Learning Workshop, 2021.
  35. S. Sarkar, V. Gundecha, S. Ghorbanpour, A. Shmakov, A. R. Babu, A. Pichard, M. Cocho, Skip training for multi-agent reinforcement learning controller for industrial wave energy converters, in: 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE), IEEE, 2022, pp. 212-219.
  36. S. Sarkar, A. Naug, A. Guillen, R. L. Gutierrez, S. Ghorbanpour, S. Mousavi, A. R. Babu, V. Gundecha, Concurrent carbon footprint reduction (c2fr) reinforcement learning approach for sustainable data center digital twin, in: 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), 2023, pp. 1-8. doi:10.1109/CASE56687.2023.10260633. · DOI
  37. A. Shmakov, A. Naug, V. Gundecha, S. Ghorbanpour, R. L. Gutierrez, A. R. Babu, A. Guillen, S. Sarkar, Rtdk-bo: High dimensional bayesian optimization with reinforced transformer deep kernels, in: 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), IEEE, 2023, pp. 1-8.
  38. S. Sarkar, A. R. Babu, S. Mousavi, S. Ghorbanpour, V. Gundecha, R. L. Gutierrez, A. Guillen, A. Naug, Reinforcement learning based black-box adversarial attack for robustness improvement, in: 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), IEEE, 2023, pp. 1-8.
  39. S. Sarkar, V. Gundecha, A. Shmakov, S. Ghorbanpour, A. R. Babu, P. Faraboschi, M. Cocho, A. Pichard, J. Fievez, Multi-agent reinforcement learning controller to maximize energy efficiency for multi-generator industrial wave energy converter, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2022, pp. 12135-12144.
  40. S. Sarkar, A. R. Babu, V. Gundecha, A. Guillen, S. Mousavi, R. Luna, S. Ghorbanpour, A. Naug, Robustness with query-efficient adversarial attack using reinforcement learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023a, pp. 2329-2336.
  41. S. Sarkar, A. R. Babu, V. Gundecha, A. Guillen, S. Mousavi, R. Luna, S. Ghorbanpour, A. Naug, Rl- cam: Visual explanations for convolutional networks using reinforcement learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023b, pp. 3860-3868.
  42. S. Sarkar, V. Gundecha, S. Ghorbanpour, A. Shmakov, A. Ramesh Babu, A. Naug, A. Pichard, M. Cocho, Function approximation for reinforcement learning controller for energy from spread waves, in: E. Elkind (Ed.), Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, International Joint Conferences on Artificial Intelligence Organization, 2023c, pp. 6201-6209. URL: https://doi.org/10.24963/ijcai.2023/688. doi:10.24963/ijcai.2023/688, aI for Good. · DOI
  43. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large language models are zero-shot reasoners, Advances in neural information processing systems 35 (2022) 22199-22213.
  44. C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, D. Jiang, Wizardlm: Empowering large language models to follow complex instructions, arXiv preprint arXiv:2304.12244 (2023). · arXiv
  45. S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosinski, R. Liu, Plug and play language models: A simple approach to controlled text generation, CoRR abs/1912.02164 (2019). URL: http://arxiv.org/abs/1912.02164. arXiv:1912.02164. · arXiv
  46. B. Krause, A. D. Gotmare, B. McCann, N. S. Keskar, S. Joty, R. Socher, N. F. Rajani, GeDi: Genera- tive discriminator guided sequence generation, in: Findings of the Association for Computational Linguistics: EMNLP 2021, Association for Computational Linguistics, Punta Cana, Dominican Republic, 2021, pp. 4929-4952. URL: https://aclanthology.org/2021.findings-emnlp. 424. doi:10.18653/v1/2021.findings-emnlp.424. · DOI · Paper page
  47. A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, Y. Choi, DExperts: Decoding- time controlled text generation with experts and anti-experts, in: Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Compu- tational Linguistics, Online, 2021, pp. 6691-6706. URL: https://aclanthology.org/2021. acl-long.522. doi:10.18653/v1/2021.acl-long.522. · DOI
  48. S. Gururangan, A. Marasovic´, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, N. A. Smith, Don't stop pretraining: Adapt language models to domains and tasks, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguis- tics, Online, 2020, pp. 8342-8360. URL: https://aclanthology.org/2020.acl-main.740. doi:10.18653/v1/2020.acl-main.740. · DOI · Paper page
  49. X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, Y. Choi, Quark: Controllable text generation with reinforced unlearning, 2022. arXiv:2205.13636. · arXiv
  50. S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, Y. Choi, Generating sequences by learning to self-correct, 2022. arXiv:2211.00053. · arXiv
  51. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. H. Chi, Q. Le, D. Zhou, Chain of thought prompting elicits reasoning in large language models, CoRR abs/2201.11903 (2022). URL: https://arxiv.org/abs/2201.11903. arXiv:2201.11903. · arXiv
  52. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, React: Synergizing reasoning and acting in language models, 2023. arXiv:2210.03629. · arXiv

Authors
Mousavi, S., Gutiérrez, R. L., Rengarajan, D., Gundecha, V., Babu, A. R., Naug, A., Guillen, A., & Sarkar, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.18679
arXiv
2310.18679
OpenAlex
W4388092811
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, self-correction, self-refinement, toxicity reduction, self-reflection, factual error correction, model credibility, bias and fairness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review