Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Open accessAbstract
Pretrained language models have demonstrated extraordinary capabilities in language generation.However, real-world tasks often require controlling the distribution of generated text in order to mitigate bias, promote fairness, and achieve personalization.Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions.However, many important distributions, such as personal preferences, are unquantified.In this work, we tackle the problem of generating text following arbitrary distributions (quantified and unquantified) by proposing NANO, a fewshot human-in-the-loop training algorithm that continuously learns from human feedback.NANO achieves state-of-the-art results on single topic/attribute as well as quantified distribution control compared to previous works.We also show that NANO is able to learn unquantified distributions, achieves personalization, and captures differences between different individuals' personal preferences with high sample efficiency.Our code is available at https: //github.com/sfanxiang/Nano.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1× · see citing works
“However, to apply them to real-world tasks, it has become increasingly desirable to reduce social bias exhibited in large language models (Bender et al., 2021), improve fairness (Baldini et al., 2022), and fit to diverse individual preferences (Xue et al., 2009).”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search
Coulom, R. (2006) · Computers and Games · doi:10.1007/978-3-540-75538-8_7 · Cited by 1,819 · see citing works
Deep Reinforcement Learning for Dialogue Generation
Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., & Gao, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1127 · Cited by 1,221 · see citing works
A survey of human-in-the-loop for machine learning
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.05.014 · Cited by 800 · see citing works
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · see citing works
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · see citing works
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550 · see citing works
Plug and Play Language Models: A Simple Approach to Controlled Text Generation
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., & Liu, R. (2019) · arXiv · doi:10.48550/arxiv.1912.02164 · Cited by 508 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
Delete, Retrieve, Generate: a Simple Approach to Sentiment and Style Transfer
Li, J., Jia, R., He, H., & Liang, P. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1169 · Cited by 476 · see citing works
Self-citationThe Woman Worked as a Babysitter: On Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1339 · Cited by 431 · cited 1× · see citing works
“Unconditional language models have been shown to perpetuate undesirable stereotypes during generation which disproportionately harm underrepresented social groups (Liang et al., 2020; Ravfogel et al., 2020; Sheng et al., 2020, 2019).”
BackgroundFine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413 · see citing works
Style transfer through back-translation
Prabhumoye, S., Tsvetkov, Y., Salakhutdinov, R., & Black, A. W. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1080 · Cited by 348 · see citing works
Self-citationGenerating Sentences by Editing Prototypes
Guu, K., Hashimoto, T. B., Oren, Y., & Liang, P. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00030 · Cited by 300 · cited 1× · see citing works
“…1987; Xu et al., 2012), neural generative models (Prabhumoye et al., 2018; Lample et al., 2019; He et al., 2020), Retrieve-and-Edit approaches (Li et al., 2018; Hashimoto et al., 2018; Guu et al., 2018; Sudhakar et al., 2019; Madaan et al., 2020), and Transformer-based approach (Lyu et al., 2021)).”
MethodSelf-citationNull It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.647 · Cited by 287 · see citing works
Neural Text Generation with Unlikelihood Training
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., & Weston, J. (2019) · arXiv · doi:10.48550/arxiv.1908.04319 · Cited by 254 · see citing works
Identifying and Reducing Gender Bias in Word-Level Language Models
Bordia, S., & Bowman, S. R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop · doi:10.18653/v1/n19-3002 · Cited by 219 · see citing works
Paraphrase Generation with Deep Reinforcement Learning
Li, Z., Jiang, X., Shang, L., & Li, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1421 · Cited by 211 · see citing works
Multiple-Attribute Text Rewriting.
Lample, G., Subramanian, S., Smith, E. M., Denoyer, L., Ranzato, M., & Boureau, Y.-L. (2018) · International Conference on Learning Representations · Cited by 181 · see citing works
Towards Debiasing Sentence Representations
Liang, P. P., Li, I., Zheng, E., Lim, Y. C., Salakhutdinov, R., & Morency, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.488 · Cited by 163 · see citing works
Self-citationTowards Understanding and Mitigating Social Biases in Language Models
Liang, P. P., Wu, C., Morency, L.-P., & Salakhutdinov, R. (2021) · arXiv · doi:10.48550/arxiv.2106.13219 · Cited by 153 · see citing works
Self-citationPoliteness Transfer: A Tag and Generate Approach
Madaan, A., Setlur, A., Parekh, T., Póczos, B., Neubig, G., Yang, Y., Salakhutdinov, R., Black, A. W., & Prabhumoye, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.169 · Cited by 143 · see citing works
Self-citation“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer
Sudhakar, A., Upadhyay, B., & Maheswaran, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1322 · Cited by 142 · see citing works
Deep reinforcement and transfer learning for abstractive text summarization: A review
Alomari, A., Idris, N. B., Sabri, A. Q. M., & Alsmadi, I. M. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101276 · Cited by 141 · see citing works
Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering
Wallace, E., Rodríguez, P., Feng, S., Yamada, I., & Boyd-Graber, J. L. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00279 · Cited by 130 · see citing works
A Retrieve-and-Edit Framework for Predicting Structured Outputs
Hashimoto, T. B., Guu, K., Oren, Y., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01194 · Cited by 126 · see citing works
Self-citationZhao, T., Xie, K., & Eskénazi, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1123 · Cited by 114 · see citing works
Generating natural language under pragmatic constraints
Hovy, E. H. (1987) · Journal of Pragmatics · doi:10.1016/0378-2166(87)90099-3 · Cited by 110 · see citing works
A Probabilistic Formulation of Unsupervised Text Style TransferPreprint
He, J., Wang, X., Neubig, G., & Berg-Kirkpatrick, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03912 · Cited by 99 · see citing works
Towards Controllable Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.291 · Cited by 95 · see citing works
CoCon: A Self-Supervised Approach for Controlled Text GenerationPreprint
Chan, A., Ong, Y.-S., Pung, B. T. W., Zhang, A., & Fu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03535 · Cited by 66 · see citing works
Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study
Yao, Z., Su, Y., Sun, H., & Yih, W.-T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1547 · Cited by 61 · see citing works
QUARK: Controllable Text Generation with Reinforced Unlearning
Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., & Choi, Y. (2022) · neural information processing systems · doi:10.52202/068431-2001 · Cited by 51 · see citing works
User language model for collaborative personalized search
Xue, G.-R., Han, J., Yu, Y. H., & Yang, Q. (2009) · ACM Transactions on Information Systems · doi:10.1145/1462198.1462203 · Cited by 44 · see citing works
Multitask Learning and Reinforcement Learning for Personalized Dialog Generation: An Empirical Study
Yang, M., Huang, W., Tu, W., Qu, Q., Shen, Y., & Lei, K. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2975035 · Cited by 43 · see citing works
MARTA: Leveraging Human Rationales for Explainable Text Classification
Arous, I., Dolamic, L., Yang, J., Bhardwaj, A., Cuccu, G., & Cudré-Mauroux, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16734 · Cited by 43 · see citing works
Your fairness may vary: Pretrained language model fairness in toxic text classification
Baldini, I., Wei, D., Ramamurthy, K. N., Singh, M., & Yurochkin, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.176 · Cited by 39 · see citing works
A Distributional Approach to Controlled Text Generation
Khalifa, M., Elsahar, H., & Dymetman, M. (2020) · arXiv · doi:10.48550/arxiv.2012.11635 · Cited by 39 · see citing works
Personalized Response Generation via Generative Split Memory Network
Wu, Y., Ma, X., & Yang, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.157 · Cited by 37 · see citing works
Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning
Yao, Z., Li, X., Gao, J., Sadler, B. M., & Sun, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012547 · Cited by 33 · see citing works
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer
Lyu, Y., Liang, P. P., Pham, H., Hovy, E. H., Póczos, B., Salakhutdinov, R., & Morency, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.171 · Cited by 29 · see citing works
Self-citationA reinforcement learning formulation to the complex question answering problem
Chali, Y., Hasan, S. A., & Mojahid, M. (2015) · Information Processing & Management · doi:10.1016/j.ipm.2015.01.002 · Cited by 28 · see citing works
Towards Controllable and Personalized Review Generation
Li, P., & Tuzhilin, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1319 · Cited by 26 · see citing works
Journalist-in-the-Loop: Continuous Learning as a Service for Rumour Analysis
Karmakharm, T., Αλέτρας, Ν., & Bontcheva, K. (2019) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d19-3020 · Cited by 25 · see citing works
Godin, F., Kumar, A., & Mittal, A. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-2016 · Cited by 22 · see citing works
Automatic Conditional Generation of Personalized Social Media Short Texts
Wang, Z., Wang, J., Gu, H., Su, F., & Zhuang, B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-97310-4_7 · Cited by 6 · see citing works
When and Why does a Model Fail? A Human-in-the-loop Error Detection Framework for Sentiment AnalysisPreprint
Liu, Z., Guo, Y., & Mahmud, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00954 · Cited by 5 · see citing works
Research on user granularity-level personalized social text generation technology
Gao, Y. B., Gao, J., Ma, R., & Yang, L. D. (2022) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2294/1/012015 · Cited by 3 · see citing works
GENIE: Toward Reproducible and Standardized Human Evaluation for Text GenerationPreprint
Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06561 · Cited by 2 · see citing works
Reinforced Generative Adversarial Network for Abstractive Text SummarizationPreprint
Xu, T., & Zhang, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.15176 · Cited by 2 · see citing works
Yuan, C., & Huang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09442 · Cited by 2 · see citing works
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Flagged first, then by citations
Research works printed but not linked: 4
- Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazaré, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In ACL.
- Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y- Lan Boureau. 2019. Multiple-attribute text rewrit- ing. In International Conference on Learning Rep- resentations.
- Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. 2020. Towards debiasing sen- tence representations. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 5502-5515.
- Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In Pro- ceedings of COLING 2012, pages 2899-2914.
Cited by
Sorted by citations
Applications, Challenges, and Future Directions of Human-in-the-Loop Learning
Kumar, S., Datta, S., Singh, V., Datta, D., Singh, S. K., & Sharma, R. (2024) · IEEE Access · doi:10.1109/access.2024.3401547 · Cited by 194 · see citing works
Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots
Puig, X., Undersander, E., Szot, A., Cote, M. D., Yang, T.-Y., Partsey, R., Desai, R., Clegg, A. W., Hlavac, M., Min, S. Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.13724 · Cited by 15 · see citing works
A Survey on the Recent Advancements in Human-Centered Dialog Systems
Oruche, R., Goruganthu, S. K., Akula, R., Cheng, X., Goni, M. A., Shibo, B. W., Kee, K. F., Zampieri, M., & Calyam, P. P. (2025) · ACM Computing Surveys · doi:10.1145/3729220 · Cited by 9 · see citing works
Reward Gaming in Conditional Text Generation
Pang, R. Y., Padmakumar, V., Sellam, T., Parikh, A. P., & He, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.262 · Cited by 4 · see citing works
- Authors
- Fan, X., Lyu, Y., Liang, P. P., Salakhutdinov, R., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- University of Michigan (US)
- Venue
- Findings of the Association for Computational Linguistics: ACL 2023
- Published
- 2023
- Pages
- 11970-11992
- OpenAlex
- W4385570669
- Cited by
- 3
- References
- 56
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 1
- FWCI
- 0.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1750439, 1722822)
- Carnegie Mellon University
- National Institutes of Health
- National Institute of Mental Health (R01MH096951, U01MH116925, R01MH125740)
- Center for Machine Learning and Health, School of Computer Science, Carnegie Mellon University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- human-in-the-loop learning, reward learning, distribution control, personalization, individual preferences, fairness, sample efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
