Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
We present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT).We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator, and the effect of the quality of reward estimates on the overall RL task.Our analysis of cardinal (5-point ratings) and ordinal (pairwise preferences) feedback shows that their intra-and inter-annotator αagreement is comparable.Best reliability is obtained for standardized cardinal feedback, and cardinal feedback is also easiest to learn and generalize from.Finally, improvements of over 1 BLEU can be obtained by integrating a regressionbased reward estimator trained on cardinal feedback for 800 translations into RL for NMT.This shows that RL is possible even from small amounts of fairly reliable human feedback, pointing to a great potential for applications at larger scale.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
(2014) · Cited by 33,419
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Subword embeddings are pre-trained on the WMT bitext with word2vec (Mikolov et al., 2013), normalized to unit length and held constant during further training.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“A typical approach is to combine REINFORCE (Williams, 1992) with policies based on deep sequence-to-sequence learning (Bahdanau et al., 2015), for example, in machine translation (Bahdanau et al.”
MethodInfluentialMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
(1999) · Cited by 5,147
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
Bradley, R. A., & Terry, M. E. (1952) · Biometrika · doi:10.2307/2334029 · Cited by 2,412 · cited 1×
“We use deep learning models to train reward estimators by regression against cardinal feedback, and by fitting a Bradley-Terry model (Bradley and Terry, 1952) to ordinal feedback.”
MethodSeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, J., & Yu, Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,285
RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS: THE METHOD OF PAIRED COMPARISONS
BRADLEY, R. A., & TERRY, M. E. (1952) · Biometrika · doi:10.1093/biomet/39.3-4.324 · Cited by 1,434
chrF: character n-gram F-score for automatic MT evaluation
Popović, M. (2015) · Proceedings of the Tenth Workshop on Statistical Machine Translation · doi:10.18653/v1/w15-3049 · Cited by 1,220 · cited 1×
“In order to compose a homogeneous data set, we first select translations with references of length 20 to 40, then sort the translation pairs by difference in character n-gram F-score (chrF, β = 3) (Popović, 2015) and length, and pick the top 400 translation pairs with the highest difference in chrF but lowest differen…”
MethodA Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182
An Introduction to Mathematical Statistics and its Applications.
Skarpness, B., Larsen, R. J., & Marx, M. L. (1983) · Journal of the American Statistical Association · doi:10.2307/2287143 · Cited by 1,149
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 4×
“We use an RL approach for tuning, where a crucial difference of our work to previous work on RL from human rewards (Knox and Stone, 2009; Christiano et al., 2017) is that our RL scenario is not interactive, but rewards are collected in an offline log.”
BackgroundMethodInfluentialKakade, S. M. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 775
BRADLEY, R. A. (1954) · Biometrika · doi:10.1093/biomet/41.3-4.502 · Cited by 716
Stanford neural machine translation systems for spoken language domains.
Luong, M.-T., & Manning, C. D. (2015) · Cited by 499
Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability
Clark, J. H., Dyer, C., Lavie, A., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473090 · Cited by 443
Eligibility Traces for Off-Policy Policy Evaluation
Precup, D., Sutton, R. S., & Singh, S. P. (2000) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 439
An Actor-Critic Algorithm for Sequence Prediction
Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1607.07086 · Cited by 423
Interactively shaping agents via human reinforcement: the TAMER framework
Knox, W. B., & Stone, P. (2009) · Proceedings of the fifth international conference on Knowledge capture · doi:10.1145/1597735.1597738 · Cited by 406
Computer Intensive Methods for Testing Hypotheses.
Hollas, E. A., & Noreen, E. W. (1990) · The Statistician · doi:10.2307/2348203 · Cited by 395
The self-normalized estimator for counterfactual learning
Swaminathan, A., & Joachims, T. (2015) · neural information processing systems · Cited by 383
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
Liang, C., Berant, J., Le, Q., Forbus, K. D., & Lao, N. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1003 · Cited by 381
Minimum Risk Training for Neural Machine Translation
Shen, S., Cheng, Y., He, Z., He, W., Wu, H., Sun, M., & Liu, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1159 · Cited by 375
Learnability, Stability and Uniform Convergence
Shalev‐Shwartz, S., Shamir, O., Srebro, N., & Sridharan, K. (2010) · doi:10.5555/1756006.1953019 · Cited by 328
Counterfactual Risk Minimization
Swaminathan, A., & Joachims, T. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2740908.2742564 · Cited by 265
Off-Policy Temporal Difference Learning with Function Approximation
Precup, D., Sutton, R. S., & Dasgupta, S. (2001) · Cited by 243
Interactive Learning from Policy-Dependent Human Feedback
MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D., Taylor, M. E., & Littman, M. L. (2017) · arXiv · doi:10.48550/arxiv.1701.06049 · Cited by 163
Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning
Pilarski, P. M., Dawson, M. R., Degris, T., Fahimi, F., Carey, J. P., & Sutton, R. S. (2011) · 2011 IEEE International Conference on Rehabilitation Robotics · doi:10.1109/icorr.2011.5975338 · Cited by 153
Fast Domain Adaptation for Neural Machine Translation
Freitag, M., & Al-Onaizan, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.06897 · Cited by 151
Kim, H., Lee, J.-H., & Na, S.-H. (2017) · Proceedings of the Second Conference on Machine Translation · doi:10.18653/v1/w17-4763 · Cited by 130
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Nguyen, K., Daumé III, H., & Boyd-Graber, J. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1153 · Cited by 110
Pushing the Limits of Translation Quality Estimation
Martins, A. F. T., Junczys-Dowmunt, M., Kepler, F., Astudillo, R. F., Hokamp, C., & Grundkiewicz, R. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00056 · Cited by 70
Human Effort and Machine Learnability in Computer Aided Translation
Green, S., Wang, S. I., Chuang, J., Heer, J., Schuster, S., & Manning, C. D. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1130 · Cited by 68 · cited 2×
“Lastly, we investigate machine learnability of the overall NMT task, in the sense of Green et al. (2014) who posed the question of how well an MT system can be tuned on post-edits.”
MethodReinforcement Learning Via Practice and Critique Advice
Judah, K., Roy, S., Fern, A., & Dietterich, T. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7690 · Cited by 67
Adversarial Neural Machine Translation
Wu, L., Xia, Y., Tian, F., Li, Z., Qin, T., Lai, J., & Liu, T. (2018) · Asian Conference on Machine Learning · Cited by 57
Fitting Sentence Level Translation Evaluation with Many Dense Features
Stanojević, M., & Sima'an, K. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1025 · Cited by 54
Neural Monkey: An Open-source Tool for Sequence Learning
Helcl, J., & Libovický, J. (2017) · The Prague Bulletin of Mathematical Linguistics · doi:10.1515/pralin-2017-0001 · Cited by 52
Can Neural Machine Translation be Improved with User Feedback?
Kreutzer, J., Khadivi, S., Matusov, E., & Riezler, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers) · doi:10.18653/v1/n18-3012 · Cited by 49 · cited 3×
“The first deployment of bandit NMT in an e-commerce translation scenario conjectured lacking reliability of user judgments as the reason for disappointing results when learning from 148k user-generated 5-star ratings for around 70k product title translations (Kreutzer et al., 2018).”
BackgroundInfluentialSelf-citationBandit Structured Prediction for Neural Sequence-to-Sequence Learning
Kreutzer, J., Sokolov, A., & Riezler, S. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1138 · Cited by 35
Self-citationLawrence, C., Sokolov, A., & Riezler, S. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1272 · Cited by 34
Self-citationLearning Structured Predictors from Bandit Feedback for Interactive NLP
Sokolov, A., Kreutzer, J., Lo, C., & Riezler, S. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1152 · Cited by 28
Self-citationStochastic Structured Prediction under Bandit FeedbackPreprint
Sokolov, A., Kreutzer, J., Lo, C., & Riezler, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00739 · Cited by 24
Self-citation
Research works printed but not linked: 10
- L´ eon Bottou, Jonas Peters, Joaquin Qui˜ noneroCandela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipanakar Ray, Patrice Simard, and Ed Snelson. 2013. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research 14:3207-3260.
- Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation (WMT) . Edinburgh, Scotland.
- Nan Jiang and Lihong Li. 2016. Doubly robust offpolicy value evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML) . New York, NY, USA.
- Vijay R. Konda and John N. Tsitsiklis. 2000. Actorcritic algorithms. In Advances in Neural Information Processing Systems (NIPS) . Vancouver, Canada.
- Klaus Krippendorff. 2013. Content Analysis. An Introduction to Its Methodology . Sage, third edition.
- Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing . Atlanta, GA, USA.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL) . Berlin, Germany.
- Philip S. Thomas and Emma Brunskill. 2016. Dataefficient off-policy policy evaluation for reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning (ICML) . New York, NY, USA.
- Louis Leon Thurstone. 1927. A law of comparative judgement. Psychological Review 34:278-286.
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR abs/1609.08144. · arXiv
Cited by
von Rueden, L., Mayer, S., Beckh, K., Georgiev, B., Giesselbach, S., Heese, R., Kirsch, B., Walczak, M., Pfrommer, J., Pick, A., et al. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3079836 · Cited by 793
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686
Fine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293
“Despite the success of instruction tuning, relative human judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of human preferences, improving proficiency in translation [16], summarization [35, 45], story-telling [45], and i…”
BackgroundDeep Reinforcement Learning for Sequence-to-Sequence Models
Keneshloo, Y., Shi, T., Ramakrishnan, N., & Reddy, C. K. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2929141 · Cited by 230
Metadata
- Authors
- Kreutzer, J., Uyheng, J., & Riezler, S.
- Affiliations
- Heidelberg University (DE)
- Ateneo de Manila University (PH)
- ORCID
- Uyheng, Joshua
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2018
- Pages
- 1777-1788
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2799082722
- Cited by
- 38
- References
- 49
- Citations by year
- 2025: 6 · 2024: 1 · 2023: 5 · 2022: 2 · 2021: 8 · 2020: 6 · 2019: 8 · 2018: 2
- FWCI
- 2.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Greater South Information System
- Funders
- Deutsche Forschungsgemeinschaft
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- feedback reliability, annotation agreement, pairwise preferences
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.