Reformulating Unsupervised Style Transfer as Paraphrase Generation
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Modern NLP defines the task of style transfer as modifying the style of a given sentence without appreciably changing its semantics, which implies that the outputs of style transfer systems should be paraphrases of their inputs.However, many existing systems purportedly designed for style transfer inherently warp the input's meaning through attribute transfer, which changes semantic properties such as sentiment.In this paper, we reformulate unsupervised style transfer as a paraphrase generation problem, and present a simple methodology based on fine-tuning pretrained language models on automatically generated paraphrase data.Despite its simplicity, our method significantly outperforms state-of-the-art style transfer systems on both human and automatic evaluations.We also survey 23 style transfer papers and discover that existing automatic metrics can be easily gamed and propose fixed variants.Finally, we pivot to a more real-world style transfer setting by collecting a large dataset of 15M sentences in 11 diverse styles, which we use for an in-depth analysis of our system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
(2011) · Cited by 14,317
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“The models are finetuned using a small learning rate of 5e-5 and converge to a good solu-tion fairly quickly as noticed by recent work (Li et al., 2020; Kaplan et al., 2020).”
MethodLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
A NEW MEASURE OF RANK CORRELATION
KENDALL, M. G. (1938) · Biometrika · doi:10.1093/biomet/30.1-2.81 · Cited by 7,269
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-4009 · Cited by 2,344
SWITCHBOARD: telephone speech corpus for research and development
Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225858 · Cited by 2,134 · cited 2×
“To create CDS, we obtain data (Table 5) from existing academic research datasets (Godfrey et al., 1992; Blodgett et al., 2016) and public APIs or online collections like Project Gutenberg (Hart, 1992).”
BackgroundMethodGenerating Sentences from a Continuous Space
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., & Bengio, S. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1002 · Cited by 2,061
Variation and the indexical field 1
Eckert, P. (2008) · Journal of Sociolinguistics · doi:10.1111/j.1467-9841.2008.00374.x · Cited by 2,046
OpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G., Kim, Y., Deng, Y., Senellart, J., & Rush, A. (2017) · Proceedings of ACL 2017, System Demonstrations · doi:10.18653/v1/p17-4012 · Cited by 1,749
Unsupervised Data Augmentation for Consistency Training
Xie, Q., Dai, Z., Hovy, E., Luong, M.-T., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1904.12848 · Cited by 1,517
Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Re-evaluating the Role of Bleu in Machine Translation Research
Callison-Burch, C., Osborne, M., & Koehn, P. (2006) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 632
Unsupervised Machine Translation Using Monolingual Corpora Only
Lample, G., Conneau, A., Denoyer, L., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1711.00043 · Cited by 584
Learning The Difference That Makes A Difference With Counterfactually-Augmented Data
Kaushik, D., Hovy, E. H., & Lipton, Z. C. (2020) · International Conference on Learning Representations · Cited by 511
Plug and Play Language Models: A Simple Approach to Controlled Text Generation
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., & Liu, R. (2019) · arXiv · doi:10.48550/arxiv.1912.02164 · Cited by 508
Delete, Retrieve, Generate: a Simple Approach to Sentiment and Style Transfer
Li, J., Jia, R., He, H., & Liang, P. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1169 · Cited by 468
Style Transfer in Text: Exploration and Evaluation
Fu, Z., Tan, X., Peng, N., Zhao, D., & Yan, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11330 · Cited by 460
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422
Fine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Curry, A. C., & Rieser, V. (2017) · arXiv · doi:10.48550/arxiv.1707.06875 · Cited by 406
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Wang, A., Cho, K., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2004.04228 · Cited by 356
Generating natural language under pragmatic constraints
Hovy, E. (1987) · Journal of Pragmatics · doi:10.1016/0378-2166(87)90109-3 · Cited by 337
Rao, S., & Tetreault, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1012 · Cited by 321
Wieting, J., & Gimpel, K. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1042 · Cited by 317
Self-citationDemographic dialectal variation in social media: a case study of African-American English
Blodgett, S. L., Green, L., & O'Connor, B. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1120 · Cited by 291 · cited 3×
“To create CDS, we obtain data (Table 5) from existing academic research datasets (Godfrey et al., 1992; Blodgett et al., 2016) and public APIs or online collections like Project Gutenberg (Hart, 1992).”
BackgroundMethodDavies, M. (2012) · Corpora · doi:10.3366/cor.2012.0024 · Cited by 235
Diverse Beam Search for Improved Description of Complex Scenes
Vijayakumar, A., Cogswell, M., Selvaraju, R., Sun, Q., Lee, S., Crandall, D., & Batra, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12340 · Cited by 219
A Simple, Fast Diverse Decoding Algorithm for Neural Generation
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08562 · Cited by 195
A Probabilistic Formulation of Unsupervised Text Style TransferPreprint
He, J., Wang, X., Neubig, G., & Berg-Kirkpatrick, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03912 · Cited by 143
“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer
Sudhakar, A., Upadhyay, B., & Maheswaran, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1322 · Cited by 142
Controllable Paraphrase Generation with a Syntactic Exemplar
Chen, M., Tang, Q., Wiseman, S., & Gimpel, K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1599 · Cited by 140
Style Transfer as Unsupervised Machine TranslationPreprint
Zhang, Z., Ren, S., Liu, S., Wang, J., Chen, P., Li, M., Zhou, M. Q., & Chen, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07894 · Cited by 125
BLEU might be Guilty but References are not Innocent
Freitag, M., Grangier, D., & Caswell, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.5 · Cited by 106
Neural Syntactic Preordering for Controlled Paraphrase Generation
Goyal, T., & Durrett, G. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.22 · Cited by 95
A4NT: Author Attribute Anonymity by Adversarial Training of Neural Machine Translation
Shetty, R., Schiele, B., & Fritz, M. (2018) · Max Planck Digital Library · doi:10.60882/cispa.24612720 · Cited by 61
A computational theory of goal-directed style in syntax
DiMarco, C., & Hirst, G. (1993) · OLAC - Open Language Archives Community · doi:10.5555/972487.972489 · Cited by 56
Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites
Tikhonov, A., Shibaev, V., Nagaev, A., Nugmanova, A., & Yamshchikov, I. P. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1406 · Cited by 47
Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning ApproachPreprint
Xu, J., Sun, X., Zeng, Q., Ren, X., Zhang, X., Wang, H., & Li, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.05181 · Cited by 47
Discovering User Attribute Stylistic Differences via Paraphrasing
Preoțiuc-Pietro, D., Xu, W. H., & Ungar, L. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10393 · Cited by 41
Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel Textual Transfer
Pang, R. Y., & Gimpel, K. (2019) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/d19-5614 · Cited by 40
Stylistic Transfer in Natural Language Generation Systems Using Recurrent Neural Networks
Kabbara, J., & Cheung, J. C. K. (2016) · doi:10.18653/v1/w16-6010 · Cited by 33
Adapting Language Models for Non-Parallel Author-Stylized Rewriting
Syed, B., Verma, G., Srinivasan, B. V., Natarajan, A., & Varma, V. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6433 · Cited by 8
Effective writing style imitation via combinatorial paraphrasingPreprint
Gröndahl, T., & Asokan, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13464 · Cited by 7
Cited by
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
Krishna, K., Song, Y., Karpinska, M., Wieting, J., & Iyyer, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1195 · Cited by 150
Self-citationDecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.11698 · Cited by 110
DecodingTrust: A comprehensive assessment of trustworthiness in GPT models
Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1361 · Cited by 107
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56
Self-citationarXivEdits: Understanding the Human Revision Process in Scientific Writing
Jiang, C., Xu, W., & Stevens, S. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.641 · Cited by 18
Metadata
- Authors
- Krishna, K., Wieting, J., & Iyyer, M.
- Venue
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Published
- 2020
- Pages
- 737-762
- OpenAlex
- W3092957620
- Cited by
- 185
- References
- 60
- FWCI
- 13.71
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.