A Simple Recipe towards Reducing Hallucination in Neural Surface Realisation
Open accessAbstract
Recent neural language generation systems often hallucinate contents (i.e., producing irrelevant or contradicted facts), especially when trained on loosely corresponding pairs of the input structure and text.To mitigate this issue, we propose to integrate a language understanding module for data refinement with selftraining iterations to effectively induce strong equivalence between the input data and the paired text.Experiments on the E2E challenge dataset show that our proposed framework can reduce more than 50% relative unaligned noise from the original data-text pairs.A vanilla sequence-to-sequence neural NLG model trained on the refined data has improved on content correctness compared with the current state-of-the-art ensemble generator.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Using commonsense knowledge can help to gain better numeral representation.
“Currently, some works [193, 246, 315] point out that using commonsense knowledge can help to gain better numeral representation.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [193]
The Discussion says that lexical choices for quantities and numbers require additional, localised numerical commonsense. It also says the paper’s method sidesteps that challenge; it does not show that using commonsense knowledge improves numeral representations. (LLM, conf=0.83)
Experimental — accuracy not yet measuredCleaning data automatically is suitable for the case where there is a low or moderate level of noise in the original data.
“This approach is suitable for the case where there is a low or moderate level of noise in the original data [72, 193].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [193]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation
Juraska, J., Karagiannis, P., Bowden, K., & Walker, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1014 · Cited by 75 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- Shubham Agarwal, Marc Dymetman, and Eric Gaussier. 2018. Char2char generation with rerank- ing for the E2E NLG challenge. In Proceedings of the 11th International Conference on Natural Language Generation, pages 451-456, Tilburg Uni- versity, The Netherlands. Association for Computa- tional Linguistics.
- Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 643-653, Melbourne, Australia. Association for Computational Linguistics.
- Andrew Chisholm, Will Radford, and Ben Hachey. 2017. Learning to generate one-sentence biogra- phies from Wikidata. In Proceedings of the 15th Conference of the European Chapter of the Associa- tion for Computational Linguistics: Volume 1, Long Papers, pages 633-642, Valencia, Spain. Associa- tion for Computational Linguistics.
- Ondřej Dušek and Filip Jurcicek. 2016. Sequence-to- sequence generation for spoken dialogue via deep syntax trees and strings. In Proceedings of the 54th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers). Asso- ciation for Computational Linguistics.
- Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the E2E NLG challenge. In Proceedings of the 11th International Conference on Natural Language Generation, pages 322-328, Tilburg University, The Netherlands. Association for Computational Linguistics.
- Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1631-1640, Berlin, Germany. Association for Computational Linguistics.
- Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsu- ruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing, pages 1923-1933, Copenhagen, Denmark. As- sociation for Computational Linguistics. · DOI · Paper page
- Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? selective gen- eration using LSTMs with coarse-to-fine alignment. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 720-730, San Diego, California. Association for Computational Linguistics.
- Feng Nie, Jinpeng Wang, Jin-Ge Yao, Rong Pan, and Chin-Yew Lin. 2018. Operation-guided neu- ral networks for high fidelity data-to-text genera- tion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 3879-3889, Brussels, Belgium. Associ- ation for Computational Linguistics.
- Jekaterina Novikova, Ondřej Dušek, Amanda Cer- cas Curry, and Verena Rieser. 2017a. Why we need new evaluation metrics for NLG. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252, Copenhagen, Denmark. Association for Computa- tional Linguistics. · DOI · Paper page
- Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017b. The E2E dataset: New challenges for end- to-end generation. In Proceedings of the 18th An- nual SIGdial Meeting on Discourse and Dialogue, pages 201-206, Saarbrücken, Germany. Association for Computational Linguistics.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Com- putational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
- Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
- Laura Perez-Beltrachini and Claire Gardent. 2017. Analysing data-to-text generation benchmarks. In Proceedings of the 10th International Conference on Natural Language Generation, pages 238-242, San- tiago de Compostela, Spain. Association for Com- putational Linguistics.
- Laura Perez-Beltrachini and Mirella Lapata. 2018. Bootstrapping generators from noisy data. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long Papers), pages 1516-1527, New Or- leans, Louisiana. Association for Computational Linguistics.
- Ehud Reiter. 2018b. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393- 401. · DOI · Paper page
- Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Compu- tational Linguistics.
- Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei- Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural lan- guage generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711-1721, Lisbon, Portugal. Association for Com- putational Linguistics.
- Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gasic, Lina M. Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network- based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Compu- tational Linguistics: Volume 1, Long Papers, pages 438-449, Valencia, Spain. Association for Compu- tational Linguistics.
- Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
- Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dia- logue state tracking. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1458- 1467, Melbourne, Australia. Association for Com- putational Linguistics.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
“Currently, some works [137, 180, 230] point out that using commonsense knowledge can help to gain better numeral representation.”
BackgroundMethodInfluentialSiriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R., & Nanayakkara, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00530 · Cited by 337 · see citing works
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Xu, Z., Jain, S., & Kankanhalli, M. (2024) · arXiv · doi:10.48550/arxiv.2401.11817 · Cited by 180 · see citing works
“For data-related issues, solutions include fact-focused datasets [17, 21] and developing automatic data cleaning techniques [42, 51, 54].”
MethodSemantic Noise Matters for Neural Natural Language Generation
Dušek, O., Howcroft, D. M., & Rieser, V. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8652 · Cited by 85 · see citing works
AlignScore: Evaluating factual consistency with a unified alignment function
Zha, Y., Yang, Y., Li, R., & Hu, Z. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.634 · Cited by 81 · see citing works
On hallucination and predictive uncertainty in conditional language generation
Xiao, Y., & Wang, W. Y. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.236 · Cited by 79 · see citing works
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.464 · Cited by 63 · see citing works
Controlling hallucinations at word level in data-to-text generation
Rebuffel, C., Roberti, M., Soulier, L., Scoutheeten, G., Cancelliere, R., & Gallinari, P. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-021-00801-4 · Cited by 50 · see citing works
Slot-consistent NLG for Task-oriented Dialogue Systems with Iterative Rectification Network
Li, Y., Yao, K., Qin, L., Che, W., Li, X., & Liu, T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.10 · Cited by 19 · see citing works
Sometimes We Want Ungrammatical Translations
Parthasarathi, P., Sinha, K., Pineau, J., & Williams, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.275 · Cited by 9 · see citing works
- Authors
- Nie, F., Yao, J.-G., Wang, J., Pan, R., & Lin, C.-Y.
- Affiliations
- Sun Yat-sen University (CN)
- Microsoft Research Asia (China) (CN)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 2673-2679
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2953035981
- Cited by
- 68
- References
- 1
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 7 · 2023: 13 · 2022: 12 · 2021: 19 · 2020: 11 · 2019: 2
- FWCI
- 4.21
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Key Research and Development Program of China
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- hallucination mitigation, self-training, data refinement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
