The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
chrF: character n-gram F-score for automatic MT evaluation
Open accessAbstract
We propose the use of character n-gram F-score for automatic evaluation of machine translation output.Character ngrams have already been used as a part of more complex metrics, but their individual potential has not been investigated yet.We report system-level correlations with human rankings for 6-gram F1-score (CHRF) on the WMT12, WMT13 and WMT14 data as well as segment-level correlation for 6gram F1 (CHRF) and F3-scores (CHRF3) on WMT14 data for all available target languages.The results are very promising, especially for the CHRF3 score -for translation from English, this variant showed the highest segment-level correlations outperforming even the best metrics on the WMT14 shared evaluation task.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CHRF3 was found to correlate well with human judgments, especially for translations out of English.
“and CHRF3 (Popović, 2015), a character n-gram F 3 score which was found to correlate well with human judgments, especially for translations out of English (Stanojević et al., 2015).”
From Neural Machine Translation of Rare Words with Subword Units (2016) · cites this paper as Popović, 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Flagged first, then by citations
Research works printed but not linked: 9
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Im- proved Correlation with Human Judgements. In Proceedings of the ACL 05 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pages 65-72, Ann Arbor, MI, June.
- Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation (WMT-13), pages 1-44, Sofia, Bulgaria, August.
- Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 Workshop on Statistical Machine Translation. In Proceedings of the Ninth Workshop on Statistical Machine Trans- lation (WMT-14), page 1258, Baltimore, Maryland, USA, June.
- Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 Workshop on Statistical Ma- chine Translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation (WMT- 12), page 1051, Montral, Canada, June. Association for Computational Linguistics.
- Kristen Parton, Joel Tetreault, Nitin Madnani, and Mar- tin Chodorow. 2011. E-Rating Machine Transla- tion. In Proceedings of the Sixth Workshop on Sta- tistical Machine Translation (WMT-11), pages 108- 115, Edinburgh, Scotland.
- Maja Popović. 2011. Morphemes and POS tags for n-gram based evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Trans- lation (WMT-11), pages 104-107, Edinburgh, Scot- land, July.
- Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin- nea Micciulla, and John Makhoul. 2006. A Study of Translation Error Rate with Targeted Human Anno- tation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Ameri- cas (AMTA 06), pages 223-231, Boston, MA, Au- gust.
- Miloš Stanojević and Khalil Sima'an. 2014a. BEER: BEtter Evaluation as Ranking. In Proceedings of the Ninth Workshop on Statistical Machine Trans- lation (WMT-14), pages 414-419, Baltimore, Mary- land, June.
- Miloš Stanojević and Khalil Sima'an. 2014b. Fitting Sentence Level Translation Evaluation with Many Dense Features. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processing (EMNLP-14), pages 202-206, Doha, Qatar, October. Association for Computational Lin- guistics.
Cited by
Sorted by citations
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337
“We evaluate the machine translation ability of Chat-GPT on both high-resource and low-resource languages using the ChrF++ metric (Popovi´c, 2015).”
BackgroundMethodExperts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation
Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., & Macherey, W. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00437 · Cited by 271
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., & Awadalla, H. H. (2023) · arXiv · doi:10.48550/arxiv.2302.09210 · Cited by 202
“Additionally, we report results using SacreBLEU11 and Chrf (Popović, 2015) for completeness, although we note that these metrics are not extensively used in our analysis.”
MethodThe Curious Case of Hallucinations in Neural Machine Translation
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.92 · Cited by 142
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing
Thompson, B., & Post, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.8 · Cited by 128
Reassessing automatic evaluation metrics for code summarization tasks
Roy, D., Fakhoury, S., & Arnaoudova, V. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468588 · Cited by 106
AlignScore: Evaluating factual consistency with a unified alignment function
Zha, Y., Yang, Y., Li, R., & Hu, Z. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.634 · Cited by 81
Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers
Marie, B., Fujita, A., & Rubino, R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.566 · Cited by 75
Seamless: Multilingual Expressive and Streaming Speech Translation
Communication, S., Barrault, L., Chung, Y.-A., Meglioli, M. C., Dale, D., Dong, N., Duppenthaler, M., Duquenne, P.-A., Ellis, B., Elsahar, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.05187 · Cited by 45
Kreutzer, J., Uyheng, J., & Riezler, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1165 · Cited by 38
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
Moramarco, F., Papadopoulos Korfiatis, A., Perera, M., Juric, D., Flann, J., Reiter, E., Belz, A., & Savkov, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.394 · Cited by 40
DEMETR: Diagnosing Evaluation Metrics for Translation
Karpinska, M., Raj, N., Thai, K., Song, Y., Gupta, A., & Iyyer, M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.649 · Cited by 22
Uncertainty-Aware Machine Translation Evaluation
Glushkova, T., Zerva, C., Rei, R., & Martins, A. F. T. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.330 · Cited by 17
Jiang, Z., & Zhang, Z. (2025) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2025.100237 · Cited by 10
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306
Metadata
- Authors
- Popović, M.
- Affiliations
- Humboldt-Universität zu Berlin (DE)
- Corresponding author
- Popović, Maja
- ORCID
- Popović, Maja
- Venue
- Proceedings of the Tenth Workshop on Statistical Machine Translation
- Published
- 2015
- Pages
- 392-395
- OpenAlex
- W2250342921
- Cited by
- 1,220
- References
- 1
- Citations by year
- 2026: 199 · 2025: 182 · 2024: 157 · 2023: 233 · 2022: 142 · 2021: 99 · 2020: 79 · 2019: 51 · 2018: 36 · 2017: 27 · 2016: 14 · 2015: 1
- FWCI
- 13.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Machine Translation (WMT) (publishedVersion, cc-by)
- Funders
- European Commission (610516, H2020-ICT, H2020-ICT-2014, 2007–2013, 644333, FP7 2007-2013, HORIZON2020, H2020, FP7/2007, H2020‐ICT‐2014‐1, 645452)
- Horizon 2020 Framework Programme (H2020-ICT-2014-1, FP7/2007-2013, ICT-2014-1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- machine translation evaluation, chrF, human evaluation correlation, translation quality metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
