The DOI matches its Crossref record.
Results of the WMT15 Metrics Shared Task
Open accessAbstract
This paper presents the results of the WMT15 Metrics Shared Task.We asked participants of this task to score the outputs of the MT systems involved in the WMT15 Shared Translation Task.We collected scores of 46 metrics from 11 research groups.In addition to that, we computed scores of 7 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines.The collected scores were evaluated in terms of system level correlation (how well each metric's scores correlate with WMT15 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CHRF3 was found to correlate well with human judgments, especially for translations out of English.
“and CHRF3 (Popović, 2015), a character n-gram F 3 score which was found to correlate well with human judgments, especially for translations out of English (Stanojević et al., 2015).”
From Neural Machine Translation of Rare Words with Subword Units (2016) · cites this paper as Stanojević et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 24
- Ondřej Bojar, Miloš Ercegovčević, Martin Popel, and Omar Zaidan. 2011. A Grain of Salt for the WMT Manual Evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 1-11, Edinburgh, Scotland, July. Association for Computational Linguistics.
- Ondřej Bojar, Matouš Macháček, Aleš Tamchyna, and Daniel Zeman. 2013. Scratching the Surface of Pos- sible Translations. In Proc. of TSD 2013, Lecture Notes in Artificial Intelligence, Berlin / Heidelberg. Západočeská univerzita v Plzni, Springer Verlag.
- Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics.
- Boxing Chen, Hongyu Guo, and Roland Kuhn. 2015. Multi-level Evaluation for Machine Translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics.
- Elisabet Comelles and Jordi Atserias. 2015. VERTa: a Linguistically-motivated Metric at the WMT15 Met- rics Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portu- gal, September. Association for Computational Lin- guistics.
- George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co- occurrence statistics. In Proceedings of the Sec- ond International Conference on Human Language Technology Research, HLT '02, pages 138-145, San Francisco, CA, USA.
- Marina Fomicheva, Núria Bel, Iria da Cunha, and An- ton Malinovskiy. 2015. UPF-Cobalt Submission to WMT15 Metrics Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lis- boa, Portugal, September. Association for Computa- tional Linguistics.
- Rohit Gupta, Constantin Orasan, and Josef van Gen- abith. 2015a. Machine Translation Evaluation us- ing Recurrent Neural Networks. In Proceedings of the Tenth Workshop on Statistical Machine Transla- tion, Lisboa, Portugal, September. Association for Computational Linguistics.
- Rohit Gupta, Constantin Orasan, and Josef van Gen- abith. 2015b. ReVal: A Simple and Effective Ma- chine Translation Evaluation Metric Based on Re- current Neural Networks. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP '15, Lisbon, Portugal.
- Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between european languages. In Proceedings of the Work- shop on Statistical Machine Translation, StatMT '06, pages 102-121, Stroudsburg, PA, USA. Asso- ciation for Computational Linguistics.
- Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. Cder: Efficient mt evaluation using block movements. In In Proceedings of EACL, pages 241- 248.
- Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 Metrics Shared Task. In Proceed- ings of the Eighth Workshop on Statistical Machine Translation, pages 45-51, Sofia, Bulgaria, August. Association for Computational Linguistics.
- Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 Metrics Shared Task. In Proceedings of the Ninth Workshop on Statistical Machine Trans- lation, pages 293-301, Baltimore, Maryland, USA, June. Association for Computational Linguistics.
- Benjamin Marie and Marianna Apidianaki. 2015. Alignment-based sense selection in METEOR and the RATATOUILLE recipe. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Com- putational Linguistics.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Com- putational Linguistics, ACL '02, pages 311-318, Stroudsburg, PA, USA. Association for Computa- tional Linguistics.
- Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Com- putational Linguistics.
- Raphael Rubino, Tommi Pirinen, Miquel Esplà- Gomis, Nikola Ljubešić, Sergio Ortiz Rojas, Vas- silis Papavassiliou, Prokopis Prokopidis, and Anto- nio Toral. 2015. Abu-MaTran at WMT 2015 Trans- lation Task: Morphological Segmentation and Web Crawling. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portu- gal, September. Association for Computational Lin- guistics.
- Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin- nea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annota- tion. In In Proceedings of Association for Machine Translation in the Americas, pages 223-231.
- Miloš Stanojević and Khalil Sima'an. 2015. BEER 1.1: ILLC UvA submission to metrics and tuning task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguis- tics.
- Miloš Stanojević, Amir Kamran, and Ondřej Bojar. 2015. Results of the WMT15 Tuning Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics.
- Yolanda Vazquez-Alvarez and Mark Huckvale. 2002. The reliability of the ITU-t p.85 standard for the evaluation of text-to-speech systems. In Proc. of IC- SLP -INTERSPEECH.
- Mihaela Vela and Liling Tan. 2015. Predicting Ma- chine Translation Adequacy with Document Embed- dings. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguis- tics.
- Sami Virpioja and Stig-Arne Grönroos. 2015. LeBLEU: N-gram-based Translation Evaluation Score for Morphologically Complex Languages. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal, September. Association for Computational Linguistics.
- Hui Yu, Qingsong Ma, Xiaofeng Wu, and Qun Liu. 2015. CASICT-DCU Participation in WMT2015 Metrics Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portu- gal, September. Association for Computational Lin- guistics.
Cited by
Sorted by citations
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159
Metadata
- Authors
- Stanojević, M., Kamran, A., Koehn, P., & Bojar, O.
- Affiliations
- University of Amsterdam (NL)
- Johns Hopkins University (US)
- Charles University (CZ)
- Venue
- Proceedings of the Tenth Workshop on Statistical Machine Translation
- Published
- 2015
- Pages
- 256-273
- OpenAlex
- W2916548775
- Cited by
- 102
- References
- 0
- Citations by year
- 2024: 6 · 2023: 18 · 2022: 7 · 2021: 6 · 2020: 5 · 2019: 10 · 2018: 12 · 2017: 15 · 2016: 19 · 2015: 3
- FWCI
- 8.77
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Machine Translation (WMT) (publishedVersion, cc-by)
- RWTH Publications (RWTH Aachen) (publishedVersion, other-oa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
- Funders
- European Commission (HORIZON2020, 645452, 644402)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Keywords
- machine translation evaluation, translation metrics, shared task, metric correlation, BLEU, NIST, PER, TER, system ranking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
