Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 31

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.

The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation

Fernandes, Patrick, Deutsch, Daniel, Finkelstein, Mara, Riley, Parker, Martins, André, Neubig, Graham, Garg, Ankush, Clark, Jonathan, Freitag, Markus, Firat, Orhan·2023·Proceedings of the Eighth Conference on Machine Translation
DOI 10.18653/v1/2023.wmt-1.100
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.wmt-1.100

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Illustration of how AUTOMQM uses LLMs to assess the quality of a translation. Rather than asking for a single quality score, AUTOMQM prompts models to identify and classify errors, and uses the MQM framework to produce a score.
Figure 1 Illustration of how AUTOMQM uses LLMs to assess the quality of a translation. Rather than asking for a single quality score, AUTOMQM prompts models to identify and classify errors, and uses the MQM framework to produce a score.CC-BY · From the PDF
Figure 3 Distribution of scores for various LLM referencebased evaluators, on the EN-DE test set. Note that the y axis is in log-scale .
Figure 3 Distribution of scores for various LLM referencebased evaluators, on the EN-DE test set. Note that the y axis is in log-scale .CC-BY · From the PDF
Figure 4 Behavior of Pearson as we scale the LLM's parameter count. Note that the x axis is not to-scale with regard to parameter count.
Figure 4 Behavior of Pearson as we scale the LLM's parameter count. Note that the x axis is not to-scale with regard to parameter count.CC-BY · From the PDF
Figure 5 Mean Pearson and its interquartile range (IQR) in the WMT22 EN-DE test set, as we increase the number of in-context examples with stratified sampling
Figure 5 Mean Pearson and its interquartile range (IQR) in the WMT22 EN-DE test set, as we increase the number of in-context examples with stratified samplingCC-BY · From the PDF
Figure 6 Distribution of scores for PaLM-2 (BISON) models for 1-shot (top) and 2-shot (bottom) setups, with various incontext learning sets for each (and their scores in the legend)
Figure 6 Distribution of scores for PaLM-2 (BISON) models for 1-shot (top) and 2-shot (bottom) setups, with various incontext learning sets for each (and their scores in the legend)CC-BY · From the PDF
Figure 7 Scatter plot of the Pearson of PaLM-2 (BISON) models, with/without including the reference in the prompt, for each in-context learning setting tried.
Figure 7 Scatter plot of the Pearson of PaLM-2 (BISON) models, with/without including the reference in the prompt, for each in-context learning setting tried.CC-BY · From the PDF
Figure 8 Distribution of scores for PaLM-2 models using AUTOMQM, on WMT22 EN-DE
Figure 8 Distribution of scores for PaLM-2 models using AUTOMQM, on WMT22 EN-DECC-BY · From the PDF
Image 8
CC-BY · From the PDF
Figure 12 Mean Pearson and its interquartile range (IQR), as we increase the number of in-context examples in the score prediction prompt, sampled with uniform (left) and stratified (right) sampling, for WMT22 ZH-EN.
Figure 12 Mean Pearson and its interquartile range (IQR), as we increase the number of in-context examples in the score prediction prompt, sampled with uniform (left) and stratified (right) sampling, for WMT22 ZH-EN.CC-BY · From the PDF
Figure 13 Distribution of scores for various LLM referencebased evaluators, on the ZH-EN test set. Note that the y axis is in log-scale .
Figure 13 Distribution of scores for various LLM referencebased evaluators, on the ZH-EN test set. Note that the y axis is in log-scale .CC-BY · From the PDF
Figure 14 Mean Pearson and its interquartile range (IQR), as we increase the number of in-context examples in the AUTOMQM prompt, for EN-DE (left) and ZH-EN (right).
Figure 14 Mean Pearson and its interquartile range (IQR), as we increase the number of in-context examples in the AUTOMQM prompt, for EN-DE (left) and ZH-EN (right).CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 16 references

Research works printed but not linked: 15

  1. Chris Callison-Burch, Philipp Koehn, Christof Monz, Josh Schroeder, and Cameron Shaw Fordyce, editors. 2008. Proceedings of the Third Workshop on Statisti- cal Machine Translation. Association for Computa- tional Linguistics, Columbus, Ohio.
  2. Erick Fonseca, Lisa Yankovskaya, Andrà c F. T. Mar- tins, Mark Fishel, and Christian Federmann. 2019. Findings of the wmt 2019 shared tasks on quality esti- mation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 1-12, Florence, Italy. Association for Computational Linguistics.
  3. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU -neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  4. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceed- ings of the Sixth Conference on Machine Translation, pages 733-774, Online. Association for Computa- tional Linguistics.
  5. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Ma- chine Translation (WMT), pages 1-45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
  6. Tom Kocmi, Christian Federmann, Roman Grund- kiewicz, Marcin Junczys-Dowmunt, Hitokazu Mat- sushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478-494, Online. Association for Computational Linguistics.
  7. Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. Error analysis prompting en- ables human-like translation evaluation in large lan- guage models: A case study on chatgpt. arXiv preprint.
  8. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT sys- tems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90, Flo- rence, Italy. Association for Computational Linguis- tics. OpenAI. 2023. Gpt-4 technical report. · DOI
  9. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. · DOI · Paper page
  10. Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Niccolò Campolungo, and Roberto Navigli. 2022. MaTESe: Machine translation evaluation as a se- quence tagging problem. In Proceedings of the Sev- enth Conference on Machine Translation (WMT), pages 569-577, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  11. Yiwei Qin, Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2022. T5score: Discriminative fine- tuning of generative evaluation metrics. ArXiv, abs/2212.05726. · arXiv
  12. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  13. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634-645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
  14. Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orȃsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 69-99, Abu Dhabi, United Arab Emi- rates (Hybrid). Association for Computational Lin- guistics.
  15. Chrysoula Zerva, Daan van Stigt, Ricardo Rei, Ana C Farinha, Pedro Ramos, José G. C. de Souza, Taisiya Glushkova, Miguel Vera, Fabio Kepler, and André F. T. Martins. 2021. IST-unbabel 2021 submission for the quality estimation shared task. In Proceed- ings of the Sixth Conference on Machine Translation, pages 961-972, Online. Association for Computa- tional Linguistics.

Cited by

See all 31

Metadata

Authors
Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Martins, A., Neubig, G., Garg, A., Clark, J., Freitag, M., & Firat, O.
Venue
Proceedings of the Eighth Conference on Machine Translation
Publisher
Association for Computational Linguistics
Published
2023
Pages
1066-1083
DOI
10.18653/v1/2023.wmt-1.100
Cited by
31
References
16

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation. https://app.cochat.ai/papers/10.18653/v1/2023.wmt-1.100
The paper
Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Martins, A., Neubig, G., Garg, A., Clark, J., Freitag, M., & Firat, O. (2023). The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation. Proceedings of the Eighth Conference on Machine Translation. https://doi.org/10.18653/v1/2023.wmt-1.100

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.