DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations
Open accessAbstract
Automatic evaluation metrics are essential for the rapid development of open-domain dialogue systems as they facilitate hyper-parameter tuning and comparison between models. Although recently proposed trainable conversation-level metrics have shown encouraging results, the quality of the metrics is strongly dependent on the quality of training data. Prior works mainly resort to heuristic text-level manipulations (e.g. utterances shuffling) to bootstrap incoherent conversations (negative examples) from coherent dialogues (positive examples). Such approaches are insufficient to appropriately reflect the incoherence that occurs in interactions between advanced dialogue models and humans. To tackle this problem, we propose DEAM, a Dialogue coherence Evaluation metric that relies on Abstract Meaning Representation (AMR) to apply semantic-level Manipulations for incoherent (negative) data generation. AMRs naturally facilitate the injection of various types of incoherence sources, such as coreference inconsistency, irrelevancy, contradictions, and decrease engagement, at the semantic level, thus resulting in more natural incoherent samples. Our experiments show that DEAM achieves higher correlations with human judgments compared to baseline methods on several dialog datasets by significant margins. We also show that DEAM can distinguish between coherent and incoherent dialogues generated by baseline manipulations, whereas those baseline models cannot detect incoherent examples generated by DEAM. Our results demonstrate the potential of AMR-based semantic manipulations for natural negative example generation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Conventional evaluation metrics for conversational agents are not well-suited for directly assessing comprehension of long-term contexts.
“However, these evaluation metrics are not well-suited for directly assessing the agent's comprehension of long-term contexts.”
From Evaluating Very Long-Term Conversational Memory of LLM Agents (2024) · cites this paper as Ghazarian et al., 2022
Cites
Still being checked.
Cited by
Sorted by citations
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37 · see citing works
MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation
Feng, J., Sun, Q., Xu, C., Zhao, P., Yang, Y., Tao, C., Zhao, D., & Lin, Q. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.405 · Cited by 26 · see citing works
Ye, G., Zhao, H., Zhang, Z., & Jiang, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104035 · Cited by 16 · see citing works
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation
Zhang, C., D’Haro, L. F., Zhang, Q., Friedrichs, T., & Li, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.220 · Cited by 14 · see citing works
Leveraging Large Language Models for Automated Dialogue Analysis
Finch, S. E., Paek, E. S., & Choi, J. D. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.20 · Cited by 12 · see citing works
EnDex: Evaluation of Dialogue Engagingness at Scale
Xu, G., Liu, R., Harel-Canada, F., Chandra, N. R., & Peng, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.359 · Cited by 10 · see citing works
Self-citationApproximating Online Human Evaluation of Social Chatbots with Prompting
Svikhnushina, E., & Pu, P. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.25 · Cited by 10 · see citing works
Zhou, J., Pang, L., Shen, H., & Cheng, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.553 · Cited by 7 · see citing works
Sataer, Y., Fan, Y., Li, B., Gao, M., Shi, C., & Gao, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103698 · Cited by 4 · see citing works
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark
Zhang, C., D’Haro, L. F., Tang, C., Shi, K., Tang, G., & Li, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.371 · Cited by 4 · see citing works
Evaluating Very Long-Term Conversational Memory of LLM Agents
Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024) · arXiv · doi:10.48550/arxiv.2402.17753 · Cited by 3 · see citing works
“…employs lexical overlap (Papineni et al., 2002) and semantic overlap (Zhang et al., 2019) between ground truth and the agent response, or consistency (Ghazarian et al., 2022), contradiction (Nie et al., 2021; Welleck et al., 2019), and empathy (Zhang et al., 2021a, 2022) of the agent response.”
BackgroundACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems
Ghazarian, S., Shao, Y., Han, R., Galstyan, A., & Peng, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.241 · Cited by 3 · see citing works
Self-citationSemantic Coherence Analysis of English Texts Based on Sentence Semantic Graphs
Deng, N., Wang, Y., Huang, G., Zhou, Y., & Li, Y. (2023) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.3312 · Cited by 3 · see citing works
Hwang, Y., Kang, D., & Bak, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110609 · Cited by 2 · see citing works
Dial-M: A Masking-based Framework for Dialogue Evaluation
Dey, S., & Desarkar, M. S. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.7 · Cited by 2 · see citing works
Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs
Komma, A., Chandrasekarasastry, N. P., Leffel, T., Goyal, A. K., Metallinou, A., Matsoukas, S., & Galstyan, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.19 · Cited by 1 · see citing works
Self-citationMERCY: Multiple Response Ranking Concurrently in Realistic Open-Domain Conversational Systems
Ghazarian, S., Hedayatnia, B., Jin, D., Liu, S., Peng, N., Liu, Y., & Hakkani-Tür, D. Z. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.58 · Cited by 1 · see citing works
Self-citationAutomatic Evaluate Dialogue Appropriateness by Using Dialogue Act
Chen, B., Wang, Y., Liu, Z., & Guo, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.492 · Cited by 1 · see citing works
Research on the Architecture Design Based on Large Language Model
Li, X., Li, M., Xia, X., Song, J., & Bu, X. (2024) · International Conference on Interactive Intelligent Systems and Techniques (IIST) · doi:10.1109/iist62526.2024.00145 · Cited by 1 · see citing works
Investigating the Representation of Open Domain Dialogue Context for Transformer Models
Padmakumar, V., Hedayatnia, B., Jin, D., Lange, P. L., Kim, S., Peng, N., Liu, Y., & Hakkani-Tur, D. Z. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.50
Self-citationDiQAD: A Benchmark Dataset for Open-domain Dialogue Quality Assessment
Zhao, Y., Yan, L., Sun, W., Meng, C., Wang, S., Cheng, Z., Ren, Z., & Yin, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1010
Enhancing Dialogue Discourse Coherence through Query Reconstruction
Zhu, Z.-A., Tseng, S.-W., & Chiang, C.-K. (2024) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-taiwan62264.2024.10674155
Ye, G., Zhao, H., Li, B., Chen, H., Zhao, Z., Jiang, Z., & Li, K. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3580272
Liu, X., Xie, Z., Zhou, J., & Jiang, S. (2025) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194025500433
A robust dialogue evaluation metric exploiting denoising, pre-training and ensembling
Rao, D., Ling, L., & Jiang, Z. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112369
- Authors
- Ghazarian, S., Wen, N., Galstyan, A., & Peng, N.
- Affiliations
- University of Southern California (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 771-785
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 2203.09711
- OpenAlex
- W4221145220
- Cited by
- 26
- References
- 0
- Citations by year
- 2025: 5 · 2024: 5 · 2023: 14 · 2022: 2
- FWCI
- 3.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- Abstract Meaning Representation, semantic manipulation, open-domain dialogue systems, contradiction, engagement, human judgment correlation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
