Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)

Abstract

Conversational models for information seeking such as question answering and dialogue systems are traditionally evaluated using exact match or word overlap between the gold answer and predicted answer.With recent advances such as instruction-tuned language models, model generated answers are often verbose and naturalistic, making word overlap unreliable for measuring correctness of the answer.Moreover, the correctness of the answer is only one dimension of the problem.A generated response may contain the gold answer along with additional hallucinated information which has to be quantified as well.In this talk, I will present a holistic evaluation of retrieval-augmented conversational models and evaluate several models such as ChatGPT, Alpaca, T5, both using automatic metrics and humans.This holistic evaluation considers aspects like correctness with respect to ground truth, hallucinated information with respect to retrieved/supplied knowledge, reasoning and generalization ability.This work is informed by the author's work on TopiOCQA [1], FaithDial [2,3], [4] and NoPE [5]. Table of ContentsKnowComp at SemEval-2023 Task 7: Fine-tuning Pre-trained Language Models for Clinical Trial