The DOI matches its Crossref record; 134 of the 143 references it cites were found in a registry or an index; 1 of the 58 sources its claims cite were checked for support.
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Open accessAbstract
Generative Artificial Intelligence is set to revolutionize healthcare delivery by transforming traditional patient care into a more personalized, efficient, and proactive process. Chatbots, serving as interactive conversational models, will probably drive this patient-centered transformation in healthcare. Through the provision of various services, including diagnosis, personalized lifestyle recommendations, dynamic scheduling of follow-ups, and mental health support, the objective is to substantially augment patient health outcomes, all the while mitigating the workload burden on healthcare providers. The life-critical nature of healthcare applications necessitates establishing a unified and comprehensive set of evaluation metrics for conversational models. Existing evaluation metrics proposed for various generic large language models (LLMs) demonstrate a lack of comprehension regarding medical and health concepts and their significance in promoting patients’ well-being. Moreover, these metrics neglect pivotal user-centered aspects, including trust-building, ethics, personalization, empathy, user comprehension, and emotional support. The purpose of this paper is to explore state-of-the-art LLM-based evaluation metrics that are specifically applicable to the assessment of interactive conversational models in healthcare. Subsequently, we present a comprehensive set of evaluation metrics designed to thoroughly assess the performance of healthcare chatbots from an end-user perspective. These metrics encompass an evaluation of language processing abilities, impact on real-world clinical tasks, and effectiveness in user-interactive conversations. Finally, we engage in a discussion concerning the challenges associated with defining and implementing these metrics, with particular emphasis on confounding factors such as the target audience, evaluation methods, and prompt techniques involved in the evaluation process.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This perspective paper argues that existing LLM evaluation metrics are inadequate for healthcare chatbots because they miss medical semantics and user-centered concerns such as trust, empathy, ethics, personalization, and hallucination. It proposes a comprehensive set of foundation metrics grouped into accuracy, trustworthiness, empathy, and performance, along with an evaluation framework and discussion of implementation challenges.
Key findings
- Existing intrinsic metrics such as BLEU and ROUGE rely on surface-form similarity and fail to capture medical concepts, semantics, context, and human-centered aspects; the paper illustrates this with two semantically similar health sentences that scored only 0.39 BLEU and 0.13 ROUGE on a 0–1 scale.
- Existing extrinsic and multi-metric evaluations remain incomplete for healthcare chatbots, often neglecting empathy, reasoning, up-to-dateness, hallucinations, personalization, relevance, and latency.
- The paper proposes four user-centered metric groups—accuracy, trustworthiness, empathy, and performance—with submetrics such as groundedness, privacy, bias, emotional support, health literacy, fairness, personalization, memory efficiency, FLOP, token limit, and number of parameters.
- It identifies three confounding variables for evaluation—user type, domain type, and task type—and presents a framework with models, environment, interface, interacting users, and leaderboard components.
Methods
- Narrative review and position paper: reviews intrinsic and extrinsic evaluation metrics for LLMs, including general-purpose, health-specific, and multi-metric approaches.
- Synthesizes a proposed set of healthcare chatbot metrics from the literature, categorizes them into four groups, and maps each metric to healthcare problems and existing benchmarks in Table 2.
- No empirical data collection, model training, or experimental evaluation is reported; the paper proposes an evaluation framework and discusses challenges and future implementation.
Limitations
- The paper states that future work is needed to implement the proposed evaluation framework, conduct extensive metric assessment using benchmarks and case studies, and establish unified benchmarks tailored to healthcare chatbots.
- It discusses challenges including metric associations and correlations, selection of evaluation methods, subjectivity and need for diverse expert annotators, and effects of prompt techniques and model parameters.
- Evident from the text: the proposed metrics and framework are not empirically validated in this paper, and no sample, dataset, or model evaluation results are reported beyond the illustrative BLEU/ROUGE example.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
32 Own interpretation · 24 Not checked yet · 6 No basis given · 1 Supported · 1 Own method
In the paper
“Despite the contextual similarity between the two sentences, the obtained BLEU and ROUGE scores are 0.39 and 0.13, respectively, on a scale of 0 to 1, reflecting low alignment.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Intrinsic evaluation metrics
What the sources say
- Not checked yetcites HuggingFace24
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Notably, Laing et al.5 have presented a multi-metric approach, as part of the HELM benchmark, to scrutinize LLMs concerning their accuracy, calibration (proficiency in assigning meaningful probabilities for generated text), robustness, fairness, bias, toxicity, and efficiency.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Multi-metric measurements
What the sources say
- Not checked yetcites [5] Liang, P. et al. Holistic evaluation of language models. Trans. Machine Learn. Res. https://openreview.net/forum?id=iO4LZibEqW (2023).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“However, they solely rely on surface-form similarity and language-specific perspectives, rendering them inadequate for healthcare chatbots.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Intrinsic evaluation metrics
What the sources say
- Not checked yetcites [19] Sai, A. B., Mohankumar, A. K. & Khapra, M. M. A survey of evaluation metrics used for NLG systems. ACM Comput. Surv. (CSUR) 55, 1–39 (2022). source
- Not checked yetcites [20] Zhang*, T., Kishore*, V., Wu*, F., Weinberger, K. Q. & Artzi, Y. Bertscore: evaluating text generation with bert. In International Conference on Learning Representations. (International Conference on Learning Representations (ICLR), 2020).
- Not checked yetcites [21] Khurana, D., Koli, A., Khatter, K. & Singh, S. Natural language processing: state of the art, current trends and challenges. Multimedia Tools Appl. 82, 3713–3744 (2023). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, if we solely use these metrics to evaluate a healthcare chatbot, an inaccurate answer may receive a high score compared with the reference answer.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Intrinsic evaluation metrics
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“However, these methods have merely concentrated on specific aspects, such as the robustness of the generated answers within a particular medical domain.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Introduction
What the sources say
- Not checked yetcites [7] Thoppilan, R. et al. LaMDA: language models for dialog applications. Preprint at https://arxiv.org/abs/2201.08239 (2022).
- Not checked yetcites [9] Adiwardana, D. et al. Towards a human-like open-domain chatbot. Preprint at https://arxiv.org/abs/2001.09977 (2020).
- Not checked yetcites [12] Peng, B. et al. GODEL: large-scale pre-training for goal-directed dialog. Preprint at https://arxiv.org/abs/2206.11309 (2022).
- Not checked yetcites [13] Touvron, H. et al. LLaMA: open and efficient foundation language models. Preprint at https://arxiv.org/abs/2302.13971 (2023).
- Not checked yetcites [14] Wang, P. et al. Large language models are not fair evaluators. Preprint at https://arxiv.org/abs/2305.17926 (2023).
- Not checked yetcites [15] Resnik, P. et al. Using intrinsic and extrinsic metrics to evaluate accuracy and facilitation in computer-assisted coding. In Perspectives in Health Information Management Computer Assisted Coding Conference Proceedings (The American Health Information Management Association (AHIMA), 2006).
- Not checked yetcites [16] Liu, P. et al. Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55, 1–35 (2023).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For example, these studies unable to assess chatbots in terms of empathy, reasoning, up-to-dateness, hallucinations, personalization, relevance, and latency.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Multi-metric measurements
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
In the paper
“Although these metrics are model-based, they lack an understanding of medical concepts (e.g., symptoms, diagnostic tests, diagnoses, and treatments), their interplay, and the priority for the well-being of the patient, all of which are crucial for medical decision-making10.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Introduction
What the sources say
- Not checked yetcites [10] Silfen, E. Documentation and coding of ED patient encounters: an evaluation of the accuracy of an electronic medical record. Am J Emerg Med. 24, 664–678 (2006). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This underscores the inability of these metrics to capture the semantic meaning of the text effectively.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Intrinsic evaluation metrics
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite these contributions, it is evident that these studies have yet to fully encompass the indispensable, multifaceted, and user-centered evaluation metrics necessary to appraise healthcare chatbots comprehensively.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Multi-metric measurements
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“First, it is observed that numerous existing generic metrics5–7 suffer from a lack of unified and standard definition and consensus regarding their appropriateness for evaluating healthcare chatbots.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Introduction
What the sources say
- Not checked yetcites [5] Liang, P. et al. Holistic evaluation of language models. Trans. Machine Learn. Res. https://openreview.net/forum?id=iO4LZibEqW (2023).
- Not checked yetcites [6] Wang, B. et al. Decodingtrust: a comprehensive assessment of trustworthiness in GPT models. Preprint at https://arxiv.org/abs/2306.11698 (2023).
- Not checked yetcites [7] Thoppilan, R. et al. LaMDA: language models for dialog applications. Preprint at https://arxiv.org/abs/2201.08239 (2022).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 41
- It may state two separate claims as one. 11
Every source the paper cites for this claim was checked and supports it.
In the paper
“temperature67, which can influence specific metric scores.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Model prompt techniques and parameters
What the sources say
- Supportedcites [67] Chung, J., Kamar, E. & Amershi, S. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 575–593 (Association for Computational Linguistics, Toronto, 2023). source
Section 4.3 reports that higher temperature increased diversity and decreased label accuracy, while also affecting model accuracy under some conditions. The Ethics Statement notes that the patterns varied by task, supporting the claim that temperature can influence specific metric scores. (LLM, conf=0.93)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evaluation of the model’s performance encompasses diverse factors, such as safety and privacy, which are contingent upon the specific users or audience involved.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“For this reason, they inadequately capture vital aspects like semantic nuances, contextual relevance, long-range dependencies, changes in critical semantic ordering, and human-centric perspectives11, thereby limiting their effectiveness in evaluating healthcare chatbots.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Introduction
What the sources say
- Not checked yetcites [11] Novikova, J., Dušek, O., Cercas Curry, A. & Rieser, V. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2241–2252 (Association for Computational Linguistics, Copenhagen, Denmark, 2017). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To address this, groundedness leverages relevant factual information, promoting sound reasoning and staying up-to-date ensuring validity.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- Rests on an assumption: In the healthcare domain, safety and currentness of information are paramount.
None of the sources cited for this claim have been checked yet.
In the paper
“The definition of these accuracy metrics is contingent upon the domain and task types involved5,25.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
What the sources say
- Not checked yetcites [5] Liang, P. et al. Holistic evaluation of language models. Trans. Machine Learn. Res. https://openreview.net/forum?id=iO4LZibEqW (2023).
- Not checked yetcites [25] AI Risk Management Framework—nist.gov. https://www.nist.gov/itl/ai-risk-management-framework (2023).
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evaluation of the model and metric scoring may differ depending on the specific task at hand.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“In addition, Chang et al.8 have presented organized evaluation methodologies for LLMs through three essential dimensions: “what to evaluate,” “where to evaluate,” and “how to evaluate.””
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Multi-metric measurements
What the sources say
- Not checked yetcites [8] Chang, Y. et al. A survey on evaluation of large language models. ACM ransactions on Intelligent Systems and Technology. https://doi.org/10.1145/3641289 (2024). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Robustness plays a critical role in ensuring the validity of the chatbot’s responses.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Human inquiries may not precisely align with their underlying issues or intent.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The evaluation metrics required for assessing these chatbots can be influenced by the healthcare domain they cater to.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, each of these studies has been confined to a distinct set of metrics, thereby neglecting to embrace the comprehensive and all-encompassing aspect concerning healthcare language models and chatbots.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Extrinsic evaluation metrics
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As outlined below, the metrics are categorized into four distinct groups: accuracy, trustworthiness, empathy, and performance, based on their dependencies on the confounding variables.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In this evaluation process, three confounding variables are taken into account: user type, domain type, and task type.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Likewise, Wang et al.6 have assessed the trustworthiness of GPT-3.5 and GPT-4 from eight discerning aspects encompassing toxicity, bias, robustness, privacy, machine ethics, and fairness.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Multi-metric measurements
What the sources say
- Not checked yetcites [6] Wang, B. et al. Decodingtrust: a comprehensive assessment of trustworthiness in GPT models. Preprint at https://arxiv.org/abs/2306.11698 (2023).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“These metrics lack the capability to capture essential elements such as semantics19,20, context19,21, distant dependencies22,23, semantically critical ordering change21, and human perspectives, particularly in real-world scenarios.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Review of existing evaluation metrics for LLMs > Intrinsic evaluation metrics
What the sources say
- Not checked yetcites [19] Sai, A. B., Mohankumar, A. K. & Khapra, M. M. A survey of evaluation metrics used for NLG systems. ACM Comput. Surv. (CSUR) 55, 1–39 (2022). source
- Not checked yetcites [20] Zhang*, T., Kishore*, V., Wu*, F., Weinberger, K. Q. & Artzi, Y. Bertscore: evaluating text generation with bert. In International Conference on Learning Representations. (International Conference on Learning Representations (ICLR), 2020).
- Not checked yetcites [21] Khurana, D., Koli, A., Khatter, K. & Singh, S. Natural language processing: state of the art, current trends and challenges. Multimedia Tools Appl. 82, 3713–3744 (2023). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, existing evaluations overlook performance aspects of models, such as computational efficiency and model size, which are crucial for practical implementation.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, it is evident that the existing evaluation metrics overlook a wide range of crucial user-centered aspects that indicate the extent to which a chatbot establishes a connection and conveys support and emotion to the patient.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Introduction
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
- Rests on an assumption: Emotional bonds play a vital role in physician–patient communications.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A chatbot’s ability to generalize enhances its validity in effectively addressing a wide range of medical scenarios.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: There is scarcity of data and information across various medical domains and categories.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Up-to-dateness significantly enhances the validity of a chatbot by ensuring that its information aligns with the latest evidence and guidelines.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: In the healthcare domain, safety and currentness of information are paramount.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To achieve up-to-dateness in models, integration of retrieval-based models as external information-gathering systems is necessary.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, the current methods fail to address the issue of hallucination, wherein chatbots generate misleading or inaccurate information.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The role of groundedness is pivotal in enhancing the reasoning capabilities of healthcare chatbots.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: In the healthcare domain, safety and currentness of information are paramount.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in the domain of medical report generation, the utmost importance lies in ensuring the reliability and factuality of the generated text, a requirement that might not be as critical when the task involves acting as an assistant.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is important to note that accuracy metrics might remain invariant with regard to the user’s type, as the ultimate objective of the generated text is to achieve the highest level of accuracy, irrespective of the intended recipient.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, metrics in trustworthiness and empathy may be correlated.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Metrics association
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A significant relationship exists between performance metrics and the other three categories.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Metrics association
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The integration of the aforementioned requirements should result in the desired scores, treating the evaluation component as a black box.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Toward an effective evaluation framework
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, a notable concern arises when employing existing benchmarks (see Table 2) to automatically evaluate relevant metrics. These benchmarks may lack comprehensive assessments of the chatbot model’s robustness concerning confounding variables specific to the target user type, domain type, and task type.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Evaluation methods
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Human-based methods involve providing questions or guidelines to human annotators who score the chatbot’s generated answers based on given criteria. This approach presents two main challenges: subjectivity and the need for a variety of domain expert annotators.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Evaluation methods
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Enhancing fairness within a healthcare chatbot’s responses contributes to increased reliability by ensuring that the chatbot consistently provides equitable and unbiased answers.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Empathy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An increase in parameters may introduce complexity, potentially affecting these metrics positively or negatively.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Metrics association
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empathy often necessitates personalization, which can potentially compromise privacy and lead to biased responses.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Metrics association
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, a model trained on an imbalanced dataset, with dominant samples from white males and limited samples from Hispanic females, might exhibit bias due to the imbalanced training dataset. Consequently, it may provide unfair responses to Hispanic females, as their patterns were not accurately learned during the training process.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Empathy
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ability to adjust these variables in the evaluation framework ensures alignment among all stakeholders evaluating the target healthcare chatbot model, fostering a consistent and uniform evaluation perspective.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Toward an effective evaluation framework
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Reducing the number of parameters, which often leads to decreased memory usage and FLOPs, is likely to improve usability and latency, making the model more efficient and effective in practical applications.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Performance
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Modifying these parameters can influence the chatbot’s behavior when responding to queries.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Model prompt techniques and parameters
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Personalization enhances the reliability of a chatbot by tailoring its interactions and healthcare recommendations to individual users, ensuring that responses align closely with their preferences and health-related data.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Empathy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Popular LLMs, such as GPT-4, Llama, and BERT, often require large memory capacity13,58–61, making it challenging to run them on devices with limited memory, such as embedded systems, laptops, and mobile phones62.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Performance
What the sources say
- Not checked yetcites [13] Touvron, H. et al. LLaMA: open and efficient foundation language models. Preprint at https://arxiv.org/abs/2302.13971 (2023).
- Not checked yetcites [58] Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).
- Not checked yetcites [59] Achiam, J. et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023).
- Not checked yetcites [60] Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186 (Association for Computational Linguistics, Minneapolis, 2019).
- Not checked yetcites [61] Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234–1240 (2019). source
- Not checked yetcites [62] Zhuang, B. et al. A survey on efficient training of transformers. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 (ed. Elkind, E.) 6823–6831 (International Joint Conferences on Artificial Intelligence Organization, 2023). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Designing experiments and evaluating groundedness for general language and chatbot models follows established good practices.7,30,34–37”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
What the sources say
- Not checked yetcites [7] Thoppilan, R. et al. LaMDA: language models for dialog applications. Preprint at https://arxiv.org/abs/2201.08239 (2022).
- Not checked yetcites [30] Jin, Z. et al. Can large language models infer causation from correlation? Preprint at https://arxiv.org/abs/2306.05836 (2023).
- Not checked yetcites [34] Glaese, A. et al. Improving alignment of dialogue agents via targeted human judgements. Preprint at https://arxiv.org/abs/2209.14375 (2022).
- Not checked yetcites [35] Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C. & Szpektor, I. Trueteacher: Learning factual consistency evaluation with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2053–2070 (Association for Computational Linguistics, 2023). source
- Not checked yetcites [36] Manakul, P., Liusie, A. & Gales, M. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 9004–9017 (Association for Computational Linguistics, 2023). source
- Not checked yetcites [37] Laban, P. et al. LLMs as factual reasoners: insights from existing benchmarks and beyond. Preprint at https://arxiv.org/abs/2305.14540 (2023).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“This assessment requires examining the presence of true-causal relations among generated words30, which must be supported by evidence from reliable reference sources7,12.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Accuracy
What the sources say
- Not checked yetcites [30] Jin, Z. et al. Can large language models infer causation from correlation? Preprint at https://arxiv.org/abs/2306.05836 (2023).
- Not checked yetcites [7] Thoppilan, R. et al. LaMDA: language models for dialog applications. Preprint at https://arxiv.org/abs/2201.08239 (2022).
- Not checked yetcites [12] Peng, B. et al. GODEL: large-scale pre-training for goal-directed dialog. Preprint at https://arxiv.org/abs/2206.11309 (2022).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The proposed metrics demonstrate both within-category and between-category associations, with the potential for negative or positive correlations among them.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Metrics association
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, the number of parameters in a language model can impact accuracy, trustworthiness, and empathy metrics.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Metrics association
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As the number of tokens increases, the memory and computation needed also increase63, leading to higher latency and reduced usability.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Performance
What the sources say
- Not checked yetcites [63] Hoffmann, J. et al. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (eds Oh, A. H. et al.) (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022).
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, within the accuracy metrics category, up-to-dateness and groundedness show a positive correlation, as ensuring the chatbot utilizes the most recent and valid information enhances the factual accuracy of answers, thereby increasing groundedness.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Metrics association
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, a low parameter count can limit the model’s knowledge acquisition and influence the values of these metrics.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Metrics association
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Various automatic and human-based evaluation methods can quantify each metric, and the selection of evaluation methods significantly impacts metric scores.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Challenges in evaluating healthcare chatbots > Evaluation methods
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Healthcare applications are life-critical.
“Due to the life-critical nature of healthcare applications, using conversational models necessitates establishing a unified and comprehensive set of foundation metrics1 that enable a meticulous evaluation of the models’ performance, capabilities, identification of potential errors, and implementation of effective feedback mechanisms.”
The paper relies on this without support.
Emotional bonds play a vital role in physician–patient communications.
“Emotional bonds play a vital role in physician–patient communications, but they are often ignored during the development and evaluation of chatbots.”
The paper relies on this without support.
In the healthcare domain, safety and currentness of information are paramount.
“In particular, in the healthcare domain, where safety and currentness of information are paramount, hallucinations pose a significant concern.”
The paper relies on this without support.
There is scarcity of data and information across various medical domains and categories.
“In the context of healthcare, the significance of the generalization metric becomes pronounced due to the scarcity of data and information across various medical domains and categories.”
The paper relies on this without support.
Human inquiries may not precisely align with their underlying issues or intent.
“In healthcare chatbots, where human inquiries may not precisely align with their underlying issues or intent, robustness assumes paramount importance.”
The paper relies on this without support.
The desired level of interpretability for a generated text may vary between a patient and a nurse.
“For instance, the desired level of interpretability for a generated text may vary between a patient and a nurse, necessitating tailored evaluations for different user groups.”
The paper relies on this without support.
Empathy plays a crucial role in building trust between users and chatbots.
“Empathy also plays a crucial role in building trust between users and chatbots.”
The paper relies on this without support.
Unempathetic responses can erode trust and credibility in the system.
“Unempathetic responses can erode trust and credibility in the system, as users may feel unheard, misunderstood, or invalidated.”
The paper relies on this without support.
Low latency ensures prompt and efficient communication, enabling users to obtain timely responses.
“Low latency ensures prompt and efficient communication, enabling users to obtain timely responses.”
The paper relies on this without support.
Performance metrics may remain invariant concerning the three confounding variables (user type, domain type, and task type).
“It is important to note that performance metrics may remain invariant concerning the three confounding variables (user type, domain type, and task type).”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The application gap is attributed to surface-form similarity and lexicographic analytic restrictions41.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [41]
In the citing paper
“Medical image reports assist in differential diagnosis and thus are characterized by complex paraphrasing, high token counts (>100), and numerous negative descriptions, which conflict with common metric evaluation contexts24,41.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [41]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
“OpenAI Evals108”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > References
“108. Aryan, A., Nain, A. K., McMahon, A., Meyer, L. A. & Sahota, H. S. The costly dilemma: are large language models the pay-day loans of machine learning?”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
“performance metrics are evaluated based on the three confounding variables.”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots > Performance
“It is important to note that performance metrics may remain invariant concerning the three confounding variables (user type, domain type, and task type).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
“WinoGender13”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > References
“13. Touvron, H. et al. LLaMA: open and efficient foundation language models.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > Essential metrics for evaluating healthcare chatbots
“BigBench112”
§ Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI > References
“112. Ghazal, A. et al. Bigbench: towards an industry standard benchmark for big data analytics.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“…93 TruthfulQA, 94 CivilComments, 95 BOLD, 96 BBQ 97 Trustworthiness Privacy Evaluatingthemodel’suseofsensitiveuserinformation Lackofprivacy DP-SGD 98,99 Bias Lackofpersonzalitionandtoxicity CrowS-Pairs, 100 WinoGender, 13 BBQ, 97 TruthfulQA, 94 RealToxicityPrompts, 101 CivilComments 102”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“Popular LLMs, such as GPT-4, Llama, and BERT, often require large memory capacity, 13,24,140–142 making it challenging to run them on devices with limited memory, such as embedded systems, laptops, and mobile phones.”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 2×
“In the following, we briefly outline these evaluation methods.”
BackgroundMethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“In the following, we briefly outline these evaluation methods.”
BackgroundMethodA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 2×
“Currently, state-of-the-art conversational models are predominantly assessed and compared based on language-specific perspectives 8 and surface-form similarity 8 using intrinsic metrics such as Bilingual Evaluation Understudy (BLEU) 9 and Recall-oriented Understudy for Gisting Evaluation (ROUGE).”
BackgroundSelf-citationGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
SuperGlue: Learning Feature Matching With Graph Neural Networks
Sarlin, P.-E., DeTone, D., Malisiewicz, T., & Rabinovich, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00499 · Cited by 3,234
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Natural language processing: state of the art, current trends and challenges
Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13428-4 · Cited by 1,823 · cited 1×
“These metrics lack the capability to capture essential elements such as semantics, 29,46 context, 46,47 distant dependencies, 48,49 semantically-critical ordering change, 47 and human perspectives, particularly in real-world scenarios.”
BackgroundCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207
Dave, T., Athaluri, S. A., & Singh, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1169595 · Cited by 1,204 · cited 1×
“However, the existing evaluation metrics introduced and employed for assessing healthcare chatbots 2–4 exhibit two significant gaps that warrant careful attention.”
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150 · cited 1×
“17 Furthermore, the current methods fail to address the issue of hallucination , wherein chat-bots generate misleading or inaccurate information.”
BackgroundDIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.30 · Cited by 1,076 · cited 1×
“They can ensure the generated answer is grammatically accurate and pertinent to the questions.”
BackgroundBoolQ: Exploring the Surprising Difficulty of Natural Yes/No QuestionsPreprint
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10044 · Cited by 1,070
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S. B., & Zhang, Y. (2023) · Cureus · doi:10.7759/cureus.40895 · Cited by 784
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
The design and implementation of Xiaoice, an empathetic social chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020) · Computational Linguistics · doi:10.1162/coli_a_00368 · Cited by 777 · cited 1×
“95,102,131,132 Empa-thy also plays a crucial role in building trust between users and chatbots.”
BackgroundExtracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
QuAC: Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · cited 1×
“7,12 118 To address this, groundedness leverages relevant factual information, promoting sound reasoning and staying up-to-date ensuring validity.”
BackgroundTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614
Self-citationThe NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
Xu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571 · cited 1×
“However, the existing evaluation metrics introduced and employed for assessing healthcare chatbots 2–4 exhibit two significant gaps that warrant careful attention.”
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548 · cited 1×
“Bias evaluation 128 can be accomplished by employing an agreement index, based on the frequency of a healthcare chatbot agreeing with stereotype statements.”
BackgroundLi, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512
MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training DataPreprint
Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Figueroa, A., Löser, A., Truhn, D., & Bressem, K. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08247 · Cited by 505 · cited 1×
“…Lackofresilienceandvalidity GLUE, 71 CoQA, 72 LAMBADA, 1 TriviaQA, 73 ANLI, 74 MNLI, 75 SQUAD 76 Generalization Assessingchatbot’sperformanceonunfamiliartasks Overfitting,limitedtransferability,andlackofvalidity TyDiQA, 77 PromptBench, 78 AdvGLUE, 74 TextFlint, 79 DDXPlus, 74 MGSM 80”
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 9×
“Gaugingtheresilienceofchatbottoanydisruptions Lackofresilienceandvalidity GLUE, 71 CoQA, 72 LAMBADA, 1 TriviaQA, 73 ANLI, 74 MNLI, 75 SQUAD 76 Generalization Assessingchatbot’sperformanceonunfamiliartasks Overfitting,limitedtransferability,andlackofvalidity TyDiQA, 77 PromptBench, 78 AdvGLUE, 74 TextFlint, 79 DDXPlus,…”
BackgroundMethodInfluentialCrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.154 · Cited by 446
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · cited 1×
“In the following, we briefly outline these evaluation methods.”
MethodParlAI: A Dialog Research Software Platform
Miller, A., Feng, W., Batra, D., Bordes, A., Fisch, A., Lu, J., Parikh, D., & Weston, J. (2017) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d17-2014 · Cited by 386
Emotional Support from AI Chatbots: Should a Supportive Partner Self-Disclose or Not?
Meng, J., & Dai, Y. H. (2021) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmab005 · Cited by 360 · cited 1×
“133 The Health Literacy metric assesses the model’s capability to communicate health-related information in a manner under-standable to individuals with varying levels of health knowledge.”
BackgroundGhazal, A., Rabl, T., Hu, M., Raab, F., Poess, M., Crolotte, A., & Jacobsen, H. (2013) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2463676.2463712 · Cited by 343
A generalist vision-language foundation model for diverse biomedical tasks
Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03185-2 · Cited by 333
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331
Large Language Models Are Human-Level Prompt Engineers
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022) · arXiv · doi:10.48550/arxiv.2211.01910 · Cited by 330 · cited 1×
“Prompt engineering 146 significantly impacts the responses generated by healthcare chatbots, and the choice of prompt technique plays a pivotal role in achieving improved answers.”
BackgroundMEGA: Multilingual Evaluation of Generative AI
Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A. U., Ganu, T., Segal, S., Ahmed, M., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.258 · Cited by 321 · cited 1×
“No-tably, while recent studies 31,77,132,149 have introduced various evaluation frameworks, it is important to recognize that these may not fully cater to the specific needs of healthcare chatbots.”
BackgroundMeasuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306
Evaluation Metrics and Evaluation
Dalianis, H. (2018) · Springer eBooks · doi:10.1007/978-3-319-78503-5_6 · Cited by 305
Demographic dialectal variation in social media: a case study of African-American English
Blodgett, S. L., Green, L., & O'Connor, B. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1120 · Cited by 291
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291
Analyzing Leakage of Personally Identifiable Information in Language Models
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., & Zanella-Béguelin, S. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179300 · Cited by 269
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244
Evaluating the Performance of Large Language Models on GAOKAO BenchmarkPreprint
Zhang, X., Li, C.-Y., Zong, Y., Ying, Z., He, L., Qiu, X.-P., Sun, T., Li, P., Meng, S.-Q., Zheng, Y.-J., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2305.12474 · Cited by 236
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235 · cited 1×
“These metrics lack the capability to capture essential elements such as semantics, 29,46 context, 46,47 distant dependencies, 48,49 semantically-critical ordering change, 47 and human perspectives, particularly in real-world scenarios.”
BackgroundHariri, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02017 · Cited by 209
Huang, H., Tang, T., Zhang, D., Zhao, X., Song, T., Xia, Y., & Wei, F. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.826 · Cited by 198
BBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186
AI Chatbots, Health Privacy, and Challenges to HIPAA Compliance
Marks, M., & Haupt, C. E. (2023) · JAMA · doi:10.1001/jama.2023.9458 · Cited by 174
Social chemistry 101: Learning to reason about social and moral norms
Forbes, M., Hwang, J. D., Shwartz, V., Sap, M., & Choi, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.48 · Cited by 169
End-to-End Knowledge-Routed Relational Dialogue System for Automatic Diagnosis
Xu, L., Zhou, Q., Гонг, K., Liang, X., Tang, J., & Lin, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017346 · Cited by 166
Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity
Zhuo, T. Y., Huang, Y., Chen, C., & Xing, Z. (2023) · arXiv · doi:10.48550/arxiv.2301.12867 · Cited by 164
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N. Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.04528 · Cited by 153
Self-citationPromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N., et al. (2023) · Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis · doi:10.1145/3689217.3690621 · Cited by 151
Self-citationWhen to Make Exceptions: Exploring Language Models as Accounts of Human Moral JudgmentPreprint
Jin, Z., Levine, S., Gonzalez, F., Kamal, O., Sap, M., Sachan, M., Mihalcea, R., Tenenbaum, J. B., & Schölkopf, B. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.01478 · Cited by 147
ICE: the International Corpus of English
Greenbaum, S. (1991) · English Today · doi:10.1017/s0266078400005836 · Cited by 143
Can Large Language Models Infer Causation from Correlation?Preprint
Jin, Z., Liu, J., Lyu, Z., Poff, S., Sachan, M., Mihalcea, R. F., Diab, M., & Schölkopf, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.05836 · Cited by 143
Sources of Hallucination by Large Language Models on Inference Tasks
McKenna, N., Li, T., Cheng, L., Hosseini, M., Johnson, M., & Steedman, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.182 · Cited by 142
Large Language Models are not Fair Evaluators
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.511 · Cited by 138
Self-citationImproving alignment of dialogue agents via targeted human judgements
Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14375 · Cited by 134
A Systematic Literature Review of Information Security in Chatbots
Jing, Y., Chen, Y., Por, L. Y., & Ku, C. S. (2023) · Applied Sciences · doi:10.3390/app13116355 · Cited by 128 · cited 1×
“124,125 The overarching goal is to ensure confidentiality, integrity, and availability of the information and systems in question.”
BackgroundOn the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.387 · Cited by 121
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Welivita, A., & Pu, P. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.429 · Cited by 110 · cited 1×
“95,102,131,132 Empa-thy also plays a crucial role in building trust between users and chatbots.”
BackgroundDecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.11698 · Cited by 110
Recurrent Memory Networks for Language Modeling
Tran, K. M., Bisazza, A., & Monz, C. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1036 · Cited by 104 · cited 4×
“BART, 22 T5,GPT-2, 25 BiomedGPT 26 Perplexity 20,27 Likelihoodofthemodelgeneratingthereferencetext.”
InfluentialEvaluating the logical reasoning ability of chatgpt and GPT-4
Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.03439 · Cited by 100
Reddy, S. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101304 · Cited by 99 · cited 1×
“No-tably, while recent studies 31,77,132,149 have introduced various evaluation frameworks, it is important to recognize that these may not fully cater to the specific needs of healthcare chatbots.”
BackgroundTextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing
Wang, X., Liu, Q., Gui, T., Zhang, Q., Zou, Y., Zhou, X., Ye, J., Zhang, Y., Zheng, R., Pang, Z., et al. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.41 · Cited by 92
Security, privacy, and healthcare-related conversational agents: a scoping review
May, R., & Denecke, K. (2021) · Informatics for Health and Social Care · doi:10.1080/17538157.2021.1983578 · Cited by 91 · cited 1×
“124,125 The overarching goal is to ensure confidentiality, integrity, and availability of the information and systems in question.”
BackgroundThe LAMBADA dataset: Word prediction requiring a broad discourse context
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., & Fernandez, R. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1144 · Cited by 87 · cited 1×
“…nature of healthcare applications, us-ing conversational models necessitates establishing a unified and comprehensive set of foundation metrics 1 that enable a meticulous evaluation of the models’ performance, capabilities, identification of potential errors, and implementation of e ff ective…”
BackgroundOn the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective
Wang, J., Hu, X., Hou, W., Chen, H., Zheng, R., Wang, Y., Yang, L., Huang, H., Ye, W., Geng, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.12095 · Cited by 86
Self-citationLeveraging GPT-4 for Automatic Translation Post-Editing
Raunak, V., Sharaf, A., Wang, Y., Awadalla, H. H., & Menezes, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.804 · Cited by 84
Self-citationResnik, P., & Lin, J. (2010) · doi:10.1002/9781444324044.ch11 · Cited by 83 · cited 3×
“Intrinsic evaluation metrics measure the proficiency of a language model in generating coherent and meaningful sentences relying on language rules and patterns.”
BackgroundInfluentialKonda, P., Das, S., C., P. S. G., et al. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/3007263.3007314 · Cited by 80
Fairness in Machine Learning for Healthcare
Ahmad, M. A., Patel, A., Eckert, C., Kumar, V., & Teredesai, A. M. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3406461 · Cited by 80 · cited 1×
“135,136 Fairness and bias are two related but distinct concepts in the context of healthcare chatbots.”
BackgroundA Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78
Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning
Peng, B., Li, X., Gao, J., Liu, J., Chen, Y.-N., & Wong, K. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461918 · Cited by 76
Psychological foundations of explainability and interpretability in artificial intelligence
Broniatowski, D. A. (2021) · doi:10.6028/nist.ir.8367 · Cited by 76 · cited 1×
“130”
Chung, J., Kamar, E., & Amershi, S. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.34 · Cited by 72 · cited 1×
“For example, adjusting the beam search parameter 147 can impact the safety level of the chatbot’s answers, and similar e ff ects apply to other model parameters like temperature, 148 which can influence specific metric scores.”
BackgroundLSTM Networks Can Perform Dynamic Counting
Süzgün, M., Belinkov, Y., Shieber, S. M., & Gehrmann, S. (2019) · Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges · doi:10.18653/v1/w19-3905 · Cited by 69
Yuan, X., Zhou, J., Shi, Z., Lü, C., & Huang, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5456 · Cited by 66
AR-LSAT: Investigating Analytical Reasoning of TextPreprint
Zhong, W., Wang, S., Tang, D., Xu, Z., Guo, D., Wang, J., Yin, J. P., Zhou, M., & Duan, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06598 · Cited by 56
A global analysis of metrics used for measuring performance in natural language processing
Blagec, K., Dorffner, G., Moradi, M., Ott, S., & Samwald, M. (2022) · doi:10.18653/v1/2022.nlppower-1.6 · Cited by 54
Silfen, E. (2006) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2006.02.005 · Cited by 44 · cited 1×
“10 For this reason, they inadequately capture vital aspects like semantic nuances, contextual relevance, long-range dependencies, changes in critical semantic ordering, and human-centric perspectives, 11 thereby limiting their e ff ectiveness in evaluating healthcare chatbots.”
BackgroundBajaj, P., Xiong, C., Ke, G., Liu, X., He, D., Tiwary, S., Liu, T., Bennett, P. N., Song, X., & Gao, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.06644 · Cited by 41
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. J. F. (2023) · arXiv · doi:10.48550/arxiv.2303.08896 · Cited by 38
Benefits, Pitfalls, and Potential Bias in Health Care AI
Hague, D. C. (2019) · North Carolina Medical Journal · doi:10.18043/ncm.80.4.219 · Cited by 36 · cited 1×
“137,138 For instance, a model trained on an imbalanced dataset, with dominant samples from white males and limited samples from Hispanic females, might exhibit bias due to the imbalanced training dataset.”
BackgroundClinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
Toma, A., Lawler, P. R., Ba, J., Krishnan, R. G., Rubin, B. B., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2305.12031 · Cited by 36
GODEL: Large-Scale Pre-Training for Goal-Directed DialogPreprint
Peng, B., Galley, M., He, P., Brockett, C., Lidén, L., Nouri, E., Zhou, Y., Dolan, B., & Gao, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.11309 · Cited by 34 · cited 2×
“…Lackofresilienceandvalidity GLUE, 71 CoQA, 72 LAMBADA, 1 TriviaQA, 73 ANLI, 74 MNLI, 75 SQUAD 76 Generalization Assessingchatbot’sperformanceonunfamiliartasks Overfitting,limitedtransferability,andlackofvalidity TyDiQA, 77 PromptBench, 78 AdvGLUE, 74 TextFlint, 79 DDXPlus, 74 MGSM 80”
MethodPandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
Wang, Y., Yu, Z., Zeng, Z., Yang, L., Wang, C., Chen, H., Jiang, C., Xie, R., Wang, J., Xie, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05087 · Cited by 34
Self-citationLarge Language Models are not Fair Evaluators
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., & Sui, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.17926 · Cited by 34
Self-citationTrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., & Szpektor, I. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.127 · Cited by 33
Pan, A., Chan, J. S., Zou, A., Li, N., Basart, S., Woodside, T., Ng, J., Zhang, H., Emmons, S., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2304.03279 · Cited by 32
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Yu, J., Wang, X., Tu, S., Cao, S., Zhang-Li, D., Lv, X., Peng, H., Yao, Z., Zhang, X., Li, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.09296 · Cited by 29
BLiMP: the benchmark of linguistic minimal pairs for English
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28
Capturing Semantics for Imputation with Pre-trained Language Models
Mei, Y., Song, S., Fang, C., Yang, H., Fang, J., & Long, J. (2021) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00013 · Cited by 21
Huang, F., Kwak, H., & An, J. (2023) · ACM Web Conference (WWW) · doi:10.1145/3543873.3587320 · Cited by 20
Ilicki, J. (2023) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2023.03.006 · Cited by 20 · cited 2×
“No-tably, while recent studies 31,77,132,149 have introduced various evaluation frameworks, it is important to recognize that these may not fully cater to the specific needs of healthcare chatbots.”
BackgroundLIME: Learning Inductive Bias for Primitives of Mathematical ReasoningPreprint
Wu, Y., Rabe, M. N., Li, W., Ba, J. L., Grosse, R., & Szegedy, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.06223 · Cited by 18
Do dependency parsing metrics correlate with human judgments?
Plank, B., Alonso, H. M., Agić, Ż., et al. (2015) · doi:10.18653/v1/k15-1033 · Cited by 15 · cited 1×
“LIMA, 28 BART, 22 Meena 9 BERTScore 20,29 Createsasimilaritymatrixbetweenreferenceandgeneratedtextandcalculatestheweightedsumofmaximumsimilarityinthematrix.”
The five Is: Key principles for interpretable and safe conversational AI
Wahde, M., & Virgolin, M. (2021) · doi:10.1145/3507623.3507632 · Cited by 13 · cited 1×
“129 This evaluation allows users and healthcare professionals to understand the reasoning behind the chatbot’s recommendations or actions.”
BackgroundLLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond
Laban, P., Kryściński, W., Agarwal, D., Fabbri, A. R., Xiong, C., Joty, S., & Wu, C.-S. (2023) · arXiv · doi:10.48550/arxiv.2305.14540 · Cited by 13 · cited 1×
“…Lackofresilienceandvalidity GLUE, 71 CoQA, 72 LAMBADA, 1 TriviaQA, 73 ANLI, 74 MNLI, 75 SQUAD 76 Generalization Assessingchatbot’sperformanceonunfamiliartasks Overfitting,limitedtransferability,andlackofvalidity TyDiQA, 77 PromptBench, 78 AdvGLUE, 74 TextFlint, 79 DDXPlus, 74 MGSM 80”
Definition modeling: literature review and dataset analysis
Gardner, N., Khan, H., & Hung, C. (2022) · Applied Computing and Intelligence · doi:10.3934/aci.2022005 · Cited by 11
Bring Your Own Data! Self-Supervised Evaluation for Large Language ModelsPreprint
Jain, N., Saifullah, K., Wen, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.13651 · Cited by 11
Oniani, D., Sreekumar, S., DeAlmeida, R., DeAlmeida, D., Hui, V. C. C., Lee, Y. J., Zhang, Y., Zhou, L., & Wang, Y. (2022) · PubMed Central · doi:10.48550/arxiv.2209.06723 · Cited by 10 · cited 1×
“134 For instance, “pneumonia is hazardous” might be challenging for a general audience, while “lung disease is danger-ous” could be a more accessible option for people with diverse health knowledge.”
BackgroundSelf-citationMeasuring Personalization, Embodiment, and Congruence in Online Learning: A Validation Study
Cook, D. A., & Skrupky, L. P. (2022) · Academic Medicine · doi:10.1097/acm.0000000000005088 · Cited by 10 · cited 1×
“Evaluatingthefactualvalidityofgeneratedresponses Out-to-dateness,lackofreasoning,lackofvalidity,andhallucination LSAT, 83 Dyck, 84 Syntheticreasoning 85 WikiFact, 64 bAbI, 86 Entitymatching, 87 Dataimputation, 88 HumanEval, 89 APPS, 90 MATH, 91 GSM8K 92”
Jethani, N., Jones, S., Genes, N. G., Major, V. J., Jaffe, I. S., Cardillo, A. B., Heilenbach, N., Ali, N. F., Bonanni, L. J., Clayburn, A. J., et al. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4520434 · Cited by 8
iEval: Interactive Evaluation Framework for Open-Domain Empathetic Chatbots
Svikhnushina, E., Filippova, A., & Pu, P. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.41 · Cited by 6
Ahmad, M. A., Eckert, C., Allen, C. M. B., Kumar, V., Hu, J., & Teredesai, A. M. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00104 · Cited by 5 · cited 1×
“Evaluatingthefactualvalidityofgeneratedresponses Out-to-dateness,lackofreasoning,lackofvalidity,andhallucination LSAT, 83 Dyck, 84 Syntheticreasoning 85 WikiFact, 64 bAbI, 86 Entitymatching, 87 Dataimputation, 88 HumanEval, 89 APPS, 90 MATH, 91 GSM8K 92”
An Adaptive Framework for Conversational Question Answering
Su, L., Guo, J., Fan, Y., Lan, Y., Zhang, R., & Cheng, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.330110041 · Cited by 4
Intrinsic and Extrinsic Automatic Evaluation Strategies for Paraphrase Generation Systems
Hailu, T. T., Yu, J., & Fantaye, T. G. (2020) · Journal of Computer and Communications · doi:10.4236/jcc.2020.82001 · Cited by 4 · cited 1×
“Intrinsic metrics are employed to address linguistic and relevance problems of healthcare chatbots in each single conversation between user and the chatbot.”
MethodShichel, Y., Kalech, M., & Tsur, O. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.139 · Cited by 4 · cited 1×
“112,113 In the domain of healthcare chat-bots, generating concise responses becomes crucial to avoid verbosity or needless repetition, as such shortcomings can lead to misunderstanding or misinterpretation of context.”
BackgroundOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Research works printed but not linked: 18
- Resnik, P. et al. Using intrinsic and extrinsic metrics to evaluate accuracy and facilitation in computer-assisted coding. In Perspectives in Health Information Management Computer Assisted Coding Conference Proceedings (The American Health Information Management Association (AHIMA), 2006).
- Zhang*, T., Kishore*, V., Wu*, F., Weinberger, K. Q. & Artzi, Y. Bertscore: evaluating text generation with bert. In International Conference on Learning Representations. (International Conference on Learning Representations (ICLR), 2020).
- Napoles, C., Van Durme, B. & Callison-Burch, C. Evaluating sentence compression: Pitfalls and suggested remedies. In Proceedings of the Workshop on Monolingual Text-To-Text Generation, 91–97 (Association for Computational Linguistics, 2011). https://aclanthology.org/volumes/W11-16/.
- Zhou, C. et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems 36 (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024).
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186 (Association for Computational Linguistics, Minneapolis, 2019).
- Hoffmann, J. et al. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (eds Oh, A. H. et al.) (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022).
- Text REtrieval Conference (TREC) Home Page — trec.nist.gov. https://trec.nist.gov/. [Accessed 28-07-2023]
- Ge, Y. et al. Openagi: When llm meets domain experts. Advances in Neural Information Processing Systems 36 (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024).
- Wei, J. et al. Finetuned language models are zero-shot learners. In International Conference on Learning Representations (ICLR, 2022).
- Radford, A. et al. Language models are unsupervised multitask learners. OpenAI Blog 1, 9 (2019).
- Zhang, K. et al. BiomedGPT: a unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. Preprint at https://arxiv.org/abs/2305.17100 (2023). · arXiv
- Banerjee, S. & Lavie, A. Meteor: an automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 65–72 (Association for Computational Linguistics, 2005).
- Peng, Y.-S., Tang, K.-F., Lin, H.-T. & Chang, E. Refuel: exploring sparse features in deep reinforcement learning for fast disease diagnosis. Adv. Neural Inf. Process. Syst. 31, (2018).
- Hermann, K. M. et al. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, 1693–1701 (MIT Press, Cambridge, 2015).
- Aryan, A., Nain, A. K., McMahon, A., Meyer, L. A. & Sahota, H. S. The costly dilemma: are large language models the pay-day loans of machine learning? https://abiaryan.com/assets/EMNLP%20Submission_Non-Anon.pdf. (2023).
- Yuan, L. et al. Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and LLMs evaluations. Advances in Neural Information Processing Systems 36 (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024).
- Hendrycks, D. et al. Measuring coding challenge competence with APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2021).
- Maas, A. et al. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 142–150 (Association for Computational Linguistics, 2011).
Cited by
Sorted by citations
Generative artificial intelligence in medicine
Teo, Z. L., Thirunavukarasu, A. J., Elangovan, K., Cheng, H., Moova, P., Soetikno, B., Nielsen, C., Pollreisz, A., Ting, D. S. J., Morris, R. J. T., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03983-2 · Cited by 119
“For a comprehensive overview of foundational metrics for clinical GAI assessment, Abbasian et al. have provided a summary grouped under the headings of accuracy, trustworthiness, empathy and performance 150 .”
Use of Generative AI for Mental Health Advice Among US Adolescents and Young Adults
McBain, R., Bozick, R., Diliberti, M., Zhang, L.-A., Zhang, F., Burnett, A., Kofner, A., Rader, B., Breslau, J., Stein, B. D., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.42281 · Cited by 128
Jacob, C., Brasier, N., Laurenzi, E., Heuss, S., Mougiakakou, S. G., Çöltekin, A., & Peter, M. K. (2024) · Journal of Medical Internet Research · doi:10.2196/67485 · Cited by 125
“Trust is built through factors such as validation, transparency, safety, privacy, and interpretability of the AI tool [62].”
InfluentialTwo Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
Tseng, Y.-M., Huang, Y.-C., Hsiao, T.-Y., Chen, W.-L., Huang, C.-W., Meng, Y., & Chen, Y.-N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.969 · Cited by 87
Prompt Engineering in Education: A Systematic Review of Approaches and Educational Applications
Qian, Y. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251365189 · Cited by 134
Metadata
- Authors
- Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., Gevaert, O., Li, L.-J., Jain, R., & Rahmani, A. M.
- Affiliations
- University of California, Irvine (US)
- University of Pittsburgh (US)
- Stanford University (US)
- National Institute of Standards and Technology (US)
- Corresponding author
- Abbasian, Mahyar
- Khatibi, Elahe
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2024-03-29
- Volume
- 7
- Issue
- 1
- Pages
- 82-82
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC10980701
- OpenAlex
- W4393335480
- Semantic Scholar
- 8d1211cbbdf161feaae2c87832ede063346e76dd
- Cited by
- 213
- References
- 143
- Citations by year
- 2026: 92 · 2025: 83 · 2024: 38
- FWCI
- 4.81
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Institute of Standards and Technology
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- generative AI in healthcare, healthcare chatbots, evaluation metrics, large language models, chatbots, patient-centered care, user trust, empathy, emotional support, personalized health recommendations, confounding factors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
