The DOI matches its Crossref record; 2 of the 7 references it cites were found in a registry or an index, and 5 are not checked yet.
MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation
Open accessAbstract
Junqing He, Liang Zhu, Rui Wang, Xi Wang, Gholamreza Haffari, Jiaxing Zhang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Benchmarks such as MADial-Bench provide practical evaluation protocols for memory recall and retrieval tasks, essential for measuring correctness and privacy of multi-user memory systems.
“Benchmarks such as MADial-Bench provide practical evaluation protocols for memory recall and retrieval tasks, essential for measuring the correctness and privacy of multi-user memory systems [22].”
From Memory Fabric for Conversational AI Agents: Enabling Shared and Persistent Memory Across Users (2025) · cites this paper as [22]
Sections 3.3 and 4.1.2 describe separate memory-recall and retrieval evaluations, including retrieval-accuracy metrics, supporting the claim that MADial-Bench provides an evaluation protocol for those tasks. The paper’s Introduction and Conclusion frame it as a benchmark for memory-augmented dialogue systems, but the cited passages do not establish a protocol for measuring privacy or for evaluating multi-user memory systems. (LLM, conf=0.90)
Experimental — accuracy not yet measuredMADial-Bench measures a model's ability to retrieve past facts, correlate entities across different sessions with minimal hallucinations by constantly updating its memory, unlike traditional metrics focused on response quality alone.
“Unlike traditional metrics focused on response quality alone, this benchmark measures a model's ability to retrieve past facts, correlate entities across different sessions with minimal hallucinations by constantly updating its memory.”
From Memory Fabric for Conversational AI Agents: Enabling Shared and Persistent Memory Across Users (2025) · cites this paper as [22]
The Introduction describes MADial-Bench as evaluating memory retrieval, recognition, and injection alongside generated responses; Section 3.1 includes passive recall of past facts and social relationships. These passages do not establish that it tests correlating entities across sessions, minimizing hallucinations, or constantly updating memory. The Introduction also does not characterize earlier metrics as focused on response quality alone. (LLM, conf=0.86)
Experimental — accuracy not yet measuredMADial-Bench was developed to evaluate dialogue systems enhanced with external or persistent memory.
“Benchmark for Memory-Augmented Dialogue Systems (MADial-Bench): MADial-Bench was developed to evaluate dialogue systems enhanced with external or persistent memory [22].”
From Memory Fabric for Conversational AI Agents: Enabling Shared and Persistent Memory Across Users (2025) · cites this paper as [22]
MADial-Bench introduces dimensions tailored to measure long term memory correctness across different architectures.
“Key contribution: The MADial benchmark introduces dimensions tailored to measure long term memory correctness across different architectures.”
From Memory Fabric for Conversational AI Agents: Enabling Shared and Persistent Memory Across Users (2025) · cites this paper as [22]
Updating long term memories is harder than retrieving them, as models keep both current and outdated information.
“Findings: Updating long term memories is harder than retrieving them as models keep both current and outdated information.”
From Memory Fabric for Conversational AI Agents: Enabling Shared and Persistent Memory Across Users (2025) · cites this paper as [22]
Section 5.1 reports that retrieving appropriate memories is difficult, and Section 5.2 reports difficulty selecting and using retrieved memories. Neither section compares updating long-term memories with retrieval or says models retain both current and outdated information. Searches for memory updating and obsolete or conflicting information found no passage addressing those points; page numbers were unavailable in our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measured
If a user changes their preference, the model sometimes retrieves outdated information based on past preferences.
“For example, if a user changes their preference, the model sometimes retrieves outdated information based on past preferences.”
From Memory Fabric for Conversational AI Agents: Enabling Shared and Persistent Memory Across Users (2025) · cites this paper as [22]
Section 3.3.2 describes selecting memories retrieved for a dialogue, and Section 5.2 reports weaker performance in a realistic retrieval setting. Neither passage says that users changed their preferences or that models retrieved outdated preferences. Searches for changed, conflicting, and outdated preferences found no passage addressing that example. (LLM, conf=0.73)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 32
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Anthropic. 2024. Claude 3.5 sonnet. Kimberly A Arditte Hall, Rudi De Raedt, Kiara R Tim- pano, and Jutta Joormann. 2018. Positive memory enhancement training for individuals with major de- pressive disorder. Cognitive Behaviour Therapy, 47(2):155-168. · arXiv
- Harry P Bahrick. 1970. Two-phase model for prompted recall. Psychological Review, 77(3):215.
- Douglas Bermingham, Robert D Hill, Dan Woltz, and Michael K Gardner. 2013. Cognitive strategy use and measured numeric ability in immediate-and long- term recall of everyday numeric information. PloS one, 8(3):e57999.
- Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. · arXiv
- Abhimanyu Dubey et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. · arXiv
- Haakon G Engen and Michael C Anderson. 2018. Mem- ory control: A fundamental mechanism of emotion regulation. Trends in Cognitive Sciences, 22(11):982- 995.
- Leonard N. Ezeh, Chukwuemeka A. F. Okoye, Chuk- wuemeka E. Etodike, and Cynthia C. Udeze. 2018. Theoretical correlation of ageing, mental activity and memory recall: Implication for security personnel. European Journal of Special Education Research, 0(0). · DOI
- Ruyi Gan, Ziwei Wu, Renliang Sun, Junyu Lu, Xi- aojun Wu, Dixiang Zhang, Kunhao Pan, Junqing He, Yuanhe Tian, Ping Yang, Qi Yang, Hao Wang, Jiaxing Zhang, and Yan Song. 2024. Ziya2: Data-centric learning is all llms need. Preprint, arXiv:2311.03301. · arXiv
- Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.00027. · arXiv
- Michael Günther, Jackmin Ong, Isabelle Mohr, Alaed- dine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. 2023. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. Preprint, arXiv:2310.19923. · arXiv
- Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanu- jan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. 2022. Matryoshka repre- sentation learning. Advances in Neural Information Processing Systems, 35:30233-30249.
- Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281. · arXiv
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Lei Liu, Xiaoyan Yang, Yue Shen, Binbin Hu, Zhiqiang Zhang, Jinjie Gu, and Guannan Zhang. 2023. Think-in-memory: Recalling and post-thinking en- able llms with long-term memory. Preprint, arXiv:2311.08719. · arXiv
- Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meet- ing of the Association for memoryforconv Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 3469-3483.
- John D Mayer, David R Caruso, and Peter Salovey. 2016. The ability model of emotional intelligence: Principles and updates. Emotion review, 8(4):290- 300.
- Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Memgpt: Towards llms as operating systems. Preprint, arXiv:2310.08560. · arXiv
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
- Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5370-5381.
- Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M. Liu, Jinfeng Zhou, Alvionna S. Sunaryo, Juanzi Li, Tatia M. C. Lee, Rada Mihalcea, and Minlie Huang. 2024. Emobench: Evaluating the emotional intelligence of large language models. Preprint, arXiv:2402.12071. · arXiv
- Sahand Sabour, Chujie Zheng, and Minlie Huang. 2022. Cem: Commonsense-aware empathetic response gen- eration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11229- 11237.
- Eugen Tarnow. 2016. First direct evidence of two stages in free recall and three corresponding esti- mates of working memory capacity. arXiv preprint arXiv:1605.05685. · arXiv
- Zhiliang Tian, Yinliang Wang, Yiping Song, Chi Zhang, Dongkyu Lee, Yingxiu Zhao, Dongsheng Li, and Nevin L Zhang. 2022. Empathetic and emotion- ally positive conversation systems with an emotion- specific query-response memory. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6364-6376.
- James Tonks, W Huw Williams, Ian Frampton, Phil Yates, and Alan Slater. 2007. Assessing emotion recognition in 9-15-years olds: Preliminary analysis of abilities in reading emotion from faces, voices and eyes. Brain injury, 21(6):623-629.
- Endel Tulving and Donald M. Thomson. 1973. En- coding specificity and retrieval processes in episodic memory. Psychological Review, 80:352-373.
- Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. 2023. Aug- menting language models with long-term memory. Preprint, arXiv:2306.07174. · arXiv
- An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao- ran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Ke- qin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 techni- cal report. Preprint, arXiv:2407.10671. · arXiv
- Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. · arXiv
- Kai Zhang, Lizhi Qing, Yangyang Kang, and Xiaozhong Liu. 2024. Personalized llm response generation with parameterized memory injection. Preprint, arXiv:2404.03565. · arXiv
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
- Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2023. Memorybank: Enhancing large language models with long-term memory. Preprint, arXiv:2305.10250. · arXiv
- Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional chatting ma- chine: Emotional conversation generation with inter- nal and external memory. In Proceedings of the AAAI conference on artificial intelligence, volume 32. An example from the test set and LLM responses in setting 3 in Chinese. 对话信息: 历史事件P: 1.{'time': '2022-06-27', 'scene': '喜好类', 'emotion': '开心', 'event': '爷爷奶奶带Lisa一 起种花，感到非常开心。'}
Cited by
Sorted by citations
Memory in the Age of AI Agents
Hu, Y., Liu, S., Yue, Y., Zhang, G., Liu, B., Zhu, F., Lin, J., Guo, H., Dou, S., Xi, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.13564 · Cited by 389
Lee, J.-H., & Rew, J. (2025) · Applied Sciences · doi:10.3390/app15179775 · Cited by 9
From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
Luo, J., Tian, Y., Cao, C., Luo, Z., Lin, H., Li, K., Kong, C., Yang, R., & Ma, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.2069 · Cited by 3
“Furthermore, certain frameworks focus on emotional support and interactive scenarios to evaluate the model’s capacity for memory recall across proactive and passive paradigms (He et al., 2024b).”
Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues
Hu, C., Li, T., Gao, X., Chen, H., Bai, Y., Xu, D., Lin, T., Li, X., Han, Y., Pei, J., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817589 · Cited by 37
From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs
Wu, Y., Liang, S., Zhang, C., Wang, Y., Zhang, Y., Guo, H., Tang, R., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.15965 · Cited by 3
“…SCM [67], ChatDev [68], MetaAgents [69], S 3 [70], TradingGPT [71], Memolet [72], Synaptic Resonance [14], MemReasoner [73] Benchmark MADial-Bench [74], LOCOMO [75], MemDaily [76], ChMapData [77], MSC [78], MMRC [79], Ego4D [80], EgoLife [62], BABILong [81, 82] III PersonalParametricShort-Term”
Xie, Z., Cao, S., Liu, Z., Ye, Z., Niu, Z., Lu, C., Xu, T. B., Chen, E., Xu, Z., Hu, Y., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-demo.40 · Cited by 2
ENPMR-Bench: Benchmarking Proactive Memory Retrieval for Emotional Support Agents
Fu, X., Hu, Y., Ji, M., Li, H., Sun, Y., Zhao, W., Zhao, Y., & Qin, B. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.2080 · Cited by 1
Memory Fabric for Conversational AI Agents: Enabling Shared and Persistent Memory Across Users
Tiwari, A., & Gupta, V. (2025) · TechRxiv · doi:10.36227/techrxiv.176523350.08289935/v1 · Cited by 2
Graph-Enhanced Transformer Framework for Context-Sensitive English Skill Assessment
Shalini, A., Orosoo, M., Shanthi, W. G., Prema, S., Farhad, S., Muniyandy, E., & Dhivya, A. C. A. (2025) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2025.0161162
When Do Memories Break? Chain-of-Stage Diagnosis for LLM Memory Systems
Wang, C., Fu, Y., Zheng, Z., Qin, S., Cao, Z., & Liu, Z. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1042
Metadata
- Authors
- He, J., Zhu, L., Wang, R., Wang, X., Haffari, G., & Zhang, J.
- Affiliations
- Southern University of Science and Technology (CN)
- University of Sheffield (GB)
- Venue
- Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 9902-9921
- OpenAlex
- W4411119391
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1
- FWCI
- 1.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- dialogue evaluation, benchmark, real-world evaluation, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
