The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination
Open accessAbstract
The Generative Pre-trained Transformer(GPT) models, renowned for generating human-like text, occasionally produce “hallucinations” - outputs that diverge from human expectations. Current mitigation strategies for these GPT hallucinations largely rely on algorithmic automation, thereby overlooking the complexities of human judgment and cultural influence, particularly in fact interpretation. Addressing this issue, we have introduced a Culturally Sensitive Test that integrates language subjectivity, cultural nuances, and GPT idiosyncrasies. We have applied this test to five GPT models—OpenAI’s ChatGPT-3.5 and ChatGPT-4, Google’s Bard, Perplexity AI and TruthGPT - evaluating their responses to 70 questions across seven categories designed to provoke hallucinations. The evaluated models demonstrated varying performance, with controversial topics, those lacking clear scientific consensus and the brain teasers proving more susceptible to GPT hallucinations. Our study has paved the way for a nuanced assessment of GPT hallucinations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our review also found that only 6 of the 23 surveyed benchmark studies ( [7], [10], [31], [39], [44], [52]) were peerreviewed at the time of this article, reflecting the early-stage nature of research in this domain.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study demonstrated that LLMs were not robust to adversarial prompts, highlighting a significant gap in evaluating models' performance in real-world, context-specific scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study acknowledged the challenge of discerning genuine reasoning from technical optimization in LLMs, indicating that it primarily evaluated models based on their ability to follow instructions and generate outputs without deeply investigating whether these outputs resulted from true understanding or merely optimization strategies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“PromptBench [7]: The study did not establish a clear criterion for balancing helpfulness and harmlessness in adversarial prompt evaluation, focusing instead on the robustness of LLMs against adversarial prompts without directly addressing how these models balance being helpful and avoiding harmful outputs in real-world scenarios.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study encountered difficulties in efficiently scaling their benchmark framework, Prompt-Bench, across various large language models, indicating substantial technical challenges in installation and scalability.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study did not address the potential variability in implementing its adversarial prompt attacks across different models or laboratories, and it did not attempt to standardize or ensure uniform implementation methodologies for these prompts.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study required iterating over the entire dataset 100 times on average to generate one adversarial prompt, indicating a potentially prolonged evaluation process, which aligned with the mentioned inadequacy of slow iteration time in benchmarks.The study did not specifically address or propose solutions to reduce these iteration time, suggesting that the challenge of aligning rapid AI development”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study focused on creating adversarial prompts to test LLM robustness but did not address the integration of diverse cultural and ideological norms, which could affect standardized answers' validity across different cultures and ideologies.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
In the citing paper
“The study did not address the diversity of human curators and evaluators, focusing instead on generating adversarial prompts across multiple levels to test LLMs' robustness, which did not directly relate to or account for the variability in human interpretation and judgment based on cultural, religious, political, and academic or commercial backgrounds.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [7]
Citations
Cites
1 retracted
Zhang, J., Pu, J., Xue, J., Yang, M. Z., Xu, X., Wang, X., & Wang, F. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3256982 · Cited by 71
RetractedLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 2×
“Building upon Alan Turing’s seminal work “Computing Ma-chinery and Intelligence” in 1950 [51], we propose a modifica-tion to the original Turing Test for evaluating hallucinations in GPT models.”
MethodSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 4×
“While existing mitigation measures such as censorship or ambiguous responses can compromise AI systems’ accuracy and utility [11], [15], [31], they highlight the need for a more comprehensive, ethically informed assessment.”
BackgroundInfluentialGPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
Notes towards a description of social representations.
Moscovici, S. (1988) · European Journal of Social Psychology · doi:10.1002/ejsp.2420180303 · Cited by 2,003
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919
Research works printed but not linked: 3
- Clayton, 2023, Sam Altman: CEO of OpenAI calls for us to regulate artificial intelligence
- Smith, 2023, Hallucinations could blunt ChatGPTs success
- Nichols, 2023, Machines hacking machines—Turings legacy [Carter], Cyber-Human Systems, Space Technologies, and Threats
Cited by
Sorted by citations
Game changers: A generative AI prompt protocol to enhance human-AI knowledge co-construction
Robertson, J., Ferreira, C. C., Botha, E., & Oosthuizen, K. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.04.008 · Cited by 161
Generative AI Prompt Engineering for Educators: Practical Strategies
Park, J., & Choo, S. (2024) · Journal of Special Education Technology · doi:10.1177/01626434241298954 · Cited by 159
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102
“This technology has not only revolutionized the industry and academia, but has also reignited critical discussions concerning AI consciousness and its potential threats to humanity [9], [10], [11].”
BackgroundSelf-citationA Systematic Review of the Limitations and Associated Opportunities of ChatGPT
Cong‐Lem, N., Soyoof, A., & Tsering, D. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2344142 · Cited by 140
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104
“This lack of consideration results in benchmarks that wrongfully assume a uniform cognitive framework, failing to capture how LLMs handle linguistic variations across different cultures and languages [13].”
BackgroundInfluentialSelf-citationMcIntosh, T. R., Sušnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., Nowrozy, R., & Halgamuge, M. N. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103964 · Cited by 150
Self-citationAutomating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning
Sušnjak, T., Hwang, P., Reyes, N. H., Barczak, A. L. C., McIntosh, T. R., & Ranathunga, S. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3715964 · Cited by 118
Self-citationMcIntosh, T. R., Susnjak, T., Liu, T., Watters, P., & Halgamuge, M. N. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3377445 · Cited by 62
Self-citationInteligencia artificial generativa y educación
García‐Peñalvo, F. J. (2024) · Education in the Knowledge Society (EKS) · doi:10.14201/eks.31942 · Cited by 179
Identity, crimes, and law enforcement in the Metaverse
Qin, H. X., Wang, Y., & Hui, P. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04266-w · Cited by 84
Generative AI in education: ChatGPT-4 in evaluating students’ written responses
Jauhiainen, J. S., & Guerra, A. G. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2024.2422337 · Cited by 81
Park, S., & Nan, X. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02620-3 · Cited by 58
Fake News Detection with Large Language Models on the LIAR DatasetPreprint
Boissonneault, D., & Hensen, E. (2024) · Research Square · doi:10.21203/rs.3.rs-4465815/v1 · Cited by 49
Microsoft Copilot and Anthropic Claude AI in education and library service
Adetayo, A. J., Aborisade, M. O., & Sanni, B. A. (2024) · Library Hi Tech News · doi:10.1108/lhtn-01-2024-0002 · Cited by 45
Swindell, A., Greeley, L., Farag, A., & Verdone, B. (2024) · Online Learning · doi:10.24059/olj.v28i2.4438 · Cited by 45
A Reasoning and Value Alignment Test to Assess Advanced GPT Reasoning
Mcintosh, T. R., Liu, T., Susnjak, T., Watters, P., & Halgamuge, M. N. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3670691 · Cited by 42
Self-citationMcIntosh, T. R., Sušnjak, T., Liu, T., Watters, P., Ng, A., & Halgamuge, M. N. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3394392 · Cited by 41
Self-citationA novel large-language-model-driven framework for named entity recognition
Wang, Z., Chen, H., Xu, G., & Ren, M. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104054 · Cited by 40
Enhancing Accuracy in Large Language Models Through Dynamic Real-Time Information InjectionPreprint
Ouyang, Q., Wang, S., & Wang, B. (2023) · Preprints.org · doi:10.20944/preprints202312.1987.v1 · Cited by 36
Lee, Y. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13129-5 · Cited by 46
Jin, L., Shen, Z., Alhur, A., & Naeem, S. B. (2025) · Information Development · doi:10.1177/02666669251340954 · Cited by 41
McDonald, D., Papadopoulos, R., & Benningfield, L. (2024) · doi:10.36227/techrxiv.171665607.76504195/v1 · Cited by 34
Yang, X., Wang, Z., Wang, Q., Ke, W., Zhang, K., & Shi, J. (2024) · International Journal of Web Information Systems · doi:10.1108/ijwis-12-2023-0256 · Cited by 34
Franc, J. M., Hertelendy, A. J., Cheng, L. T. W., Hata, R., & Verde, M. (2024) · Journal of Medical Internet Research · doi:10.2196/55648 · Cited by 38
Kirchenbauer, J., & Barns, C. (2024) · doi:10.31219/osf.io/pv7r5 · Cited by 31
VaxBot-HPV: a GPT-based chatbot for answering HPV vaccine-related questions
Li, Y., Li, J., Li, M., Yu, E., Rhee, D., Amith, M. “., Tang, L., Savas, L. S., Cui, L., & Tao, C. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooaf005 · Cited by 29
Kumamoto, T., Yoshida, Y., & Fujima, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3719038/v1 · Cited by 28
Salminen, J. O., Jung, S., Medina, J., Aldous, K. K., Azem, J. Y., Akhtar, W., & Jansen, B. J. J. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3664690 · Cited by 27
Analyzing and Mitigating Cultural Hallucinations of Commercial Language Models in TurkishPreprint
Boztemir, Y., & Çalışkan, N. (2024) · doi:10.36227/techrxiv.171504678.80895811/v1 · Cited by 26
Kaate, I., Salminen, J. O., Jung, S., Xuan, T. T. T., Häyhänen, E., Azem, J. Y., & Jansen, B. J. J. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712160 · Cited by 40
Cost-efficient prompt engineering for unsupervised entity resolution in the product matching domain
Nananukul, N., Sisaengsuwanchai, K., & Kejriwal, M. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00159-8 · Cited by 36
Efficient Ransomware Detection via Portable Executable File Image Analysis By LLaMA-7bPreprint
Li, X., Zhu, T., & Zhang, W. (2023) · Research Square · doi:10.21203/rs.3.rs-3679775/v1 · Cited by 22
Wong, S. M., Leung, H.-F., & Wong, K. Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4063228/v1 · Cited by 23
Mardiansyah, K., & Surya, W. (2024) · Research Square · doi:10.21203/rs.3.rs-4005702/v1 · Cited by 23
McPhee, S., & Jerowsky, M. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1593278 · Cited by 40
Reducing Hallucinations in Large Language Models Through Contextual Position EncodingPreprint
Desrochers, S., Wilson, J., & Beauchesne, M. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/exjqb · Cited by 22
A Comparative Analysis to Evaluate Bias and Fairness Across Large Language Models with BenchmarksPreprint
陳文意, & 黃兆明 (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/mc762 · Cited by 23
盖振华, Tong, L., & Ge, Q. (2024) · doi:10.31219/osf.io/ctw8v · Cited by 19
Birillo, A., Artser, E., Potriasaeva, A., Vlasov, I., Dzialets, K., Golubev, Y., Герасимов, И. В., Keuning, H., & Bryksin, T. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699556 · Cited by 30
MisCaltral: Reducing Numeric Hallucinations of Mistral with Precision Numeric CalculationPreprint
Chen, S.-W., & Hsu, H.-J. (2023) · Research Square · doi:10.21203/rs.3.rs-3789011/v1 · Cited by 18
Implementing Automated Error Correction and Feedback Loops in Kimi, A Chinese Large Language ModelPreprint
Cheung, W.-L., & Luk, C.-Y. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/7vpxr · Cited by 18
Combining LoRA to GPT-Neo to Reduce Large Language Model HallucinationPreprint
Huang, S.-H., & Chen, C.-Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4515250/v1 · Cited by 17
Integrating Deep Learning with Symbolic Reasoning in TinyLlama for Accurate Information RetrievalPreprint
Xiong, X., & Zheng, M. (2024) · Research Square · doi:10.21203/rs.3.rs-3883562/v1 · Cited by 18
Tsai, H.-C., Huang, Y.-F., & Kuo, C.-W. (2024) · Research Square · doi:10.21203/rs.3.rs-4022248/v1 · Cited by 21
Kanato, S. A. T. O., Kaneko, H., & Fujimura, M. (2024) · doi:10.31219/osf.io/4hzya · Cited by 16
Does Social Bot Help Socialize? Evidence from a Microblogging Platform
Gao, Y. A., Zhang, M. M., & Lysyakov, M. (2025) · Information Systems Research · doi:10.1287/isre.2024.1089 · Cited by 18
Semantic Analysis of Phishing Emails Leading to Ransomware with ChatGPTPreprint
Fujima, H., Takeuchi, K., & Kumamoto, T. (2023) · Research Square · doi:10.21203/rs.3.rs-3670198/v1 · Cited by 14
RansomCillin: Leveraging NTFS Spare Space to Recover from Ransomware AttacksPreprint
Takeuchi, K., Fujima, H., Kumamoto, T., & Yoshida, Y. (2023) · doi:10.36227/techrxiv.24596754.v1 · Cited by 14
Zhang, Y., & Chen, X. (2023) · Research Square · doi:10.21203/rs.3.rs-3778538/v1 · Cited by 15
Dynamic Supplementation of Federated Search Results for Reducing Hallucinations in LLMsPreprint
Chen, J., Huang, X., & Li, Y. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/x5vge · Cited by 14
Knowledge Accuracy and Reducing Hallucinations in LLMs via Dynamic Domain Knowledge InjectionPreprint
Capellini, R., Atienza, F., & Sconfield, M. (2024) · Research Square · doi:10.21203/rs.3.rs-4540506/v1 · Cited by 14
A Comparative Analysis of Cultural Alignment in Large Language Models in Bilingual ContextsPreprint
Yuan, X., Hu, J., & Zhang, Q. (2024) · doi:10.31219/osf.io/6hpcf · Cited by 14
Optimizing Machine Translation: A Benchmarking Suite for Efficiency and Quality Enhancement
Kamuni, N., Dodda, S., Chintala, S., & Kunchakuri, N. (2024) · International Conference on Computing, Sciences and Communications (ICCSC) · doi:10.1109/iccsc62048.2024.10830335 · Cited by 14
The citation catastrophe: Propagation of AI-generated counterfeit citations in scholarship
Camp, N. T., Bengtson, J. A., & Sandstrom, J. C. (2025) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2025.103065 · Cited by 12
Assessing the Impact of GPT-4 Turbo in Generating Defeaters for Assurance Cases
Shahandashti, K. K., Sivakumar, M., Mohajer, M. M., Belle, A. B., Wang, S., & Lethbridge, T. C. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652291 · Cited by 24
Chen, J., Lu, S., & Zhong, L. (2024) · Applied Sciences · doi:10.3390/app14177716 · Cited by 12
Unmoderated Usability Studies Evolved: Can GPT Ask Useful Follow-up Questions?
Kuric, E., Demcak, P., & Krajcovic, M. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2427978 · Cited by 12
Cross-Cultural Implications of Large Language Models: An Extended Comparative Analysis
Shan, X., Xu, Y., Wang, Y., Lin, Y.-S., & Bao, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-76806-4_8 · Cited by 14
Adaptive Query Contextualization Algorithm for Enhanced Information Retrieval in Alpaca LLMPreprint
Kuo, C.-W., Huang, Y.-F., & Tsai, H.-C. (2023) · Research Square · doi:10.21203/rs.3.rs-3806145/v1 · Cited by 12
Shadiev, R., Nguyen, T. M., & Hwang, W. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2025.2612252 · Cited by 14
Analysing the Role of Generative AI in Software Engineering - Results from an MLR
Bazzan, T., Olojo, B., Majda, P., Kelly, T., Yılmaz, M., Marks, G., & Clarke, P. M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-71139-8_11 · Cited by 16
Using ChatGPT to Analyze Ransomware Messages and to Predict Ransomware ThreatsPreprint
Fujima, H., Kumamoto, T., & Yoshida, Y. (2023) · Research Square · doi:10.21203/rs.3.rs-3645967/v1 · Cited by 11
Measuring the Visual Hallucination in ChatGPT on Visually Deceptive ImagesPreprint
Ping, L., Gu, Y., & Feng, L. (2024) · doi:10.31219/osf.io/v23fr · Cited by 11
Kaate, I., Salminen, J. O., Jung, S., Santos, J. M., Häyhänen, E., Xuan, T. T. T., Azem, J. Y., & Jansen, B. J. J. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2469659 · Cited by 10
Investigating Hallucination Tendencies of Large Language Models in Japanese and EnglishPreprint
Tsuruta, H., & Sakaguchi, R. (2024) · Research Square · doi:10.21203/rs.3.rs-4521710/v1 · Cited by 10
Salminen, J. O., Jung, S., Medina, J., Aldous, K. K., Azem, J. Y., Akhtar, W., Häyhänen, E., & Jansen, B. J. J. (2024) · International Academic Mindtrek Conference (Mindtrek) · doi:10.1145/3681716.3681727 · Cited by 9
Wei, X. (2025) · Theoretical and Natural Science · doi:10.54254/2753-8818/2025.20110 · Cited by 12
Nowrozy, R., Ahmed, K. E. U., & Wang, H. (2025) · PLoS ONE · doi:10.1371/journal.pone.0310553 · Cited by 12
Su, X., & Gu, Y. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ns2v3 · Cited by 10
VaxBot-HPV: A GPT-based Chatbot for Answering HPV Vaccine-related QuestionsPreprint
Tao, C., Li, Y., Li, J., Li, M., Yu, E., Amith, M. “., Tang, L., Savas, L. S., & Cui, L. (2024) · Research Square · doi:10.21203/rs.3.rs-4876692/v1 · Cited by 9
Bane and boon of hallucinations in context of generative AIPreprint
Patel, H. (2024) · doi:10.36227/techrxiv.171198062.20183635/v1 · Cited by 11
Ho-tin, W., & Yip, G.-L. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/wkb32 · Cited by 8
Zhao, J., Huang, C., & Li, X. (2024) · Research Square · doi:10.21203/rs.3.rs-4566507/v1 · Cited by 8
Mitigating Hallucinations in Large Language Models with Sliding Generation and Self-ChecksPreprint
Harrington, F., Rosenthal, E., & Swinburne, M. (2024) · doi:10.36227/techrxiv.172297712.23792600/v1 · Cited by 8
Automated Methodologies for Evaluating Lying, Hallucinations, and Bias in Large Language ModelsPreprint
Ecurali, G., & Thackeray, Z. (2024) · Research Square · doi:10.21203/rs.3.rs-4855434/v1 · Cited by 8
Liu, Z., Kou, J., Zhang, W., Gu, C., Fang, X., Huang, Z., Hao, Y., Li, H., Lu, X., Yin, A., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189137 · Cited by 9
Using GPT-4 Turbo to Automatically Identify Defeaters in Assurance Cases
Shahandashti, K. K., Belle, A. B., Mohajer, M. M., Odu, O., Lethbridge, T. C., Hemmati, H., & Wang, S. (2024) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew61692.2024.00011 · Cited by 7
LLaMALoop: Enhancing Information Retrieval in LLaMA with Semantic Relevance Feedback LoopPreprint
Tsai, H.-C., Kuo, C.-W., & Huang, Y.-F. (2023) · Research Square · doi:10.21203/rs.3.rs-3810235/v1 · Cited by 7
Li, P.-H., & Lai, Y.-Y. (2024) · doi:10.31219/osf.io/ma6cq · Cited by 7
Detecting LLM Hallucinations Using Monte Carlo Simulations on Token ProbabilitiesPreprint
Ledger, G., & Mancinni, R. (2024) · doi:10.36227/techrxiv.171822396.61518693/v1 · Cited by 9
Boosting Long-term Factuality in Large Language Model with Real-World Entity QueriesPreprint
Davies, L., & Bellington, S. (2024) · Research Square · doi:10.21203/rs.3.rs-4834886/v1 · Cited by 7
Nadeem, M., Sohail, S. S., Madsen, D. Ø., Alzahrani, A. I., Ser, J. D., & Muhammad, K. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536937 · Cited by 7
LENS: Layers of Evaluation of Hallucination in GenAI Systems
Gu, C., Zhang, W., Huang, Z., Kou, J., Liu, Z., Zhao, C., Liu, C., Zhang, L., Lin, W., Wang, Z., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189150 · Cited by 32
AI Asyet Another Tool in Undergraduate Student Projects: Preliminary Results
Pérez-Colado, I. J., Freire, M., Calvo-Morata, A., Pérez-Colado, V. M., & Fernández‐Manjón, B. (2024) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578883 · Cited by 6
Investigating the Utility of ChatGPT in the Issue Tracking System: An Exploratory Study
Das, J. K., Mondal, S., & Roy, C. K. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645083 · Cited by 6
Reinforcement Learning for Optimized Information Retrieval in LLaMAPreprint
Tu, C.-H., Hsu, H.-J., & Chen, S.-W. (2024) · Research Square · doi:10.21203/rs.3.rs-3847100/v1 · Cited by 7
Gong, X., Liu, M., & Chen, X. (2024) · doi:10.31219/osf.io/srgpx · Cited by 6
Relational & culture-sensitive AI innovation
Herzog, C. (2025) · Journal of Responsible Technology · doi:10.1016/j.jrt.2025.100135 · Cited by 5
A systematically derived AI-based framework for student-centered learning in higher education
Albasry, H., Carmona-Cejudo, E., Rauf, A., & Chen, D. (2025) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2025.102085 · Cited by 12
Thao, N. T. P., Tam, D. H., Duc, M. L., & Sang, N. Q. (2025) · Sustainable Futures · doi:10.1016/j.sftr.2025.101523 · Cited by 8
Ethical Guidelines for the Application of Generative AI in German Journalism
Hofeditz, L., Jung, A.-K., Mirbabaie, M., & Stieglitz, S. (2025) · Digital Society · doi:10.1007/s44206-024-00151-w · Cited by 10
Implementing An Automated Socratic Method to Reduce Hallucinations in Large Language ModelsPreprint
Underwood, H., & Fenwick, Z. (2024) · doi:10.31219/osf.io/grh4b · Cited by 5
van Eerde, A. M., Teixeira, A., Galletti, F., Maternik, M., Capone, V. P., Westland, R., Mulder, J., Halbritter, J., Osterholt, T., Neukel, V., et al. (2025) · Pediatric Nephrology · doi:10.1007/s00467-025-06746-w · Cited by 4
ChatGPT and library users: AI risks of hallucinations and misinformation
Oladokun, B. D., Emmanuel, V. O., Osagie, O., & Alabi, A. O. (2025) · Cybrarians Journal · doi:10.70000/cj.2025.76.642 · Cited by 6
Using LLMs to Adapt Serious Games with Educators in the Loop
Bonetti, F., Bucchiarone, A., & Wanick, V. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78269-5_7 · Cited by 5
Opanasenko, Y., Bardone, E., Pedaste, M., & Siiman, L. A. (2024) · Education Sciences · doi:10.3390/educsci15010028 · Cited by 4
Anthony, A., Sharma, S., & Sharma, S. (2025) · Bulletin of Science Technology & Society · doi:10.1177/02704676251353105 · Cited by 10
RansomCillin: Leveraging NTFS Spare Space to Recover from Ransomware AttacksPreprint
Takeuchi, K., Fujima, H., Kumamoto, T., & Yoshida, Y. (2023) · Research Square · doi:10.21203/rs.3.rs-3648063/v1 · Cited by 4
The Double-Edged Sword of AI Writing ToolsPreprint
Xu, Y. (2024) · doi:10.31235/osf.io/wnpd5 · Cited by 6
Large Language Model Understands Chinese Better with Mega TokenizationPreprint
Lu, X., Wang, Q., & Liu, X. (2024) · Research Square · doi:10.21203/rs.3.rs-4542532/v1 · Cited by 4
Metadata
- Authors
- McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., & Halgamuge, M. N.
- Affiliations
- La Trobe University (AU)
- Massey University (NZ)
- Massey University, Auckland, New Zealand
- Cyberstronomy Pty Ltd, Ballarat, VIC, Australia
- RMIT University (AU)
- Venue
- IEEE Transactions on Artificial Intelligence
- Published
- 2024-06
- Volume
- 5
- Issue
- 6
- Pages
- 2739-2751
- ISSN
- 2691-4581 (ISSN-L)
- OpenAlex
- W4388691793
- Cited by
- 145
- References
- 49
- Citations by year
- 2026: 12 · 2025: 31 · 2024: 90
- FWCI
- 13.67
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- Locations
- IEEE Transactions on Artificial Intelligence (publishedVersion)
- Figshare (submittedVersion, cc-by)
- La Trobe University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- hallucination evaluation, ChatGPT-4, Google Bard, Perplexity AI, controversial issues, brain teasers, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
