Navigating the Web of Disinformation and Misinformation: Large Language Models as Double-Edged Swords
Open accessThe DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
This paper explores the dual role of Large Language Models (LLMs) in the context of online misinformation and disinformation. In today’s digital landscape, where the internet and social media facilitate the rapid dissemination of information, discerning between accurate content and falsified information presents a formidable challenge. Misinformation, often arising unintentionally, and disinformation, crafted deliberately, are at the forefront of this challenge. LLMs such as OpenAI’s GPT-4, equipped with advanced language generation abilities, present a double-edged sword in this scenario. While they hold promise in combating misinformation by fact-checking and detecting LLM-generated text, their ability to generate realistic, contextually relevant text also poses risks for creating and propagating misinformation. Further, LLMs are plagued with many problems such as biases, knowledge cutoffs, and hallucinations, which may further perpetuate misinformation and disinformation. The paper outlines historical developments in misinformation detection and how it affects social media consumption, especially among youth, and introduces LLMs and their applications in various domains. It then critically analyzes the potential of LLMs to generate and counter misinformation and disinformation in sensitive topics such as healthcare, COVID-19, and political agendas. Further, it discusses mitigation strategies, ethical considerations, and regulatory measures, summarizing previous methods and proposing future research direction toward leveraging the benefits of LLMs while minimizing misuse risks. The paper concludes by acknowledging LLMs as powerful tools with significant implications in both spreading and combating misinformation in the digital age.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 1×
“These advanced language generation models, exemplified by OpenAI’s GPT-3 [30], boast remarkable capabilities in comprehending and generating text that closely resembles human language [31], [32], [33].”
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“Transformers have significantly improved NLP by effectively managing sequential data, enabling parallel processing, and capturing extensive contextual relationships in textual information [89].”
GPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519
Fake news on Twitter during the 2016 U.S. presidential election
Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B., & Lazer, D. (2019) · Science · doi:10.1126/science.aau2706 · Cited by 1,803
Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA
Loomba, S., de Figueiredo, A., Piatek, S. J., de Graaf, K., & Larson, H. J. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01056-1 · Cited by 1,800 · cited 1×
“For example, the spread of misinformation about COVID-19 vaccines contributed to lower vaccination rates even in educated countries [124].”
Susceptibility to misinformation about COVID-19 around the world
Roozenbeek, J., Schneider, C. R., Dryhurst, S., Kerr, J., Freeman, A. L. J., Recchia, G., van der Bles, A. M., & van der Linden, S. (2020) · Royal Society Open Science · doi:10.1098/rsos.201199 · Cited by 1,447
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
ChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008 · cited 1×
“Nevertheless, the present condition of LLMs may give rise to ethical concerns, as researchers could potentially exploit them to compose academic articles, or students could employ them to engage in academic dishonesty throughout examinations [84].”
An overview of online fake news: Characterization, detection, and discussion
Zhang, X., & Ghorbani, A. A. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2019.03.004 · Cited by 997 · cited 1×
“Studies on fake news indicate that bogus information can easily be disseminated without substantial manipulation of content [118].”
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.03825 · Cited by 861
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
Fake news, disinformation and misinformation in social media: a review
Aïmeur, E., Amri, S., & Brassard, G. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01028-5 · Cited by 772 · cited 2×
“While the issue of fake news may seem like a contemporary problem, it has deep historical roots [13].”
“Fake News” Is Not Simply False Information: A Concept Explication and Taxonomy of Online Content
Molina, M. D., Sundar, S. S., Le, T., & Lee, D. (2021) · American Behavioral Scientist · doi:10.1177/0002764219878224 · Cited by 485 · cited 1×
“LLMs can also be trained to identify the sensitive discernible patterns that accompany misinformation and disinformation such as sensationalization, incorrect analogies, deceptive headlines, and emotional manipulation [136].”
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441 · cited 1×
“While many commercial LLM vendors do not allow users to prompt them with any potentially dangerous, unethical, or immoral tasks [165], [166], the safety measures taken to achieve this are prone to failing against jailbreaking [167], [168] or adversarial attacks [169].”
Power Hungry Processing: Watts Driving the Cost of AI Deployment?
Luccioni, S., Jernite, Y., & Strubell, E. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658542 · Cited by 392
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · cited 1×
“While many commercial LLM vendors do not allow users to prompt them with any potentially dangerous, unethical, or immoral tasks [165], [166], the safety measures taken to achieve this are prone to failing against jailbreaking [167], [168] or adversarial attacks [169].”
Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581318 · Cited by 288
Jailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2307.02483 · Cited by 280
Jailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3508 · Cited by 280
Jaiswal, J., LoSchiavo, C., & Perlman, D. C. (2020) · AIDS and Behavior · doi:10.1007/s10461-020-02925-y · Cited by 276 · cited 1×
“Misinformation and disinformation have been pervasive in various critical contexts such as elections, global pandemics like COVID-19, and political movements [4], [12], [17], [18], [19], [20], [21].”
A Survey of Knowledge-enhanced Text Generation
Yu, W., Zhu, C., Li, Z., Hu, Z., Wang, Q., Ji, H., & Jiang, M. (2022) · ACM Computing Surveys · doi:10.1145/3512467 · Cited by 257
Neural User Response Generator: Fake News Detection with Collective User Intelligence
Qian, F., Gong, C., Sharma, K., & Liu, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/533 · Cited by 237 · cited 1×
“Adding to the complexity of this problem, the ability of LLMs to detect disinformation cannot be improved by using only a single source of data [131].”
Selective Classification for Deep Neural Networks
Geifman, Y., & El-Yaniv, R. (2017) · arXiv · doi:10.48550/arxiv.1705.08500 · Cited by 220
On the Risk of Misinformation Pollution with Large Language Models
Pan, Y., Pan, L., Chen, W., Nakov, P., Kan, M.-Y., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.97 · Cited by 210
Universal and Transferable Adversarial Attacks on Aligned Language Models
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15043 · Cited by 197
Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation
Zhang, J., Bao, K., Zhang, Y., Wang, W., Feng, F., & He, X. (2023) · arXiv · doi:10.48550/arxiv.2305.07609 · Cited by 196
Healthcare professionals’ acts of correcting health misinformation on social media
Bautista, J. R., Zhang, Y., & Gwizdka, J. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104375 · Cited by 171
Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity
Zhuo, T. Y., Huang, Y., Chen, C., & Xing, Z. (2023) · arXiv · doi:10.48550/arxiv.2301.12867 · Cited by 164
Combating misinformation in the age of LLMs: Opportunities and challenges
Chen, C., & Shu, K. (2024) · AI Magazine · doi:10.1002/aaai.12188 · Cited by 148
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., & Zhang, Y. (2025) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/taslpro.2025.3606231 · Cited by 136
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.08747 · Cited by 135
A Survey of Hallucination in Large Foundation Models
Rawte, V., Sheth, A., & Das, A. (2023) · arXiv · doi:10.48550/arxiv.2309.05922 · Cited by 133 · cited 2×
“Unex-pectedly, an alternate perspective has emerged that notes AI hallucination as non-harmful in special cases [160], as even hallucinating LLMs can be used collaboratively as partners to provide research leads, support creative writing, and alleviate writer’s block.”
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.2.12 · Cited by 132 · cited 1×
“Similarly, while misinformation generated by LLMs can be difficult to detect for humans as well as LLMs [109], the recent development of methods that can detect LLM-generated text [132], [133] may help alleviate this issue by enabling stricter moderation of online content.”
The Science of Detecting LLM-Generated Text
Tang, R., Chuang, Y.-N., & Hu, X. (2024) · Communications of the ACM · doi:10.1145/3624725 · Cited by 125 · cited 1×
“One current line of research investigates the problem of detection of all LLM-generated text [112].”
On the creativity of large language models
Franceschelli, G., & Musolesi, M. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02127-3 · Cited by 113
ChatGPT, Bard, and large language models for biomedical research: opportunities and pitfalls
Thapa, S., & Adhikari, S. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03284-0 · Cited by 111
Self-citationArceneaux, K., & Truex, R. (2023) · Perspectives on Politics · doi:10.1017/s1537592722000901 · Cited by 97 · cited 1×
“Notably, in the 2020 US presidential elections, multiple campaigns were organized to spread disinformation about candidates [117].”
Detecting LLM-Generated Text in Computing Education: Comparative Study for ChatGPT Cases
Orenstrakh, M. S., Karnalim, O., Suárez, C. A., & Liut, M. (2024) · 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00027 · Cited by 79
Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
Liu, Y., Yao, Y., Ton, J.-F., Zhang, X., Guo, R., Cheng, H., Klochkov, Y., Taufiq, M. F., & Li, H. (2023) · arXiv · doi:10.48550/arxiv.2308.05374 · Cited by 79
Truth, Lies, and Automation: How Language Models Could Change Disinformation
Center for Security and Emerging Technology, Buchanan, B., Lohn, A., Musser, M., & Sedova, K. (2021) · doi:10.51593/2021ca003 · Cited by 74
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.813 · Cited by 73
ChatGPT and medical education: a double-edged sword
Khairatun Hisan, U., & Miftahul Amri, M. (2023) · Journal of Pedagogy and Education Science · doi:10.56741/jpes.v2i01.302 · Cited by 63
On the Possibilities of AI-Generated Text Detection
Chakraborty, S., Bedi, A. S., Zhu, S., An, B., Manocha, D., & Huang, F. (2023) · arXiv · doi:10.48550/arxiv.2304.04736 · Cited by 59
Privacy in the Time of Language Models
Peris, C., Dupuy, C., Majmudar, J., Parikh, R., Smaili, S., Zemel, R., & Gupta, R. (2023) · Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3539597.3575792 · Cited by 41
Using ChatGPT and Other Large Language Model (LLM) Applications for Academic Paper Assignments
Jungherr, A. (2023) · doi:10.31235/osf.io/d84q6 · Cited by 37
Artificial intelligence—friend or foe in fake news campaigns
Węcel, K., Sawiński, M., Stróżyna, M., Lewoniewski, W., Księżniak, E., Stolarski, P., & Abramowicz, W. (2023) · Economics and Business Review · doi:10.18559/ebr.2023.2.736 · Cited by 28
Pretraining Language Models with Human Preferences
Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C. L., Phang, J., Bowman, S. R., & Perez, E. (2023) · arXiv · doi:10.48550/arxiv.2302.08582 · Cited by 23 · cited 1×
“While many commercial LLM vendors do not allow users to prompt them with any potentially dangerous, unethical, or immoral tasks [165], [166], the safety measures taken to achieve this are prone to failing against jailbreaking [167], [168] or adversarial attacks [169].”
GPT-who: An Information Density-based Machine-Generated Text Detector
Venkatraman, S., Uchendu, A., & Lee, D. (2023) · arXiv · doi:10.48550/arxiv.2310.06202 · Cited by 21
Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithms
Törnberg, P., Valeeva, D., Uitermark, J., & Bail, C. (2023) · arXiv · doi:10.48550/arxiv.2310.05984 · Cited by 20 · cited 1×
“Consequently, LLM-generated content has seen exponential growth on social media, for better [148] or worse [149], [150], [151], analogous to a double-edged sword.”
Tricking llms into disobedience: Formalizing, analyzing, and detecting jailbreaks
Rao, A., Vashistha, S., Naik, A., Aditya, S., & Choudhury, M. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/2ivv5y3ufu9y · Cited by 19 · cited 1×
“Non-expert users can exploit prompts to jailbreak LLMs, resulting in degenerate output behavior, privacy and security breaches, and offensive outputs [101].”
Sun, K., Xu, Y. E., Zha, H., Liu, Y., & Dong, X. L. (2023) · arXiv · doi:10.48550/arxiv.2308.10168 · Cited by 13
EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models
Wang, P., Zhang, N., Tian, B., Xi, Z., Yao, Y., Xu, Z., Wang, M., Mao, S., Wang, X., Cheng, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.07269 · Cited by 9 · cited 1×
“On the other hand, the knowledge of LLMs can be updated in small capacities without the need for re-training by harnessing the knowledge mining abilities of search engines [190] or knowledge injection [191].”
What's the Magic Word? A Control Theory of LLM Prompting
Bhargava, A., Witkowski, C., Looi, S.-Z., & Thomson, M. (2023) · arXiv · doi:10.48550/arxiv.2310.04444 · Cited by 7
Attention Sorting Combats Recency Bias In Long Context Language Models
Peysakhovich, A., & Lerer, A. (2023) · arXiv · doi:10.48550/arxiv.2310.01427 · Cited by 6 · cited 1×
“Intuitively, LLMs should be finetuned regularly on new data, but this might be computationally unfeasible and can create problems like recency bias [189].”
Cited by
Thapa, S., Shiwakoti, S., Shah, S. B., Adhikari, S., Veeramani, H., Nasim, M., & Naseem, U. (2025) · Social Network Analysis and Mining · doi:10.1007/s13278-025-01428-9 · Cited by 74
Self-citationBrohi, S., Mastoi, Q.-U.-A., Jhanjhi, N. Z., & Pillai, T. R. (2025) · Algorithms · doi:10.3390/a18080499 · Cited by 69
“As the use of LLMs continues to grow across public and private sectors, concerns about misinformation, misuse, and bias have become more prominent [51–60].”
The Importance of AI Data Governance in Large Language Models
(2025) · doi:10.3390/bdcc9060147 · Cited by 58
The Importance of AI Data Governance in Large Language Models
Pahune, S., Akhtar, Z., Mandapati, V., & Siddique, K. (2025) · doi:10.20944/preprints202504.0219.v1 · Cited by 58
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
Metadata
- Authors
- Shah, S. B., Thapa, S., Acharya, A., Rauniyar, K., Poudel, S., Jain, S., Masood, A., & Naseem, U.
- Venue
- IEEE Access
- Published
- 2025
- Volume
- 13
- Pages
- 169262-169282
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4399146384
- Cited by
- 39
- References
- 55
- FWCI
- 28.63
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.