CoChat
Sign Up
PreprintCited by 1Spam and Phishing DetectionInformation and Cyber SecurityAdversarial Robustness in Machine Learning

The Shadow of Fraud: The Emerging Danger of AI-powered Social Engineering and its Possible Cure

Open access
Yu, Jingru, Yu, Yi, Wang, Xuhong, Lin, Yilun, Yang, Manzhi, Qiao, Yu, Wang, Fei-Yue·2024·arXiv
DOI 10.48550/arxiv.2407.15912arXiv 2407.15912
Open this paper in CoChat — free accountOpen PDF

Abstract

Social engineering (SE) attacks remain a significant threat to both individuals and organizations. The advancement of Artificial Intelligence (AI), including diffusion models and large language models (LLMs), has potentially intensified these threats by enabling more personalized and convincing attacks. This survey paper categorizes SE attack mechanisms, analyzes their evolution, and explores methods for measuring these threats. It highlights the challenges in raising awareness about the risks of AI-enhanced SE attacks and offers insights into developing proactive and adaptable defense strategies. Additionally, we introduce a categorization of the evolving nature of AI-powered social engineering attacks into "3E phases": Enlarging, wherein the magnitude of attacks expands through the leverage of digital media; Enriching, introducing novel attack vectors and techniques; and Emerging, signifying the advent of novel threats and methods. Moreover, we emphasize the necessity for a robust framework to assess the risk of AI-powered SE attacks. By identifying and addressing gaps in existing research, we aim to guide future studies and encourage the development of more effective defenses against the growing threat of AI-powered social engineering.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.15912

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI systems that can reliably masquerade as people can be used by criminals for fraud, by governments for foreign influence, and by commercial entities to create manipulative experiences for users.

    “AI systems that can reliably masquerade as people can be used by criminals for fraud [1,14,59], by governments for foreign influence [26,35], and by commercial entities to create engaging but ultimately manipulative experiences for users [7,57].”

    From People cannot distinguish GPT-4 from a human in a Turing test (2025) · cites this paper as [59]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 172

  1. T. C. King, N. Aggarwal, M. Taddeo, and L. Floridi, "Artificial intelligence crime: An interdisciplinary analysis of foreseeable threats and solutions," Science and engineering ethics, vol. 26 2020, 89-120, 2020.
  2. M. P. Stoecklin, J. Jang, and D. Kirat, "Deeplocker: How ai can power a stealthy new breed of malware," Security Intelligence, vol. 8 no. 2018 2018, 2018.
  3. K. F. Steinmetz, A. Pimentel, and W. R. Goe, "Per- forming social engineering: A qualitative study of in- formation security deceptions," Computers in Human Behavior, vol. 124 2021, 106930, 2021.
  4. Z. Wang, H. Zhu, P. Liu, and L. Sun, "Social en- gineering in cybersecurity: A domain ontology and knowledge graph application examples," Cybersecu- rity, vol. 4 2021, 1-21, 2021.
  5. M. F. M. Firdhous, W. Elbreiki, I. Abdullahi, B. Su- dantha, and R. Budiarto, "Wormgpt: A large language model chatbot for criminals," in 2023 24th Interna- tional Arab Conference on Information Technology (ACIT), IEEE, 2023, 1-6.
  6. F. Salahdine and N. Kaabouch, "Social engineering attacks: A survey," Future internet, vol. 11 no. 4 2019, 89, 2019.
  7. N. Mashtalyar, U. N. Ntaganzwa, T. Santos, S. Hakak, and S. Ray, "Social engineering attacks: Recent ad- vances and challenges," in International Conference on Human-Computer Interaction, Springer, 2021, 417- 431.
  8. W. Fuertes, D. Arévalo, J. D. Castro, et al., "Impact of social engineering attacks: A literature review," Developments and Advances in Defense and Security: Proceedings of MICRADS 2021 2022, 25-35, 2022.
  9. R. Salama, F. Al-Turjman, S. Bhatia, and S. P. Ya- dav, "Social engineering attack types and prevention techniques-a survey," in 2023 International Confer- ence on Computational Intelligence, Communication Technology and Networking (CICTN), IEEE, 2023, 817-820.
  10. N. Kaloudi and J. Li, "The ai-based cyber threat land- scape: A survey," ACM Computing Surveys (CSUR), vol. 53 no. 1 2020, 1-34, 2020.
  11. T. F. Blauth, O. J. Gstrein, and A. Zwitter, "Artificial intelligence crime: An overview of malicious use and abuse of ai," Ieee Access, vol. 10 2022, 77110-77122, 2022.
  12. E. N. J. van and L. Waltman, "Software survey: Vosviewer, a computer program for bibliometric map- ping," Scientometrics, vol. 84 no. 2 Aug. 1, 2010, 523-538, Aug. 1, 2010.
  13. P. Damle, "Social engineering: A tip of the iceberg," Information Systems Control Journal, vol. 2 2002, 51- 52, 2002.
  14. P. Maan and M. Sharma, "Social engineering: A partial technical attack," International Journal of Computer Science Issues, vol. 9 no. 2 2012, 557-559, 2012.
  15. D. J. Dance, "Pretexting: A necessary means to a necessary end," Drake L. Rev., vol. 56 2007, 791, 2007.
  16. G. Desolda, L. S. Ferro, A. Marrella, T. Catarci, and M. F. Costabile, "Human factors in phishing attacks: A systematic literature review," ACM Computing Surveys (CSUR), vol. 54 no. 8 2021, 1-35, 2021.
  17. S. E. Griffin and C. C. Rackley, "Vishing," en, in Pro- ceedings of the 5th annual conference on Information security curriculum development, Kennesaw Georgia: ACM, Sep. 2008, 33-35.
  18. S. Lohani, "Social engineering: Hacking into humans," International Journal of Advanced Studies of Scientific Research, vol. 4 no. 1 2019, 2019.
  19. D. R. Murphy and R. H. Murphy, "Phishing, Pharming, and Vishing: Fraud in the Internet Age," REVIEW 2007, 37, 2007, Publisher: Citeseer.
  20. D.-W. Park and J.-M. Seo, "A study of informa- tion leakage prevention through certified authentica- tion in phishing, vishing, smishing attacks," Journal of the Korea Society of Computer and Information, vol. 12 no. 2 2007, 171-180, 2007, Publisher: Korean Society of Computer Information.
  21. A. S. Alazri, "The awareness of social engineering in information revolution: Techniques and challenges," in 2015 10th International Conference for Internet Technology and Secured Transactions (ICITST), 2015, 198-201.
  22. M. Mattera and M. M. Chowdhury, "Social engineer- ing: The looming threat," in 2021 IEEE International Conference on Electro Information Technology (EIT), 2021, 056-061.
  23. Y. V. Zotina, "Pretexting as a Social Engineering Tech- nique Used by Telephone Scammers: A Criminological View of the Problem," Bull. Kazan L. Inst. MIA Russ. 2022, 93, 2022, Publisher: HeinOnline.
  24. S. C. Bennett, "Ethics of pretexting in a cyber world," McGeorge L. Rev., vol. 41 2009, 271, 2009.
  25. K. S. Jones, M. E. Armstrong, M. K. Tornblad, and A. Siami Namin, "How social engineers use persua- sion principles during vishing attacks," Information & Computer Security, vol. 29 no. 2 2021, 314-331, 2021, Publisher: Emerald Publishing Limited.
  26. E. M. Maseno, "Vishing attack detection model for mobile users.," PhD Thesis, KCA University, 2017.
  27. M. E. Armstrong, K. S. Jones, and A. S. Namin, "How Perceptions of Caller Honesty Vary During Vishing Attacks That Include Highly Sensitive or Seemingly Innocuous Requests," en, Human Factors: The Jour- nal of the Human Factors and Ergonomics Society, vol. 65 no. 2 Mar. 2023, 275-287, Mar. 2023.
  28. S. Ashfaq, P. Chandre, S. Pathan, U. Mande, M. Nim- balkar, and P. Mahalle, "Defending Against Vishing Attacks: A Comprehensive Review for Prevention and Mitigation Techniques," en, in Cyber Security and Digital Forensics, N. R. Roy, S. Tanwar, and U. Batra, Eds., vol. 896, Series Title: Lecture Notes in Networks and Systems, Singapore: Springer Nature Singapore, 2024, 411-422.
  29. S. Tu, M. Waqas, S. U. Rehman, et al., "Reinforcement learning assisted impersonation attack detection in device-to-device communications," IEEE Transactions on Vehicular Technology, vol. 70 no. 2 2021, 1474- 1479, 2021.
  30. S. Bala, G. Sharma, and A. K. Verma, "Impersonation attack on certificateless key agreement protocol," In- ternational Journal of Ad Hoc and Ubiquitous Com- puting, vol. 27 no. 2 2018, 108-120, 2018.
  31. M. Westerlund, "The emergence of deepfake technol- ogy: A review," Technology innovation management review, vol. 9 no. 11 2019, 2019.
  32. M. Albahar and J. Almalki, "Deepfakes: Threats and countermeasures systematic review," Journal of Theoretical and Applied Information Technology, vol. 97 no. 22 2019, 3242-3250, 2019, Publisher: Little Lion Scientific.
  33. K. A. Pantserev, "The Malicious Use of AI-Based Deepfake Technology as the New Threat to Psycholog- ical Security and Political Stability," en, in Cyber De- fence in the Age of AI, Smart Societies and Augmented Humanity, H. Jahankhani, S. Kendzierskyj, N. Chel- vachandran, and J. Ibarra, Eds., Series Title: Advanced Sciences and Technologies for Security Applications, Cham: Springer International Publishing, 2020, 37-55.
  34. R. Katarya and A. Lal, "A study on combating emerging threat of deepfake weaponization," in 2020 Fourth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC), 2020, 485-490.
  35. M. Sharma and M. Kaur, "A Review of Deepfake Technology: An Emerging AI Threat," en, in Soft Com- puting for Security Applications, G. Ranganathan, X. Fernando, F. Shi, and Y. El Allioui, Eds., vol. 1397, Se- ries Title: Advances in Intelligent Systems and Com- puting, Singapore: Springer Singapore, 2022, 605-619.
  36. V. A. Jones, "Artificial intelligence enabled deepfake technology: The emergence of a new threat," PhD Thesis, Utica College, 2020.
  37. J. Botha and H. Pieterse, "Fake news and deepfakes: A dangerous threat for 21st century information secu- rity," in ICCWS 2020 15th International Conference on Cyber Warfare and Security. Academic Conferences and publishing limited, 2020, 57.
  38. V. Karasavva and A. Noorbhai, "The Real Threat of Deepfake Pornography: A Review of Canadian Policy," en, Cyberpsychology, Behavior, and Social Networking, vol. 24 no. 3 Mar. 2021, 203-209, Mar. 2021.
  39. J. Langa, "Deepfakes, real consequences: Crafting legislation to combat threats posed by deepfakes," BUL Rev., vol. 101 2021, 761, 2021, Publisher: HeinOnline.
  40. E. I. Galyashina and V. D. Nikishin, "The protec- tion of megascience projects from deepfake technolo- gies threats: Information law aspects," in Journal of Physics: Conference Series, Issue: 1, vol. 2210, 2022, 012007.
  41. H. Chung and S. Lee, Intelligent Virtual Assistant knows Your Life, arXiv:1803.00466 [cs], Feb. 2018. · arXiv
  42. A. Maedche, C. Legner, A. Benlian, et al., "AI- Based Digital Assistants: Opportunities, Threats, and Research Perspectives," en, Business & Information Systems Engineering, vol. 61 no. 4 Aug. 2019, 535- 544, Aug. 2019.
  43. R. Mitev, M. Miettinen, and A.-R. Sadeghi, "Alexa Lied to Me: Skill-based Man-in-the-Middle Attacks on Virtual Assistants," en, in Proceedings of the 2019 ACM Asia Conference on Computer and Communi- cations Security, Auckland New Zealand: ACM, Jul. 2019, 465-478.
  44. N. Zhang, X. Mi, X. Feng, X. Wang, Y. Tian, and F. Qian, "Dangerous skills: Understanding and mitigating security risks of voice-controlled third-party functions on virtual personal assistant systems," in 2019 IEEE Symposium on Security and Privacy (SP), 2019, 1381- 1396.
  45. N. An, A. Duff, M. Noorani, S. Weber, and S. Mancoridis, "Malware anomaly detection on virtual assistants," in 2018 13th International Conference on Malicious and Unwanted Software (MALWARE), 2018, 124-131.
  46. T. Bolton, T. Dargahi, S. Belguith, M. S. Al-Rakhami, and A. H. Sodhro, "On the security and privacy challenges of virtual assistants," Sensors, vol. 21 no. 7 2021, 2312, 2021, Publisher: MDPI.
  47. B. Kumar, A. V. Singh, and P. Agarwal, "AI based Computational Trust Model for Intelligent Virtual As- sistant," Journal of Information Systems and Telecom- munication (JIST), vol. 4 no. 32 2021, 263, 2021, Publisher: RICEST.
  48. S. Pandit, J. Liu, R. Perdisci, and M. Ahamad, Fight- ing Voice Spam with a Virtual Assistant Prototype, arXiv:2008.03554 [cs], Aug. 2020. · arXiv
  49. S. Pandit, K. Sarker, R. Perdisci, M. Ahamad, and D. Yang, "Combating robocalls with phone virtual assistant mediated interaction," in USENIX Security, 2023.
  50. E. Ferrara, ""Manipulation and abuse on social me- dia" by Emilio Ferrara with Ching-man Au Ye- ung as coordinator," en, ACM SIGWEB Newsletter, vol. 2015 no. Spring Apr. 2015, 1-9, Apr. 2015.
  51. H.-C. H. Chang, E. Chen, M. Zhang, G. Muric, and E. Ferrara, "Social bots and social media manipulation in 2020: The year in review," in Handbook of Compu- tational Social Science, Volume 1, 2021, 304-323.
  52. Z. Weng and A. Lin, "Public opinion manipulation on social media: Social network analysis of Twitter bots during the COVID-19 pandemic," International journal of environmental research and public health, vol. 19 no. 24 2022, 16376, 2022, Publisher: MDPI.
  53. M. Paquet-Clouston, O. Bilodeau, and D. Décary- Hétu, "Can we trust social media data? social network manipulation by an iot botnet," in Proceedings of the 8th international conference on social media & society, 2017, 1-9.
  54. S. Bradshaw and P. Howard, "Troops, trolls and trou- blemakers: A global inventory of organized social media manipulation," Computational Propaganda Re- search Project 2017, 2017, Publisher: Oxford Internet Institute.
  55. A. Deb, L. Luceri, A. Badaway, and E. Ferrara, "Perils and Challenges of Social Media and Election Ma- nipulation Analysis: The 2018 US Midterms," en, in Companion Proceedings of The 2019 World Wide Web Conference, San Francisco USA: ACM, May 2019, 237-247.
  56. R. Gorwa and D. Guilbeault, "Unpacking the So- cial Media Bot: A Typology to Guide Research and Policy," Policy & Internet, vol. 12 no. 2 Jun. 2020, 225-248, Jun. 2020.
  57. M. Orabi, D. Mouheb, Z. Al Aghbari, and I. Kamel, "Detection of bots in social media: A system- atic review," Information Processing & Management, vol. 57 no. 4 2020, 102250, 2020.
  58. M. C. Benigni, K. Joseph, and K. M. Carley, "Bot- ivistm: Assessing Information Manipulation in So- cial Media Using Network Analytics," in Emerging Research Challenges and Opportunities in Computa- tional Social Network Analysis and Mining, N. Agar- wal, N. Dokoohaki, and S. Tokdemir, Eds., Cham: Springer International Publishing, 2019, 19-42.
  59. W. Ye and Q. Li, "Chatbot security and privacy in the age of personal assistants," in 2020 IEEE/ACM Symposium on Edge Computing (SEC), IEEE, 2020, 388-393.
  60. K. M. Huang-Isherwood, J. Cho, J.-W. Hong, and E. Lee, "Human or Not?: An Experiment With Chatbot Manipulations to Test Machine Heuristics and Political Self-Concepts," en, Social Science Computer Review May 2024, 08944393241252027, May 2024.
  61. M. Carroll, A. Chan, H. Ashton, and D. Krueger, "Characterizing Manipulation from AI Systems," en, in Equity and Access in Algorithms, Mechanisms, and Optimization, Boston MA USA: ACM, Oct. 2023, 1-13.
  62. A. Ho, J. Hancock, and A. S. Miner, "Psychological, relational, and emotional effects of self-disclosure after conversations with a chatbot," Journal of Communica- tion, vol. 68 no. 4 2018, 712-733, 2018, Publisher: Oxford University Press.
  63. C. Crolic, F. Thomaz, R. Hadi, and A. T. Stephen, "Blame the Bot: Anthropomorphism and Anger in Customer-Chatbot Interactions," en, Journal of Mar- keting, vol. 86 no. 1 Jan. 2022, 132-148, Jan. 2022.
  64. M. De Gennaro, E. G. Krumhuber, and G. Lucas, "Effectiveness of an Empathic Chatbot in Combating Adverse Effects of Social Exclusion on Mood," Fron- tiers in Psychology, vol. 10 Jan. 2020, 3061, Jan. 2020.
  65. J. Yoo and Y. Cho, "Icsa: Intelligent chatbot secu- rity assistant using text-cnn and multi-phase real-time defense against sns phishing attacks," Expert Systems with Applications, vol. 207 2022, 117893, 2022.
  66. R. Roy and V. Naidoo, "Enhancing chatbot effec- tiveness: The role of anthropomorphic conversational styles and time orientation," Journal of Business Re- search, vol. 126 2021, 23-34, 2021, Publisher: Else- vier.
  67. M. Shumanov and L. Johnson, "Making conversa- tions with chatbots more personalized," Computers in Human Behavior, vol. 117 2021, 106627, 2021, Publisher: Elsevier.
  68. Y. Yu, S. Yao, T. Zhou, et al., "Data on the move: Traffic-oriented data trading platform powered by ai agent with common sense," in 2024 IEEE Intelligent Vehicles Symposium (IV), Jun. 2024, 521-526.
  69. A. Panda, C. A. Choquette-Choo, Z. Zhang, Y. Yang, and P. Mittal, Teach LLMs to Phish: Stealing Private Information from Language Models, arXiv:2403.00871 [cs], Mar. 2024. · arXiv
  70. Q. Zhang, B. Zeng, C. Zhou, G. Go, H. Shi, and Y. Jiang, Human-Imperceptible Retrieval Poisoning Attacks in LLM-Powered Applications, arXiv:2404.17196 [cs], Apr. 2024. · arXiv
  71. W. Yang, X. Bi, Y. Lin, S. Chen, J. Zhou, and X. Sun, Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents, arXiv:2402.11208 [cs], Feb. 2024. · arXiv
  72. P. He, H. Xu, Y. Xing, H. Liu, M. Yamada, and J. Tang, "Data Poisoning for In-context Learning," arXiv preprint arXiv:2402.02160 2024, 2024. · arXiv
  73. R. Navigli, S. Conia, and B. Ross, "Biases in Large Language Models: Origins, Inventory, and Discus- sion," en, Journal of Data and Information Quality, vol. 15 no. 2 Jun. 2023, 1-21, Jun. 2023.
  74. P. P. Ray, "ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope," Internet of Things and Cyber-Physical Systems 2023, 2023, Publisher: Else- vier.
  75. S. Kolisko and C. J. Anderson, "Exploring social biases of large language models in a college artificial intelligence course," in Proceedings of the AAAI Con- ference on Artificial Intelligence, Issue: 13, vol. 37, 2023, 15825-15833.
  76. A. Mishra, G. Nayak, S. Bhattacharya, T. Kumar, A. Shah, and M. Foltin, "LLM-Guided Counterfactual Data Generation for Fairer AI," en, in Companion Proceedings of the ACM on Web Conference 2024, Singapore Singapore: ACM, May 2024, 1538-1545.
  77. A. Taubenfeld, Y. Dover, R. Reichart, and A. Gold- stein, Systematic Biases in LLM Simulations of De- bates, arXiv:2402.04049 [cs], Feb. 2024. · arXiv
  78. G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang, Humans or LLMs as the Judge? A Study on Judgement Biases, arXiv:2402.10669 [cs], Apr. 2024. · arXiv
  79. W. Zou, R. Geng, B. Wang, and J. Jia, Poisone- dRAG: Knowledge Poisoning Attacks to Retrieval- Augmented Generation of Large Language Models, arXiv:2402.07867 [cs], Feb. 2024. · arXiv
  80. A. Elmahdy and A. Salem, Deconstructing Classifiers: Towards A Data Reconstruction Attack Against Text Classification Models, arXiv:2306.13789 [cs], Jun. 2023. · arXiv
  81. J. G. Wang, J. Wang, M. Li, and S. Neel, Pandora's White-Box: Increased Training Data Leakage in Open LLMs, arXiv:2402.17012 [cs], Feb. 2024. · arXiv
  82. X. Li, R. Wang, M. Cheng, T. Zhou, and C.-J. Hsieh, DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers, arXiv:2402.16914 [cs], Mar. 2024. · arXiv
  83. V. Raina, A. Liusie, and M. Gales, Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment, arXiv:2402.14016 [cs], Feb. 2024. · arXiv
  84. A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, Universal and Transferable Adversarial Attacks on Aligned Language Models, arXiv:2307.15043 [cs], Dec. 2023. · arXiv
  85. J.-Y. Yao, K.-P. Ning, Z.-H. Liu, M.-N. Ning, and L. Yuan, LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples, arXiv:2310.01469 [cs], Oct. 2023. · arXiv
  86. J. Li, K. Gao, Y. Bai, J. Zhang, S.-t. Xia, and Y. Wang, FMM-Attack: A Flow-based Multi-modal Adversarial Attack on Video-based LLMs, arXiv:2403.13507 [cs], Mar. 2024. · arXiv
  87. J. Chu, Y. Liu, Z. Yang, X. Shen, M. Backes, and Y. Zhang, Comprehensive Assessment of Jailbreak Attacks Against LLMs, arXiv:2402.05668 [cs], Feb. 2024. · arXiv
  88. Z. Xu, Y. Liu, G. Deng, Y. Li, and S. Picek, LLM Jailbreak Attack versus Defense Techniques -A Com- prehensive Study, arXiv:2402.13457 [cs], Feb. 2024. · arXiv
  89. Z. Chang, M. Li, Y. Liu, J. Wang, Q. Wang, and Y. Liu, Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues, arXiv:2402.09091 [cs], Feb. 2024. · arXiv
  90. Y. Huang, S. Gupta, M. Xia, K. Li, and D. Chen, Catastrophic Jailbreak of Open-source LLMs via Ex- ploiting Generation, arXiv:2310.06987 [cs], Oct. 2023. · arXiv
  91. A. Wei, N. Haghtalab, and J. Steinhardt, "Jailbroken: How does llm safety training fail?" Advances in Neural Information Processing Systems, vol. 36 2024, 2024.
  92. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection," en, in Proceedings of the 16th ACM Workshop on Artifi- cial Intelligence and Security, Copenhagen Denmark: ACM, Nov. 2023, 79-90.
  93. S. Sai, U. Yashvardhan, V. Chamola, and B. Sikdar, "Generative ai for cyber security: Analyzing the poten- tial of chatgpt, dall-e and other models for enhancing the security space," IEEE Access 2024, 2024, Pub- lisher: IEEE.
  94. W. Fu, H. Wang, C. Gao, G. Liu, Y. Li, and T. Jiang, Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration, arXiv:2311.06062 [cs], Dec. 2023. · arXiv
  95. M. Duan, A. Suri, N. Mireshghallah, et al., Do Mem- bership Inference Attacks Work on Large Language Models? arXiv:2402.07841 [cs], Feb. 2024. · arXiv
  96. B. Wang, W. Chen, H. Pei, et al., DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models, arXiv:2306.11698 [cs], Feb. 2024. · arXiv
  97. C. Lu, C. Qian, G. Zheng, et al., From GPT-4 to Gem- ini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities, arXiv:2401.15071 [cs], Jan. 2024. · arXiv
  98. L. Häufglöckner, "Self-supervised language models in journalism: Quality perception of GPT-3-written articles," PhD Thesis, Hochschule Darmstadt, 2023.
  99. I. Greevink, T. Offerman, and G. Romagnoli, "AI- Powered Promises: The Influence of ChatGPT on Trust and Trustworthiness," Technical report, AEA RCT Registry, https://doi. org/10.1257/rct. 11511-1.0, Tech. Rep., 2023. · DOI
  100. D. Agarwal, A. R. Fabbri, P. Laban, et al., Investigat- ing the prompt leakage effect and black-box defenses for multi-turn LLM interactions, arXiv:2404.16251 [cs], Apr. 2024. · arXiv
  101. A. Kumar, S. Singh, S. V. Murty, and S. Ragupathy, The Ethics of Interaction: Mitigating Security Threats in LLMs, arXiv:2401.12273 [cs], Jan. 2024. · arXiv
  102. X. Liu, Z. Yu, Y. Zhang, N. Zhang, and C. Xiao, Automatic and Universal Prompt Injection Attacks against Large Language Models, arXiv:2403.04957 [cs], Mar. 2024. · arXiv
  103. D. Pasquini, M. Strohmeier, and C. Troncoso, Neural Exec: Learning (and Learning from) Execution Trig- gers for Prompt Injection Attacks, arXiv:2403.03792 [cs], May 2024. · arXiv
  104. Q. Zhan, Z. Liang, Z. Ying, and D. Kang, In- jecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents, arXiv:2403.02691 [cs], Mar. 2024. · arXiv
  105. E. Bagdasaryan, T.-Y. Hsieh, B. Nassi, and V. Shmatikov, Abusing Images and Sounds for In- direct Instruction Injection in Multi-Modal LLMs, arXiv:2307.10490 [cs], Oct. 2023. · arXiv
  106. Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong, Prompt Injection Attacks and Defenses in LLM-Integrated Ap- plications, arXiv:2310.12815 [cs], Oct. 2023. · arXiv
  107. K. Aryal, M. Gupta, M. Abdelsalam, and M. Saleh, Intra-Section Code Cave Injection for Adversarial Evasion Attacks on Windows PE Malware File, arXiv:2403.06428 [cs], Mar. 2024. · arXiv
  108. H. A. Noman and O. M. Abu-Sharkh, "Code injection attacks in wireless-based Internet of Things (IoT): A comprehensive review and practical implementations," Sensors, vol. 23 no. 13 2023, 6067, 2023, Publisher: MDPI.
  109. D. Oliynyk, R. Mayer, and A. Rauber, "I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences," en, ACM Computing Surveys, vol. 55 no. 14s Dec. 2023, 1-41, Dec. 2023.
  110. N. Carlini, D. Paleka, K. D. Dvijotham, et al., Stealing Part of a Production Language Model, arXiv:2403.06634 [cs], Mar. 2024. · arXiv
  111. R. Yang, H. He, Y. Xu, et al., "Efficient intrusion de- tection toward IoT networks using cloud-edge collab- oration," Computer Networks, vol. 228 2023, 109724, 2023, Publisher: Elsevier.
  112. S. Shourya, I. Venkatachalam, H. Patel, and M. Mittal, "Comparative and Preventive Analysis of Dictionary Attacks Check for updates," Recent Advances in Elec- trical and Electronic Engineering: Select Proceedings of ICSTE 2023, vol. 1071 2023, 443, 2023, Publisher: Springer Nature.
  113. H. Huang, Z. Zhao, M. Backes, Y. Shen, and Y. Zhang, Composite Backdoor Attacks Against Large Language Models, arXiv:2310.07676 [cs], Mar. 2024. · arXiv
  114. X. He, J. Wang, Q. Xu, et al., Transferring Troubles: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning, arXiv:2404.19597 [cs], Apr. 2024. · arXiv
  115. Y. Yu, S. Yao, J. Li, F.-Y. Wang, and Y. Lin, "Swdpm: A social welfare-optimized data pricing mechanism," in 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Oct. 2023, 2900-2906.
  116. Y. Wang, Y. Pan, M. Yan, Z. Su, and T. H. Luan, "A survey on ChatGPT: AI-generated contents, chal- lenges, and solutions," IEEE Open Journal of the Computer Society 2023, 2023, Publisher: IEEE.
  117. L. Basyoni and J. Qadir, "AI Generated Content in the Metaverse: Risks and Mitigation Strategies," in 2023 International Symposium on Networks, Computers and Communications (ISNCC), 2023, 1-4.
  118. B. Grimme, J. Pohl, H. Winkelmann, L. Stampe, and C. Grimme, "Lost in transformation: Rediscovering llm-generated campaigns in social media," in Multidis- ciplinary International Symposium on Disinformation in Open Online Media, Springer, 2023, 72-87.
  119. C. Liu, X. Ouyang, R. Bai, and L. Yuan, "Risk Analysis of AIGC in Market Regulation," in FFIT 2023: Proceedings of the 2nd International Conference on Financial Innovation, FinTech and Information Technology, FFIT 2023, July 7-9, 2023, Chongqing, China, European Alliance for Innovation, 2023, 237.
  120. D. Chen and Y. Tian, "Challenges, Risks, and Oppor- tunities: Marxist Political Economy Review of AIGC," in Computer Sciences & Mathematics Forum, Issue: 1, vol. 8, MDPI, 2023, 80.
  121. D. Guo, H. Chen, R. Wu, and Y. Wang, "AIGC challenges and opportunities related to public safety: A case study of ChatGPT," Journal of Safety Science and Resilience, vol. 4 no. 4 2023, 329-339, 2023, Publisher: Elsevier.
  122. G. Xuan, "Risks And Regulatory Framework of AI- Generated Content (AIGC) in the Judicial Field," Journal of Education, Humanities and Social Sciences, vol. 19 2023, 278-282, 2023.
  123. K. R. Ahern and D. Sosyura, "Rumor has it: Sensa- tionalism in financial media," The Review of Financial Studies, vol. 28 no. 7 2015, 2050-2093, 2015, Pub- lisher: Oxford University Press.
  124. Z. He and A. Manela, "Information Acquisition in Rumor-Based Bank Runs," en, The Journal of Finance, vol. 71 no. 3 Jun. 2016, 1113-1158, Jun. 2016.
  125. Z. He, Z. Cai, J. Yu, X. Wang, Y. Sun, and Y. Li, "Cost-efficient strategies for restraining rumor spread- ing in mobile social networks," IEEE Transactions on Vehicular Technology, vol. 66 no. 3 2016, 2789-2800, 2016, Publisher: IEEE.
  126. S. Mishra, "Exploring the impact of ai-based cyber se- curity financial sector management," Applied Sciences, vol. 13 no. 10 2023, 5875, 2023, Publisher: MDPI.
  127. X. Yang and Y. Luo, "Rumor Clarification and Stock Returns: Do Bull Markets Behave Differently from Bear Markets?" en, Emerging Markets Finance and Trade, vol. 50 no. 1 Jan. 2014, 197-209, Jan. 2014.
  128. W. Dong, S. Liao, and Z. Zhang, "Leveraging Finan- cial Social Media Data for Corporate Fraud Detec- tion," en, Journal of Management Information Systems, vol. 35 no. 2 Apr. 2018, 461-487, Apr. 2018.
  129. K. M. Greenhill and B. Oppenheim, "Rumor has it: The adoption of unverified information in conflict zones," International Studies Quarterly, vol. 61 no. 3 2017, 660-676, 2017, Publisher: Oxford University Press.
  130. S. O. Olabanji, O. B. Oladoyinbo, C. U. Asonze, T. O. Oladoyinbo, S. A. Ajayi, and O. O. Olaniyi, "Effect of adopting AI to explore big data on Personally Identifi- able Information (PII) for financial and economic data transformation," Available at SSRN 4739227 2024, 2024.
  131. Y. Yu, S. Yao, K. Wang, Y. Chen, F.-Y. Wang, and Y. Lin, "Pursuing equilibrium of medical resources via data empowerment in parallel healthcare system," in 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Oct. 2023, 3186-3191.
  132. X. Lv, Y. Yu, X. Xie, F.-Y. Wang, Y. Lin, and Y. Chen, "Optimize the accessibility of healthcare facilities via acp-based approach," in 2023 IEEE International Con- ference on Systems, Man, and Cybernetics (SMC), Hawaii, USA: IEEE, Oct. 2023.
  133. Y. Chen and P. Esmaeilzadeh, "Generative AI in Medical Practice: In-Depth Exploration of Privacy and Security Challenges," Journal of Medical Internet Re- search, vol. 26 2024, e53008, 2024, Publisher: JMIR Publications Toronto, Canada.
  134. A. Rahman, M. S. Hossain, N. A. Alrajeh, and F. Alsolami, "Adversarial examples-Security threats to COVID-19 deep learning systems in medical IoT de- vices," IEEE Internet of Things Journal, vol. 8 no. 12 2020, 9603-9610, 2020, Publisher: IEEE.
  135. C. Gongye, H. Li, X. Zhang, et al., "New passive and active attacks on deep neural networks in medical applications," en, in Proceedings of the 39th Interna- tional Conference on Computer-Aided Design, Virtual Event USA: ACM, Nov. 2020, 1-9.
  136. B. Tag, N. Van Berkel, S. Verma, et al., "DDoD: Dual Denial of Decision Attacks on Human-AI Teams," IEEE Pervasive Computing, vol. 22 no. 1 2023, 77-84, 2023, Publisher: IEEE.
  137. B. Schippers, "Artificial Intelligence and Democratic Politics," en, Political Insight, vol. 11 no. 1 Mar. 2020, 32-35, Mar. 2020.
  138. F. Filgueiras, "The politics of AI: Democracy and authoritarianism in developing countries," en, Journal of Information Technology & Politics, vol. 19 no. 4 Oct. 2022, 449-464, Oct. 2022.
  139. A. Jungherr, "Artificial Intelligence and Democracy: A Conceptual Framework," en, Social Media + Society, vol. 9 no. 3 Jul. 2023, 20563051231186353, Jul. 2023.
  140. C. Starke and M. Lünich, "Artificial intelligence for political decision-making in the European Union: Ef- fects on citizens' perceptions of input, throughput, and output legitimacy," Data & Policy, vol. 2 2020, e16, 2020, Publisher: Cambridge University Press.
  141. P. C. Magalhães, "Introduction -Financial Crisis, Aus- terity, and Electoral Politics," en, Journal of Elections, Public Opinion and Parties, vol. 24 no. 2 Apr. 2014, 125-133, Apr. 2014.
  142. Y. Alahmed and R. Abadla, "Exploring the potential implications of ai-generated content in social engineer- ing attacks," International Journal of Computing and Digital Systems, vol. 16 no. 1 2024, 1-11, 2024.
  143. M. Hejase, U. Ozguner, M. Barbier, and J. Ibanez- Guzman, "A methodology for model-based validation of autonomous vehicle systems," in 2020 IEEE Intel- ligent Vehicles Symposium (IV), IEEE, 2020, 2097- 2103.
  144. T. Khan and M. Faisal, "An efficient bayesian network model (bnm) for software risk prediction in design phase development," International Journal of Informa- tion Technology, vol. 15 no. 4 2023, 2147-2160, 2023.
  145. M. C. Ghanem and T. M. Chen, "Reinforcement learn- ing for intelligent penetration testing," in 2018 second world conference on smart trends in systems, security and sustainability (WorldS4), IEEE, 2018, 185-192.
  146. P. P. Parthy and G. Rajendran, "Identification and prevention of social engineering attacks on an enter- prise," in 2019 International Carnahan Conference on Security Technology (ICCST), IEEE, 2019, 1-5.
  147. W. Syafitri, Z. Shukur, U. Asma'Mokhtar, R. Su- laiman, and M. A. Ibrahim, "Social engineering at- tacks prevention: A systematic literature review," IEEE access, vol. 10 2022, 39325-39343, 2022, Publisher: IEEE.
  148. M. Xue, Z. Wu, Y. Zhang, J. Wang, and W. Liu, "AdvParams: An active DNN intellectual property protection technique via adversarial perturbation based parameter encryption," IEEE Transactions on Emerg- ing Topics in Computing 2022, 2022, Publisher: IEEE.
  149. S. Turgay and İ. İlter, "Perturbation Methods for Protecting Data Privacy: A Review of Techniques and Applications," Automation and Machine Learning, vol. 4 no. 2 2023, 31-41, 2023, Publisher: Clausius Scientific Press.
  150. W. Mazurczyk, D. Lee, and A. Vlachos, "Disinforma- tion 2.0 in the age of AI: A cybersecurity perspective," Communications of the ACM, vol. 67 no. 3 2024, 36-39, 2024, Publisher: ACM New York, NY, USA.
  151. A. K. Jain, S. R. Sahoo, and J. Kaubiyal, "Online social networks security and privacy: Comprehensive review and analysis," en, Complex & Intelligent Sys- tems, vol. 7 no. 5 Oct. 2021, 2157-2177, Oct. 2021.
  152. G. Shrivastava, P. Kumar, R. P. Ojha, P. K. Srivas- tava, S. Mohan, and G. Srivastava, "Defensive mod- eling of fake news through online social networks," IEEE Transactions on Computational Social Systems, vol. 7 no. 5 2020, 1159-1167, 2020, Publisher: IEEE.
  153. H. Hu, Y. Liu, C. Chen, H. Zhang, and Y. Liu, "Optimal decision making approach for cyber security defense using evolutionary game," IEEE Transactions on Network and Service Management, vol. 17 no. 3 2020, 1683-1700, 2020, Publisher: IEEE.
  154. X. Liu, H. Cheng, P. He, et al., Adversarial Training for Large Neural Language Models, arXiv:2004.08994 [cs], Apr. 2020. · arXiv
  155. I. Staliūnaitė, P. J. Gorinski, and I. Iacobacci, "Im- proving commonsense causal reasoning by adversarial training and data augmentation," in Proceedings of the AAAI Conference on Artificial Intelligence, Issue: 15, vol. 35, 2021, 13834-13842.
  156. K. Song, H. Sun, X. Tan, et al., LightPAFF: A Two- Stage Distillation Framework for Pre-training and Fine-tuning, arXiv:2004.12817 [cs], Apr. 2020. · arXiv
  157. S. Padmanabhan, Y. Onoe, M. Zhang, G. Durrett, and E. Choi, "Propagating knowledge updates to lms through distillation," Advances in Neural Information Processing Systems, vol. 36 2024, 2024.
  158. J. Flemings and M. Annavaram, Differentially Private Knowledge Distillation via Synthetic Text Generation, arXiv:2403.00932 [cs], Mar. 2024. · arXiv
  159. L. Hu, I. Habernal, L. Shen, and D. Wang, Differ- entially Private Natural Language Models: Recent Advances and Future Directions, arXiv:2301.09112 [cs], Oct. 2023. · arXiv
  160. T. Ali and P. Kostakos, HuntGPT: Integrating Ma- chine Learning-Based Anomaly Detection and Ex- plainable AI with Large Language Models (LLMs), arXiv:2309.16021 [cs], Sep. 2023. · arXiv
  161. A. Zytek, S. Pidò, and K. Veeramachaneni, LLMs for XAI: Future Directions for Explaining Explanations, arXiv:2405.06064 [cs], May 2024. · arXiv
  162. T. Ifargan, L. Hafner, M. Kern, O. Alcalay, and R. Kishony, Autonomous LLM-driven research from data to human-verifiable research papers, arXiv:2404.17605 [cs, q-bio], Apr. 2024. · arXiv
  163. J. Zhang, M. Marone, T. Li, B. Van Durme, and D. Khashabi, Verifiable by Design: Aligning Lan- guage Models to Quote from Pre-Training Data, arXiv:2404.03862 [cs], Apr. 2024. · arXiv
  164. A. Goossens and J. Vanthienen, "Integrating GPT- Technologies with Decision Models for Explainabil- ity," in World Conference on Explainable Artificial Intelligence, Springer, 2023, 428-448.
  165. E. Giboulot and F. Teddy, WaterMax: Breaking the LLM watermark detectability-robustness-quality trade- off, arXiv:2403.04808 [cs], Mar. 2024. · arXiv
  166. L. Pan, A. Liu, Z. He, et al., MarkLLM: An Open-Source Toolkit for LLM Watermarking, arXiv:2405.10051 [cs], May 2024. · arXiv
  167. T. Shevlane, Structured access: An emerging paradigm for safe AI deployment, arXiv:2201.05159 [cs], Apr. 2022. · arXiv
  168. Y. Wu, F. Roesner, T. Kohno, N. Zhang, and U. Iqbal, SecGPT: An Execution Isolation Architecture for LLM- Based Systems, arXiv:2403.04960 [cs], Mar. 2024. · arXiv
  169. Z. Zhang, Y. Lu, J. Ma, et al., "ShieldLM: Empower- ing LLMs as Aligned, Customizable and Explainable Safety Detectors," arXiv preprint arXiv:2402.16444 2024, 2024. · arXiv
  170. J. O'Brien, S. Ee, and Z. Williams, "Deployment cor- rections: An incident response framework for frontier AI models," arXiv preprint arXiv:2310.00328 2023, 2023. · arXiv
  171. J. Clymer, N. Gabrieli, D. Krueger, and T. Larsen, Safety Cases: How to Justify the Safety of Advanced AI Systems, arXiv:2403.10462 [cs], Mar. 2024. · arXiv
  172. Y. Yigit, M. A. Ferrag, I. H. Sarker, et al., Critical Infrastructure Protection: Generative AI, Challenges, and Opportunities, arXiv:2405.04874 [cs], May 2024. · arXiv

Cited by

Sorted by citations

Authors
Yu, J., Yu, Y., Wang, X., Lin, Y., Yang, M., Qiao, Y., & Wang, F.-Y.
Affiliations
  • Shanghai Artificial Intelligence Laboratory (CN)
  • Macau University of Science and Technology (MO)
ORCID
Yang, Manzhi
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.15912
arXiv
2407.15912
OpenAlex
W6929221453
Cited by
1
References
0
Citations by year
2025: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
  • Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
social engineering attacks, large language models, diffusion models, attack classification, risk assessment framework, defense strategy, fraud, digital media

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review