CoChat
Sign Up
Cited by 199Spam and Phishing DetectionAdversarial Robustness in Machine LearningInformation and Cyber Security

Digital deception: generative artificial intelligence in social engineering and phishing

Open access
Schmitt, Marc, Flechais, Ivan·2024·Artificial Intelligence Review
DOI 10.1007/s10462-024-10973-2
Open this paper in CoChat — free account

Abstract

The advancement of Artificial Intelligence (AI) and Machine Learning (ML) has profound implications for both the utility and security of our digital interactions. This paper investigates the transformative role of Generative AI in Social Engineering (SE) attacks. We conduct a systematic review of social engineering and AI capabilities and use a theory of social engineering to identify three pillars where Generative AI amplifies the impact of SE attacks: Realistic Content Creation, Advanced Targeting and Personalization, and Automated Attack Infrastructure. We integrate these elements into a conceptual model designed to investigate the complex nature of AI-driven SE attacks—the Generative AI Social Engineering Framework. We further explore human implications and potential countermeasures to mitigate these risks. Our study aims to foster a deeper understanding of the risks, human implications, and countermeasures associated with this emerging paradigm, thereby contributing to a more secure and trustworthy human-computer interaction.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10462-024-10973-2

Cites

50 of 52 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Coordinated fleets of specialized agents can launch thousands of subtle, context-aware interactions that, taken together, are far more likely to sway or manipulate individuals than a single adversary could.

    “Coordinated fleets of specialized agents can launch thousands of subtle, context-aware interactions that, taken together, are far more likely to sway or manipulate individuals than a single adversary could (Schmitt & Flechais, 2024;Falade, 2023).”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Schmitt & Flechais, 2024

    Section 3.2 (p. 7) discusses coordination among AI agents, and Section 4.3 (p. 10) describes botnets delivering personalized messages at scale and adapting to responses. The paper does not specify thousands of interactions or establish that their cumulative effect is more likely to sway an individual than a single adversary would be. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. [8][8] OpenAI, GPT-4 Technical Report, (2023). http://arxiv.org/abs/2303.08774. - · arXiv
  2. [9][9] R. Anil, A.M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J.H. Clark, L. El Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G.H. Abrego, J. Ahn, J. Austin, P. Barham, J. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C.A. Choquette-Choo, A. Chowdhery, C. Crepy, S. Dave, M. Dehghani, S. Dev, J. Devlin, M. Díaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, L. Gonzalez, G. Gur-Ari, S. Hand, H. Hashemi, L. Hou, J. Howland, A. Hu, J. Hui, J. Hurwitz, M. Isard, A. Ittycheriah, M. Jagielski, W. Jia, K. Kenealy, M. Krikun, S. Kudugunta, C. Lan, K. Lee, B. Lee, E. Li, M. Li, W. Li, Y. Li, J. Li, H. Lim, H. Lin, Z. Liu, F. Liu, M. Maggioni, A. Mahendru, J. Maynez, V. Misra, M. Moussalem, Z. Nado, J. Nham, E. Ni, A. Nystrom, A. Parrish, M. Pellat, M. Polacek, A. Polozov, R. Pope, S. Qiao, E. Reif, B. Richter, P. Riley, A.C. Ros, A. Roy, B. Saeta, R. Samuel, R. Shelby, A. Slone, D. Smilkov, D.R. So, D. Sohn, S. Tokumine, D. Valter, V. Vasudevan, K. Vodrahalli, X. Wang, P. Wang, Z. Wang, T. Wang, J. Wieting, Y. Wu, K. Xu, Y. Xu, L. Xue, P. Yin, J. Yu, Q. Zhang, S. Zheng, C. Zheng, W. Zhou, D. Zhou, S. Petrov, Y. Wu, PaLM 2 Technical Report, (2023). http://arxiv.org/abs/2305.10403. - · arXiv
  3. [10][10] S. Neupane, I.A. Fernandez, S. Mittal, S. Rahimi, Impacts and Risk of Generative AI Technology on Cyber Defense, (2023). http://arxiv.org/abs/2306.13033. - · arXiv
  4. [11][11] S. Reed, K. Zolna, E. Parisotto, S.G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J.T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, N. de Freitas, A Generalist Agent, (2022) 2022 -2027. http://arxiv.org/abs/2205.06175. - · arXiv
  5. [17][17] J. Seymour, P. Tully, Generative Models for Spear Phishing Posts on Social Media, (2018). http://arxiv.org/abs/1802.05196. - · arXiv
  6. [24][24] I.J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative Adversarial Networks, (2014). http://arxiv.org/abs/1406.2661. - · arXiv
  7. [25][25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, I. Polosukhin, Attention Is All You Need, (2017). http://arxiv.org/abs/1706.03762. - · arXiv
  8. [28][28] A. Zou, Z. Wang, J.Z. Kolter, M. Fredrikson, Universal and Transferable Adversarial Attacks on Aligned Language Models, (2023). http://arxiv.org/abs/2307.15043. - · arXiv

Cited by

Sorted by citations

Related

Authors
Schmitt, M., & Flechais, I.
Affiliations
University of Oxford (GB)
Corresponding author
Schmitt, Marc
ORCID
Schmitt, Marc
Venue
Artificial Intelligence Review
Publisher
Springer Science and Business Media LLC
Published
2024-10-12
Volume
57
Issue
12
ISSN
  • 0269-2821 (ISSN-L)
  • 1573-7462
DOI
10.1007/s10462-024-10973-2
OpenAlex
W4403360640
Cited by
198
References
50
Citations by year
2026: 100 · 2025: 79 · 2024: 18 · 2023: 1
FWCI
74.07
Type
journal article
Language
en
Open access
Licence
Locations
  • Artificial Intelligence Review (publishedVersion, cc-by)
  • SSRN Electronic Journal (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
Keywords
generative AI, social engineering, phishing, AI-driven attacks, countermeasures, human-computer interaction security, deception

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review