CoChat
Sign Up
PreprintCited by 1Spam and Phishing DetectionMisinformation and Its ImpactsUser Authentication and Security Systems

The DOI matches its DataCite record.

Assessing AI-Generated vs. Human-Authored Spear Phishing SMS Attacks: An Empirical Study

Open access
Francia, Jerson, Hansen, Derek, Schooley, Benjamin, Taylor, Matthew, Murray, Shydra Valynn, Cornelius, Rebekah, Snow, Greg·2024·arXiv
DOI 10.48550/arxiv.2406.13049arXiv 2406.13049

Preprint — published as 10.3390/jcp6040129

Open this paper in CoChat — free accountOpen PDF

Abstract

Personalized phishing is difficult to defend against because messages can be tailored to a target's work, interests, and social context. Large language models may make such tailoring faster and easier, but it remains unclear whether messages produced from simple prompts are more convincing than those written by people. This 25-target pilot study compared personalized smishing messages generated by GPT-4 with messages written by novice student authors working under time constraints. Using the proposed Threshold Ranking Approach for Personalized Deception (TRAPD), participants ranked 12 messages written for them, indicated the point at which they would intend to click, explained their reasoning, and judged whether each message was authored by GPT-4 or a human. GPT-4-generated messages elicited an intention to click more often than student-authored messages (28% versus 21%), although the difference was uncertain. More broadly, our findings suggest that a simple prompt can produce personalized messages that participants found comparably convincing within the uncertainty of this pilot study. Job-related messages were significantly more likely to elicit an intention to click than hobby- or social-media-related messages. When asked whether a message was written by a human or generated by AI, participants identified the source no more accurately than chance, although the two study-specific message sets remained computationally distinguishable based on their text. Together, these findings suggest that accessible AI-assisted personalization may increase the practical scale of social-engineering threats, while also demonstrating both the value and current limitations of TRAPD for controlled and ethical comparison.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.13049

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 42

  1. Rawan A. Alsharida, Bander Ali Saleh Al-rimy, Mostafa Al-Emran, and Anazida Zainal. A systematic review of multi perspectives on human cybersecurity behavior. Technology in Society, 73:102258, May 2023.
  2. Ezer Osei Yeboah-Boateng and Priscilla Mateko Amanor. Phishing, SMiShing & Vishing: An Assess- ment of Threats against Mobile Devices. Journal of Emerging Trends in Computing and Information Sci- ences, 5(4):297-307, April 2014.
  3. Zinaida Benenson, Freya Gassmann, and Robert Land- wirth. Unpacking Spear Phishing Susceptibility. In Michael Brenner, Kurt Rohloff, Joseph Bonneau, An- drew Miller, Peter Y.A. Ryan, Vanessa Teague, An- drea Bracciali, Massimiliano Sala, Federico Pintore, and Markus Jakobsson, editors, Financial Cryptogra- phy and Data Security, pages 610-627, Cham, 2017. Springer International Publishing.
  4. Julian Hazell. Large Language Models Can Be Used To Effectively Scale Spear Phishing Campaigns. arXiv preprint arXiv:2305.06972, May 2023. Accessed: 2024- 09-30. · arXiv
  5. Daniela Seabra Oliveira, Tian Lin, Harold Rocha, Dono- van Ellis, Sandeep Dommaraju, Huizi Yang, Devon Weir, Sebastian Marin, and Natalie C. Ebner. Em- pirical analysis of weapons of influence, life domains, and demographic-targeting in modern spam: an age- comparative perspective. Crime Science, 8(1):3, De- cember 2019.
  6. Tian Lin, Daniel E. Capecci, Donovan M. Ellis, Harold A. Rocha, Sandeep Dommaraju, Daniela S. Oliveira, and Natalie C. Ebner. Susceptibility to Spear- Phishing Emails: Effects of Internet User Demographics and Email Content. ACM Transactions on Computer- Human Interaction, 26(5):32:1-32:28, July 2019.
  7. John Seymour and Philip Tully. Generative Models for Spear Phishing Posts on Social Media, February 2018. arXiv:1802.05196 [cs, stat]. · arXiv
  8. Robert Karamagi. A Review of Factors Affecting the Effectiveness of Phishing. Computer and Information Science, 15(1):p20, November 2021. Number: 1.
  9. Sandhya Mishra and Devpriya Soni. Sms phishing and mitigation approaches. In 2019 Twelfth International Conference on Contemporary Computing (IC3), pages 1-5, 2019.
  10. Michael Bossetta. The Weaponization of Social Media: Spear Phishing and Cyberattacks on Democracy. Jour- nal of International Affairs, 71(1.5):97-106, 2018. Pub- lisher: Journal of International Affairs Editorial Board.
  11. Heather J. Parker and Stephen V. Flowerday. Contribut- ing factors to increased susceptibility to social media phishing attacks. SA Journal of Information Manage- ment, 22(1), June 2020.
  12. H. Khan, M. Alam, S. Al-Kuwari, and Y. Faheem. Of- fensive AI: Unification of Email Generation Through GPT-2 Model With a Game-Theoretic Approach for Spear-Phishing Attacks. Proceedings of the 2021 Inter- national Conference on Security and Emerging Trends in Cyber Technologies, pages 178-184, January 2021. Publisher: IET Digital Library.
  13. Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D. Griffin. Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities, Au- gust 2023. arXiv:2308.12833 [cs]. · arXiv
  14. Audrey de Rancourt-Raymond and Nadia Smaili. The unethical use of deepfakes. Journal of Financial Crime, 30(4):1066-1077, 2023. Accessed: 2024-01-17.
  15. Vijaya R Saraswathi, Iftequar Sk Ahmed, Sriveda M Reddy, S Akshay, Vrushik M Reddy, and Sanjana M Reddy. Automation of Recon Process for Ethical Hack- ers. In 2022 International Conference for Advancement in Technology (ICONAT), pages 1-6, January 2022.
  16. Mika Koivisto and Simone Grassini. Best humans still outperform artificial intelligence in a creative divergent thinking task. Scientific Reports, 13(1):13512, 2023. Accessed: 2024-01-17.
  17. Piotr Wilczyński, Wiktoria Mieleszczenko-Kowszewicz, and Przemysław Biecek. Resistance Against Manipu- lative AI: key factors and possible actions, April 2024. arXiv:2404.14230 [cs]. · arXiv
  18. Phillip Good. Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses. Springer, 2000.
  19. David W. Hosmer, Stanley Lemeshow, and Rodney X. Sturdivant. Applied Logistic Regression. Wiley, 3rd edition, 2013.
  20. Benjamin M. Bolker, Mollie E. Brooks, Connor J. Clark, Shane W. Geange, John R. Poulsen, M. Henry H. Stevens, and Jada S. S. White. Generalized Linear Mixed Models: A Practical Guide for Ecology and Evo- lution. Trends in Ecology & Evolution, 24(3):127-135, 2009.
  21. Juho Piironen and Aki Vehtari. On the Hyperprior Choice for the Global Shrinkage Parameter in the Horse- shoe Prior. In Proceedings of the 20th Interna- tional Conference on Artificial Intelligence and Statis- tics (AISTATS), 2017. Available online: https:// proceedings.mlr.press/v54/piironen17a.html.
  22. Marcus Butavicius, Kathryn Parsons, Malcolm Pattin- son, and Agata McCormac. Breaching the Human Fire- wall: Social engineering in Phishing and Spear-Phishing Emails, May 2016. arXiv:1606.00887 [cs]. · arXiv
  23. Alexis Palmer and Arthur Spirling. Large Language Models Can Argue in Convincing Ways About Politics, But Humans Dislike AI Authors: implications for Gov- ernance. Political Science, 75(3):281-291, September 2023.
  24. Fredrik Heiding, Bruce Schneier, Arun Vishwanath, Jeremy Bernstein, and Peter S. Park. Devising and De- tecting Phishing: Large Language Models vs. Smaller Human Models, November 2023. arXiv:2308.12287 [cs]. · arXiv
  25. Bartlomiej Hanus, Yu Andy Wu, and James Par- rish. Phish Me, Phish Me Not. Journal of Computer Information Systems, 62(3):516-526, May 2022. Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/08874417.2020.1858730. · DOI
  26. A. J. Burns, M. E. Johnson, and D. D. Caputo. Spear phishing in a barrel: Insights from a targeted phishing campaign. Journal of Organizational Computing and Electronic Commerce, 29(1):24-39, 2019. Accessed: 2024-01-17.
  27. Emma J. Williams, Joanne Hinds, and Adam N. Join- son. Exploring susceptibility to phishing in the work- place. International Journal of Human-Computer Stud- ies, 120:1-13, 2018.
  28. Nathalie Stembert, Arne Padmos, Mortaza S. Bargh, Sunil Choenni, and Frans Jansen. A Study of Preventing Email (Spear) Phishing by Enabling Human Intelligence. In 2015 European Intelligence and Security Informatics Conference, pages 113-120, September 2015.
  29. Simona Eftimie, Radu Moinescu, and Cȃtȃlin Rȃcuciu. Spear-phishing susceptibility stemming from personal- ity traits. IEEE Access, 10:73825-73835, 2022. Ac- cessed: 2024-01-17.
  30. Paul B. Baltes. Theoretical propositions of life- span developmental psychology: On the dynamics be- tween growth and decline. Developmental Psychology, 23(5):611-626, 1987. Place: US Publisher: American Psychological Association.
  31. Prateek Dewan, Anand Kashyap, and Ponnurangam Ku- maraguru. Analyzing social and stylometric features to identify spear phishing emails. In 2014 APWG Sym- posium on Electronic Crime Research (eCrime), pages 1-13, Sep. 2014.
  32. Nils Köbis and Luca Mossink. Artificial Intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human- written poetry, September 2020. arXiv:2005.09980 [cs, econ, q-fin]. · arXiv
  33. Maurice Jakesch, Jeffrey Hancock, and Mor Naaman. Human heuristics for AI-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11):e2208839120, March 2023. arXiv:2206.07271 [cs]. · arXiv
  34. George Thomopoulos, Dimitrios Lyras, and Christos Fi- das. Methodologies and ethical considerations in phish- ing research: A comprehensive review. In Proceedings of the 2nd International Conference of the ACM Greek SIGCHI Chapter, CHIGREECE '23, New York, NY, USA, 2023. Association for Computing Machinery.
  35. Verena Distler. The influence of context on response to spear-phishing attacks: an in-situ deception study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, USA, 2023. Association for Computing Machinery.
  36. Jared James Meyers, Derek L. Hansen, Justin S. Gi- boney, and Dale C. Rowe. Training Future Cybersecu- rity Professionals in Spear Phishing using SiEVE. In Proceedings of the 19th Annual SIG Conference on In- formation Technology Education, SIGITE '18, pages 135-140, New York, NY, USA, September 2018. Asso- ciation for Computing Machinery.
  37. Amandeep K. Raja and Jian Zhou. AI accountability: Approaches, affecting factors, and challenges. Com- puter, 56(4):46-56, 2023. Accessed: 2024-01-17.
  38. Rasha Zieni, Luisa Massari, and Maria Carla Calzarossa. Phishing or not phishing? a survey on the detection of phishing websites. IEEE Access, 11:18499-18519, 2023.
  39. Zuochao Dou, Issa Khalil, Abdallah Khreishah, Ala Al-Fuqaha, and Mohsen Guizani. Systematization of knowledge (sok): A systematic review of software- based web phishing detection. IEEE Communications Surveys & Tutorials, 19(4):2797-2819, Fourthquarter 2017.
  40. Hans Hoeken, Karin Fikkers, Anita Eerland, Bregje Holleman, Jos van Berkum, and Henk Pander Maat. The perceived convincingness model: why and under what conditions processing fluency and emotions are valid indicators of a message's perceived convincing- ness. Communication Theory, 32(4):488-496, 09 2022.
  41. Gregory D. Moody, Dennis F. Galletta, and Brian Kim- ball Dunn. Which phish get caught? an exploratory study of individuals' susceptibility to phishing. Euro- pean Journal of Information Systems, 26(6):564-584, 2017.
  42. Sijie Zhuo, Robert Biddle, Yun Sing Koh, Danielle Lot- tridge, and Giovanni Russello. Sok: Human-centered phishing susceptibility. ACM Trans. Priv. Secur., 26(3), April 2023.

Cited by

Sorted by citations

Metadata

Authors
Francia, J., Hansen, D., Schooley, B., Taylor, M., Murray, S. V., Cornelius, R., & Snow, G.
Affiliations
Brigham Young University (US)
Corresponding author
  • Francia, Jerson
  • Hansen, Derek
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
6
Issue
4
Pages
129-129
ISSN
2624-800X (ISSN-L)
DOI
10.48550/arxiv.2406.13049
arXiv
2406.13049
OpenAlex
W7172214483
Cited by
1
References
0
Citations by year
2025: 1
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Cybersecurity and Privacy (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Information Systems › User Authentication and Security Systems
Keywords
spear phishing, smishing, GPT-4, social engineering, click intention, deception detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review