Generative AI-powered social engineering: an analysis of phishing artifacts across an ethical gradient

Recent advancements in generative AI large language models (LLMs) have introduced new challenges for phishing detection and prevention. While LLM on platforms such as OpenAI may use content moderation and report malicious activity to authorities, standalone local LLMs are isolated and concealed, thus hidden from oversight. In this article, we examine the behavior of the most popular selected LLMs using Ollama and apply an ethical gradient to input requests for a malicious social engineering message, moving from ethically acceptable to ethically wrongful. We found that the locally run model's willingness to generate output varies significantly as the ethical appropriateness changes. We analyzed the artifacts generated by the model to identify novel GenAI-based characteristics that could help users recognize social engineering attacks. We discovered that some traditional markers, such as typos, urgency, and use of authority continue to remain relevant. However, we identified several distinct features specific to LLMs that organizations should consider for inclusion in current best practices. These include pervasive use of placeholder text and markdown syntax, discernible mismatches in tone throughout the artifacts, among others. Our contribution includes new and valuable insights into the cutting-edge of AI-generated social engineering attempts using locally available LLMs.