AI Accelerates Social Engineering Campaigns
Artificial intelligence (AI), particularly large language models (LLMs) and generative AI (genAI), is significantly changing the landscape of social engineering. While many AI-enabled social engineering tactics can still be addressed by existing defenses, the emergence of synthetic media, such as deepfakes and voice alteration, demands that organizations adapt their security strategies. AI allows threat actors to execute social engineering more quickly, cheaply, and at a larger scale, impacting everything from personalized phishing to sophisticated impersonation attacks, according to Recorded Future.
AI’s Role in Scaling Traditional Social Engineering
Threat actors are leveraging genAI to enhance established social engineering techniques. This includes crafting highly personalized phishing messages, performing target research, translating content, analyzing stolen inboxes, and automating follow-up communications. The goal remains consistent: to manipulate individuals into divulging information, transferring funds, or granting unauthorized access. Examples of AI’s impact include:
- Personalized Phishing: GenAI can create compelling and contextually relevant phishing emails, making them more difficult to detect than generic lures. This directly impacts the effectiveness of AI-enabled social engineering defenses that rely on identifying common patterns.
- Fraudulent Websites: AI is used to build sophisticated fake websites and login pages that mimic legitimate services, increasing the success rate of credential theft.
- Automated Scams: GenAI tools automate responses for employment, customer service, and business email compromise (BEC) scams, allowing attackers to manage multiple engagements simultaneously.
Malicious LLMs and Their Capabilities
Although many commercial LLMs incorporate safeguards against malicious use, a growing ecosystem of ‘malicious models’ designed to circumvent these controls has emerged. Platforms like WormGPT, EscapeGPT, FraudGPT, WolfGPT, DarkGPT, BlackhatGPT, KawaiiGPT, and WormGPT4 offer capabilities for generating phishing messages, harmful code, data theft tools, and even ransom notes. For example, WormGPT4, identified in late 2025, provides specific WormGPT phishing capabilities through consumer-friendly subscription models, illustrating the professionalization of these tools.
Legitimate AI tools are also being repurposed for malicious ends. Proofpoint reported that the AI website builder Lovable has been used to create tens of thousands of malicious phishing sites impersonating major brands like Microsoft and UPS. Similarly, GLM-5.2, an open-weight model from Chinese AI company Z.ai, raises concerns due to its potential for private agentic workflows, offensive code development, and vulnerability identification outside developer oversight.
Furthermore, the Phishing-as-a-Service (PhaaS) ecosystem has integrated AI. Services like EvilTokens, appearing in early 2026, combine AI-generated lures, research tools, account validation, and customizable phishing infrastructure into comprehensive packages. The Gentlemen ransomware group has also been observed using Chinese LLMs such as Kimi, DeepSeek, Qwen, and HUIHUI-AI to automate payload generation, technical analysis, and PII triage.
The Unique Challenge of Synthetic Media
While AI enhances existing threats, synthetic media—such as deepfakes and AI-generated voice alteration—presents a distinct and more critical challenge. These technologies weaken the audiovisual and biometric signals traditionally used as proof of identity. Research indicates that both humans and automated detection systems struggle to reliably identify deepfakes, particularly outside controlled environments. This means that defenses relying solely on recognizing a familiar face, voice, or identity document are becoming increasingly insufficient.
Synthetic Media Attack Mitigation and Recommendations
Organizations must fundamentally adapt their security posture to address the new realities presented by AI-driven social engineering, especially concerning synthetic media. Treating all AI-enabled threats as uniform risks can create a false sense of security.
Key recommendations for defenders include:
- Enhance Verification Procedures: Implement multi-factor authentication (MFA) and out-of-band verification processes for sensitive transactions or identity confirmations. Do not rely solely on visual or auditory cues that can be spoofed by synthetic media.
- Continuous Security Awareness Training: Educate employees on the evolving nature of AI-enabled social engineering, including deepfake and voice clone risks. Emphasize the importance of questioning unusual requests, even from seemingly familiar individuals.
- Update Incident Response Plans: Ensure incident response protocols account for sophisticated AI-driven social engineering and impersonation attempts, including how to verify identities when synthetic media is suspected.
- Monitor for Malicious AI Tools: Stay informed about the capabilities and proliferation of malicious LLMs and PhaaS offerings that integrate AI to anticipate and counter emerging tactics.
Related: AI-Enhanced Service Desk Attacks: Impersonation & Prevention, ScamBuster: AI-Driven Phishing Engagement for Threat Intel