Google Vertex AI SDK Model Hijacking via Bucket Squatting

A significant vulnerability in the Google Cloud Vertex AI SDK for Python has been uncovered, potentially allowing unauthorized actors to intercept machine learning model uploads. According to The Hacker News, this flaw allows an attacker with no prior access to a victim’s project to execute arbitrary code within Google’s model-serving infrastructure. The research, conducted by Palo Alto Networks Unit 42, identifies a technique dubbed “Pickle in the Middle,” which leverages the inherent risks of Python’s serialization format combined with insecure storage defaults.

Technical Analysis of the Vertex AI SDK Flaw

The vulnerability stems from how the SDK handles the transition of machine learning models from local development environments to the Google Cloud Model Registry. When a developer utilizes the SDK to upload or deploy a model, the library often requires a staging bucket to store assets temporarily. If the developer does not explicitly provide a Cloud Storage bucket, the SDK may automatically generate a bucket name based on predictable parameters, such as the project ID and region.

This predictability enables a technique known as bucket squatting. An attacker can preemptively create a bucket with the expected name in their own project. Because Cloud Storage bucket names are globally unique across all Google Cloud accounts, the victim’s SDK, finding that a bucket with the target name already exists, may attempt to use it. If the bucket permissions are misconfigured or the SDK lacks sufficient ownership verification, the victim inadvertently uploads their model artifacts to the attacker’s controlled storage.

When analyzing the Google Cloud Vertex AI SDK for Python vulnerability, researchers found that the threat is amplified by the use of the pickle format. Many machine learning frameworks use pickle for model serialization. However, pickle is notoriously vulnerable to RCE because it can be manipulated to execute arbitrary code during the deserialization process. By substituting a legitimate model with a malicious one in the squatted bucket, the attacker ensures that when Google’s serving infrastructure pulls and loads the model, the malicious payload is executed.

How to Prevent Bucket Squatting in Vertex AI Environments

The most effective way to defend against this TTP is to move away from default configurations. SOC teams and developers should collaborate to ensure that all staging buckets are explicitly defined and owned by the organization’s internal projects. Relying on the SDK’s automatic bucket creation is a security risk that bypasses traditional Zero Trust controls.

While Palo Alto Networks Unit 42 reported that there has been no evidence of this flaw being exploited in the wild, the potential impact is high. A successful exploit would allow an attacker to breach the isolation of Google’s serving environment, leading to a possible Supply Chain Attack if the malicious model is subsequently served to end-users.

Impact and Mitigation Strategies

Although a specific CVE ID was not assigned to this cloud-side configuration flaw in the initial report, its CVSS equivalent would likely reach the high range due to the potential for unauthenticated code execution. The primary risk involves the compromise of intellectual property and the integrity of the machine learning pipeline.

To implement a comprehensive Google Vertex AI RCE mitigation, organizations should follow these steps:

• Update the SDK: Ensure the Google Cloud Vertex AI SDK for Python is updated to the latest version, which includes improved checks for bucket ownership.
• Explicit Bucket Definition: Never rely on the SDK to choose a staging bucket. Always define a specific, pre-existing bucket in the staging_bucket parameter when calling aiplatform.init().
• IAM Restrictions: Use Service Accounts with the minimum necessary permissions. Ensure that the account running the SDK does not have broad permissions that would allow it to write to or read from external projects unexpectedly.
• Model Signing: Implement digital signatures for model artifacts to ensure that only verified models are loaded into the production environment.

By addressing these configuration gaps, organizations can protect their AI workflows from hijacking and ensure the security of their cloud infrastructure.