VPC Service Controls: GCP's Most Powerful and Most Painful Security Feature

I am writing this from inside a VPC Service Controls perimeter at a global banking institution, and I mean that almost literally. No GCP console access. No direct outbound internet from any compute resource. Every API call audited. Every cross-project data movement blocked unless explicitly permitted. Workload Identity Federation for all service-to-service communication — no service account keys, no exceptions.

This is the most locked-down GCP environment I have ever worked in, and I have worked in some locked-down environments. It is also, grudgingly, the correct way to run GCP for an organisation that handles the kind of data a global bank handles.

VPC Service Controls is the right tool. It is also the tool that will add 2-3x to your development time and make you question your career choices at least once a week.

What VPC-SC actually does

The concept is simple. VPC Service Controls creates security perimeters around GCP projects. Any API call that would move data across the perimeter boundary is blocked. Data stays inside the perimeter unless you’ve explicitly configured an exception.

Think of it as a firewall for GCP APIs. Not for network traffic — for API calls. A BigQuery query that reads from a dataset in Project A and writes to a table in Project B will fail if those projects are in different perimeters (or if one is outside the perimeter entirely). A Cloud Storage copy from a perimeter-protected bucket to an external bucket: blocked. A Vertex AI custom training job that tries to pull a base image from Docker Hub: blocked.

The threat model is data exfiltration. A compromised service account shouldn’t be able to copy your BigQuery datasets to an attacker-controlled project. A malicious insider shouldn’t be able to gsutil cp your data to their personal GCP account. VPC-SC prevents this by making the perimeter boundary impermeable to data movement unless explicitly allowed.

Simple concept. Now let me tell you what happens when you actually turn it on.

What breaks

Everything. Everything breaks the first time.

Cloud Build

Cloud Build needs to pull base images. Your Dockerfile says FROM python:3.11-slim. That image lives in Docker Hub. Docker Hub is outside your perimeter. Build fails.

Solution: mirror every base image you need into a private Artifact Registry repository inside the perimeter. Maintain a process to keep those mirrors updated. This is table stakes and every VPC-SC guide mentions it.

What the guides don’t always mention: Cloud Build itself runs in a Google-managed project. The build steps execute in a worker pool that needs to be a private pool inside your VPC for VPC-SC compatibility. The default shared worker pool won’t work. Private pools cost more, take longer to provision, and have different machine type options. Budget for this.

Vertex AI

Vertex AI is a constellation of services, and each one has its own VPC-SC considerations.

Custom training jobs execute in Google-managed infrastructure. The training container needs access to your Cloud Storage bucket for data and model artifacts. If your bucket is in a VPC-SC perimeter, the training job needs to be configured with a VPC network and a service account that has the right perimeter access. The Vertex AI VPC-SC documentation covers the basics, but the interaction between Vertex AI, the perimeter, the VPC network, and the service account permissions is a matrix of configuration that’s easy to get wrong.

Vertex AI Pipelines has an additional wrinkle: each pipeline step runs as a separate job, potentially with a different service account. If your pipeline has steps that access different services — BigQuery, Cloud Storage, Vertex AI Model Registry — each of those services needs to be included in the perimeter, and the service account for each step needs the right access.

Pre-trained models and AutoML features that download model artefacts from Google’s infrastructure: blocked by default. You need specific ingress rules to allow Vertex AI’s internal services to bring data into the perimeter.

Pub/Sub cross-project subscriptions

If Project A publishes to a Pub/Sub topic and Project B has a subscription, and they’re in different perimeters: blocked. Even if the service accounts have the right IAM permissions. VPC-SC operates at a layer above IAM — you can have permission to read the data and still be blocked from reading it because the perimeter says no.

BigQuery cross-project queries

Same story. A federated query that reads from a dataset in another project, even within the same organisation, will fail if the perimeter configuration doesn’t explicitly allow it. BigQuery is one of the most common sources of VPC-SC violations in my experience, because analysts write queries that join across datasets without thinking about which project each dataset lives in.

The error messages

This is where I lose my patience.

A VPC-SC violation produces an error like:

Request is prohibited by organization's policy.
vpcServiceControlsUniqueIdentifier: <some-uuid>

Which policy? Which rule? What specifically about this request was prohibited? The error message doesn’t say. You get a UUID. You take that UUID to the Cloud Audit Logs. You search for it. You find the VPC_SC_VIOLATION log entry. You read the violationInfo field. And there, buried in the JSON, you find the service, the method, the resource, and the violation reason.

This debugging cycle takes 5-10 minutes per violation. When you’re setting up a new service and hitting twenty violations in an hour, it’s maddening. There is no faster path. The error messages at the API level are deliberately opaque (for security reasons, I assume — you don’t want to leak perimeter configuration to potentially malicious callers). But for legitimate developers trying to configure things correctly, it’s painful.

Access levels and ingress/egress rules

The perimeter isn’t binary. You can define granular access through several mechanisms.

Access levels

Access levels define conditions under which access to the perimeter is allowed. Conditions can include:

IP address ranges (your corporate VPN CIDR, for example)
Device policy (managed devices only, specific OS versions)
Identity (specific users or service accounts)
Geographic region

Access levels are defined in the Access Context Manager and attached to the perimeter. They’re useful for allowing developer access from corporate networks while blocking everything else.

Ingress and egress rules

For service-to-service communication, ingress and egress rules are the primary tool. They specify:

Ingress: what external identities can access what services inside the perimeter
Egress: what internal identities can access what services outside the perimeter

The configuration looks like this in Terraform:

resource "google_access_context_manager_service_perimeter" "main" {
  parent = "accessPolicies/${var.access_policy_id}"
  name   = "accessPolicies/${var.access_policy_id}/servicePerimeters/main"
  title  = "main"

  status {
    resources = [
      "projects/${var.project_number}"
    ]
    restricted_services = [
      "bigquery.googleapis.com",
      "storage.googleapis.com",
      "aiplatform.googleapis.com",
      # ... many more
    ]

    ingress_policies {
      ingress_from {
        identity_type = "ANY_IDENTITY"
        sources {
          access_level = google_access_context_manager_access_level.corp_vpn.name
        }
      }
      ingress_to {
        resources = ["*"]
        operations {
          service_name = "bigquery.googleapis.com"
          method_selectors {
            method = "*"
          }
        }
      }
    }

    egress_policies {
      egress_from {
        identity_type = "ANY_IDENTITY"
      }
      egress_to {
        resources = ["projects/${var.shared_services_project_number}"]
        operations {
          service_name = "artifactregistry.googleapis.com"
          method_selectors {
            method = "*"
          }
        }
      }
    }
  }
}

This is the simplified version. A real perimeter in a banking environment has dozens of ingress and egress rules, each with specific service and method restrictions. Managing this in Terraform is the only sane approach — doing it via the console or gcloud would be error-prone and un-auditable.

Dry-run mode

Dry-run mode is essential. It lets you apply a perimeter configuration without actually enforcing it — violations are logged but not blocked. Use it. Always.

The workflow:

Define your perimeter in dry-run mode
Deploy your services
Monitor the audit logs for dry-run violations
Adjust ingress/egress rules based on violations
Repeat until clean
Switch to enforced mode

Steps 3-5 will take longer than you think. Every new service, every new integration, every new pipeline step is a potential new violation. Dry-run mode lets you discover them without breaking production.

Practical patterns

Over months of working inside VPC-SC perimeters, a set of patterns has emerged.

Private Artifact Registry for everything

Mirror every container image you need into a private Artifact Registry repository. Every base image, every third-party tool image, everything. Automate the mirroring process. Run it on a schedule. If an image isn’t in your private registry, it doesn’t exist.

This is annoying. It’s also non-negotiable. The alternative is build failures at the worst possible time because someone added a new base image to a Dockerfile without checking whether it was mirrored.

Private Google Access

Private Google Access allows VMs without external IP addresses to reach Google APIs through internal IP ranges. Combined with VPC-SC, it means your compute resources can call Google APIs (BigQuery, Cloud Storage, Vertex AI) without any public internet access.

Configure your VPC subnets with Private Google Access enabled. Use the private.googleapis.com or restricted.googleapis.com VIP ranges in your DNS configuration. The restricted.googleapis.com range (199.36.153.4/30) only allows access to services that support VPC-SC, which is an additional layer of protection.

Workload Identity Federation

No service account keys. Full stop. Workload Identity Federation allows external workloads (CI/CD pipelines, on-prem services) to authenticate to GCP without long-lived service account keys.

Inside GKE, Workload Identity maps Kubernetes service accounts to GCP service accounts. Your pods authenticate using short-lived tokens, automatically rotated, with no keys to leak.

In a VPC-SC environment, this isn’t just best practice — it’s policy. Service account keys are a data exfiltration vector. A leaked key could be used from outside the perimeter to access services inside it (depending on access level configuration). Workload Identity Federation eliminates this vector.

Dedicated proxy VMs for controlled outbound

Some workloads genuinely need outbound internet access. Pip packages that aren’t in your private PyPI mirror. API calls to external services for data enrichment. Webhook notifications.

The pattern: dedicated proxy VMs (Squid or similar) in a specific subnet, with specific firewall rules, and specific VPC-SC egress rules. All outbound traffic routes through the proxy. The proxy has an allowlist of permitted domains. Everything else is blocked.

This is belt-and-braces. The VPC-SC perimeter prevents API-level data exfiltration. The proxy prevents network-level data exfiltration. The firewall rules prevent lateral movement. Each layer catches things the others might miss.

The debugging workflow

When something fails with a VPC-SC violation, this is the process:

Find the violation UUID in the error response

Go to Cloud Logging (or use gcloud):

gcloud logging read 'protoPayload.metadata.violationReason="SERVICE_NOT_ALLOWED_FROM_VPC" OR protoPayload.metadata.violationReason="NO_MATCHING_ACCESS_LEVEL"' \
  --project=your-project \
  --freshness=1h \
  --format=json

Find the log entry matching your UUID
Read the violationInfo:
- violationReason: why it was blocked (e.g., NO_MATCHING_ACCESS_LEVEL, RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER)
- servicePerimeter: which perimeter blocked it
- resourceType and resourceName: what resource was involved
Cross-reference with your perimeter configuration
Add the appropriate ingress or egress rule
Test again

Do this enough times and you develop a sixth sense for which operations will trigger violations. New GCP service integration? Violation. Cross-project data access? Violation. Any operation that involves Google-managed infrastructure accessing your resources? Probably a violation.

The audit logs are your single source of truth. Not the documentation, which is often incomplete for edge cases. Not the error messages, which are deliberately uninformative. The audit logs.

The honest take

VPC Service Controls is the right tool for organisations that genuinely need data exfiltration prevention. Banks, healthcare providers, government agencies, defence contractors — if you handle data where unauthorised exfiltration is a regulatory violation or an existential risk, VPC-SC is not optional.

But it has costs. Real, significant costs that most teams underestimate.

Development time: Every new service, every new integration, every new pipeline needs VPC-SC configuration. What takes an afternoon outside a perimeter takes two or three days inside one. Over the lifecycle of a platform, this adds up to months of additional engineering effort.

Operational complexity: The perimeter configuration is another piece of critical infrastructure to manage, version, review, and test. Terraform helps, but the blast radius of a misconfigured perimeter is “everything stops working.”

Team knowledge: VPC-SC requires deep GCP knowledge that many teams don’t have. Understanding which services need which API access, how data flows between services, and how perimeter rules interact with IAM is specialist knowledge. Budget for training, or budget for consultants who’ve done this before.

Design from day one: This is the most important point. If you know your environment will require VPC-SC, design for it from the start. Choose architectures that minimise cross-perimeter data movement. Use private connectivity from the beginning. Mirror container images from day one. Build your CI/CD pipelines with private worker pools from the start.

Bolting VPC-SC onto an existing platform that was designed without it is, in my experience, a rewrite. Not in theory. In practice. I have watched teams spend six months retrofitting VPC-SC onto a platform that took three months to build. The ratio doesn’t get better with scale.

VPC-SC is powerful. VPC-SC is painful. VPC-SC is necessary. Plan accordingly.