From KubeFlow to Vertex AI: The Migration Nobody Warns You About

I have now been through two KubeFlow builds and one KubeFlow-to-Vertex-AI migration, and I can report with confidence that the migration is worse than either of the builds.

The first KubeFlow deployment was at an AI consultancy — a full platform build on GKE, KubeFlow 1.x, all the bells and whistles. Notebooks, pipelines, KFServing, Katib for hyperparameter tuning. It worked. Mostly. When someone remembered to restart the Istio gateway after a node pool rotation.

The second was at a century-old industrial manufacturer. Different scale, same technology, same fundamental experience of data scientists emailing me at 3pm on a Friday because their notebook server had vanished. KubeFlow on GKE is powerful, but “powerful” and “manageable” are not synonyms.

By early 2023, Google was making the pitch hard: Vertex AI is the managed future. Stop running your own KubeFlow cluster. Let us handle the infrastructure. You focus on the models. And honestly? For most teams, they’re right. But the distance between “you should migrate” and “you have migrated” is measured in months and scars, not sprints.

What maps cleanly (the good news)

I’ll start positive, because there genuinely is a reasonable migration story for one specific component: pipelines.

KFP SDK v2 to Vertex AI Pipelines

The KubeFlow Pipelines SDK v2 was explicitly designed with Vertex AI Pipelines as a target. If you’ve already migrated from the KFP v1 SDK to v2 — which is itself a non-trivial refactor — your pipeline definitions mostly translate. The @component decorator, the @pipeline decorator, the compiled JSON/YAML output: it all works.

The key change is the runner. Instead of:

from kfp.client import Client
client = Client(host='https://your-kubeflow-endpoint')
client.create_run_from_pipeline_package('pipeline.yaml')

You switch to:

from google.cloud import aiplatform
aiplatform.init(project='your-project', location='europe-west2')
job = aiplatform.PipelineJob(
    display_name='my-pipeline',
    template_path='pipeline.yaml',
    pipeline_root='gs://your-bucket/pipeline-root',
)
job.run(service_account='your-sa@your-project.iam.gserviceaccount.com')

That part is genuinely straightforward. If your pipeline is pure Python components with well-defined inputs and outputs, you might get it running on Vertex AI Pipelines in a day or two.

The catch — and there’s always a catch — is that most real-world KFP pipelines aren’t pure Python. They have custom container components, they reference cluster-local services, they mount PVCs for intermediate storage, they assume network access patterns that don’t exist in Vertex AI’s execution environment. More on that shortly.

Artifact tracking

KubeFlow’s ML Metadata (MLMD) store maps reasonably well to Vertex AI’s Metadata and Artifact Registry integration. The concepts are the same: track inputs, outputs, metrics, and lineage across pipeline runs. The APIs are different, but the mental model translates.

What doesn’t map cleanly (the reality)

Notebook servers → Vertex AI Workbench

KubeFlow notebook servers are Kubernetes pods. Your data scientists pick an image, get a JupyterLab instance, and it runs on your GKE cluster with access to whatever your cluster has access to. PVCs for persistent storage. Kubernetes RBAC for access control. Direct network access to cluster services.

Vertex AI Workbench instances are Compute Engine VMs. Different auth model (IAM, not K8s RBAC). Different persistence story (persistent disk attached to the VM, not a PVC). Different networking (VPC-native, not cluster-native). Different lifecycle management.

The migration isn’t “point your data scientists at a new URL.” It’s “redesign your notebook environment.” Every custom Docker image you built for KubeFlow notebook servers needs to be rebuilt or replaced. Every workflow that depended on in-cluster access to databases, model registries, or feature stores needs a new networking path.

I spent three weeks on notebook migration alone at the industrial manufacturer. Three weeks. For what was supposed to be the easy part.

KFServing → Vertex AI Endpoints

This is where it gets genuinely painful.

KFServing (now KServe) gives you a Kubernetes-native model serving layer. You define an InferenceService, point it at a model artifact, and KServe handles the REST/gRPC endpoint, autoscaling, canary deployments, and traffic splitting. It runs on your cluster, scales to zero if you want, and you control the entire networking stack.

Vertex AI Endpoints are a different beast. You build a custom container that conforms to Vertex AI’s container requirements — health check on a specific route, prediction on a specific route, specific environment variable handling. The scaling model is different: you define machine types, min/max replicas, and traffic split percentages. There’s no scale-to-zero (as of writing — they keep promising it). The cold start characteristics are different. The autoscaling triggers are different.

If you had a KFServing setup with custom transformers, explainers, and multi-model serving, you’re essentially rebuilding your serving infrastructure from scratch. The model artifacts might be the same. Everything around them changes.

Katib → Vertex AI Vizier

Katib is KubeFlow’s hyperparameter tuning component. You define a search space, an objective metric, and a trial template. Katib runs experiments as Kubernetes jobs and reports results.

Vertex AI Vizier is conceptually similar but operationally different. Different API. Different SDK. Different way of defining search spaces, suggesting trials, and reporting measurements. It’s not a config file migration; it’s a rewrite of your tuning workflows.

To be fair, most teams I’ve worked with barely used Katib. Data scientists tend to prefer running their own hyperparameter sweeps in notebooks using Optuna or Ray Tune. But if you had Katib integrated into your pipelines, budget time for the Vizier migration.

The hidden costs

Per-run pricing

This is the one that catches every finance team off-guard.

KubeFlow on GKE: you pay for the GKE cluster. The nodes. The storage. Whether you run one pipeline or a thousand, the infrastructure cost is roughly the same. (Yes, autoscaling exists, but in practice most teams run a fairly stable node pool.)

Vertex AI Pipelines: you pay per pipeline run. Per step. Per vCPU-hour and GB-hour of each step. For a team running dozens of pipeline runs a day — training experiments, data validation, feature engineering — the Vertex AI bill can be significantly higher than the GKE bill it replaced.

At the industrial manufacturer, the first month’s Vertex AI Pipelines bill was roughly 3x what the GKE cluster cost. We spent two weeks optimising: reducing step resource requests, consolidating small steps, caching aggressively. Got it down to about 1.5x. Still more expensive, but within the acceptable range for a managed service. This is the kind of thing that should be in the migration planning doc, not discovered in the first invoice.

Custom training container requirements

Vertex AI custom training jobs need containers that meet specific requirements. The container must be in Artifact Registry (not an arbitrary registry). The entrypoint must handle specific environment variables for output paths. If you’re using GPUs, you need the right CUDA base images from the approved list.

If your KubeFlow training jobs used arbitrary Docker images from your own registry with custom entrypoints, you’re rebuilding containers.

VPC Service Controls compatibility

If you’re operating inside a VPC-SC perimeter — and if you’re a large enterprise on GCP, you probably are — Vertex AI services need specific ingress and egress rules in the perimeter configuration. The documentation for this is, charitably, incomplete.

I found myself reading GitHub issues and Stack Overflow threads more than official docs for VPC-SC + Vertex AI compatibility. Things that should work according to the docs don’t. Services that aren’t listed in the VPC-SC supported services page are actually required by Vertex AI internally. The aiplatform.googleapis.com API needs access to storage.googleapis.com, containerregistry.googleapis.com, artifactregistry.googleapis.com, and several others, and the egress rules need to be specific.

We lost a full week to VPC-SC issues during the migration. A full week of reading audit logs, adjusting perimeter rules, testing, and swearing.

The team impact

This doesn’t get enough attention in migration planning documents.

Your data scientists and ML engineers learned KubeFlow. They learned its quirks, its failure modes, its workarounds. They know that when a pipeline step fails with an OOM error, you need to check the node pool autoscaling, not the step resource request. They know that KFServing’s canary rollout sometimes stalls and you need to check the Knative revision. They have tribal knowledge.

Vertex AI has different quirks. Different failure modes. Different workarounds. The debugging workflow is different (Cloud Logging instead of kubectl logs). The deployment workflow is different (gcloud commands or SDK instead of kubectl apply). The mental model is different (managed service with opaque internals instead of Kubernetes resources you can inspect).

Budget three months for the team to become productive with Vertex AI. Not because it’s harder — it’s arguably easier once you know it — but because replacing tribal knowledge takes time. And during that three months, your pipeline reliability will dip, your deployment velocity will slow, and your data scientists will be grumpy.

What Google’s consultants estimated vs. what actually happened

Google Consulting Services scoped the migration at 2-3 sprints. Four to six weeks. For a mature KubeFlow platform with dozens of pipelines, multiple model serving endpoints, integrated notebook environments, and VPC-SC constraints.

The actual migration took closer to nine months. Here’s where the time went:

• Pipeline migration: 6 weeks. Mostly straightforward, but custom components needed rework, and the VPC-SC issues added two weeks.
• Notebook environment migration: 3 weeks. New images, new auth, new persistence, new networking.
• Model serving migration: 8 weeks. KFServing to Vertex AI Endpoints was effectively a rebuild. Custom containers, new scaling configuration, new monitoring.
• Hyperparameter tuning: 2 weeks. Katib to Vizier rewrite.
• CI/CD pipeline updates: 3 weeks. Every pipeline that deployed to KubeFlow needed to deploy to Vertex AI instead.
• VPC-SC fixes: 2 weeks (spread across the entire migration).
• Team enablement: Ongoing, but the first 4 weeks were dedicated training and pair programming.
• Testing, validation, parallel running: 6 weeks. Running both platforms simultaneously to validate results matched.

That’s roughly 30 weeks of effort. Call it 7-8 months with holidays, illness, and the inevitable discovery of undocumented pipelines that someone built two years ago and nobody knew about.

The honest assessment

Vertex AI is better than self-hosted KubeFlow for most teams. I believe that genuinely. Managed services exist for a reason, and that reason is that most organisations shouldn’t be in the business of operating Kubernetes clusters for ML workloads. The operational burden of KubeFlow — upgrades, node pool management, Istio, storage, GPU drivers — is real and ongoing and expensive in human time.

But the migration is not a sprint. It’s not a quarter. It’s six to twelve months of real work for a mature platform, and anyone who tells you otherwise is selling something.

Plan for it. Budget for it. And for the love of all that is holy, do a spike on VPC-SC compatibility before you commit to a timeline.