KubeFlow: Twelve Months of Pain

I’ve been running KubeFlow in production — or trying to — for about twelve months now. Across multiple client deployments at an AI consultancy, and now at a century-old industrial manufacturer where we’re attempting to industrialise ML for predictive maintenance. KubeFlow 1.2 is current. I’ve read the docs. I’ve filed the GitHub issues. I’ve stared at Argo DAGs at two in the morning wondering what I’ve done with my life.

This is a post-mortem. Not of a single incident, but of a year spent trying to make KubeFlow do what it promises on the tin: end-to-end machine learning pipelines on Kubernetes.

The short version: most teams should not be using KubeFlow. The long version follows.

Installation: choose your own adventure

Let’s start at the beginning, because the beginning is where KubeFlow first betrays you.

There is no single, canonical way to install KubeFlow. As of version 1.2, you have:

• kfctl — the original CLI installer, which was deprecated and then un-deprecated and then deprecated again. The kfctl repository has a README that reads like a warning label.
• Kubeflow manifests — raw Kustomize overlays in the manifests repository. This is now the “official” installation method, which means you clone a repo, run kustomize build on a dozen directories, and pray.
• Distribution-specific installers — AWS, GCP, Azure, and various vendors each have their own deployment mechanisms with their own quirks, their own version lag, and their own incompatibilities.

At the consultancy, we tried kfctl first. It worked on GKE with the default configuration. It did not work on our client’s locked-down AKS cluster because kfctl made assumptions about RBAC, storage classes, and Istio installation that didn’t hold. We spent three days debugging before switching to the manifests approach, which at least let us see what was being deployed and modify it.

The manifests approach is “bring your own Kustomize.” You clone the repo, check out the version tag, and run kustomize build on each component you want. There’s a top-level kustomization.yaml that includes everything, but “everything” includes components you may not want, and the dependency graph between components is documented approximately nowhere. Install KFServing without Istio? Good luck — it’ll create CRDs that reference Istio VirtualServices that don’t exist, and you won’t find out until you try to deploy a model.

I’ve installed KubeFlow from scratch at least six times across different clients. It has never worked on the first attempt. The average was three attempts, with failures caused by: Istio version mismatches, missing CRDs, storage class assumptions, cert-manager conflicts with existing cert-manager installations, and my personal favourite — a Kustomize version incompatibility that silently generated invalid YAML.

Component sprawl: the buffet nobody ordered

KubeFlow is not one product. It’s a collection of loosely related components that share a name and a website. As of 1.2, the lineup includes:

• KubeFlow Pipelines (KFP) — DAG-based ML pipeline orchestration, built on Argo Workflows
• Katib — hyperparameter tuning and neural architecture search
• KFServing — model serving (now evolving toward something called KServe)
• Notebook Servers — managed Jupyter notebooks on Kubernetes
• Metadata — experiment and artifact tracking
• Training Operators — TFJob, PyTorchJob, MXNetJob for distributed training
• Fairing — build and deploy from notebooks (barely maintained)
• MinIO — S3-compatible object storage (bundled as a pipeline artifact store)

Each component has its own maturity level, its own release cycle, its own bug tracker, and its own community of maintainers (or lack thereof). Pipelines and Notebooks are reasonably mature. Katib works but is niche. KFServing is undergoing an identity crisis. Fairing appears to have been abandoned. Metadata exists but I’ve never met anyone who uses it intentionally.

The problem is that KubeFlow installs all of this by default. You wanted ML pipelines? Here’s also a hyperparameter tuner, a model server, a notebook platform, a metadata store, and a MinIO instance. Each one consumes cluster resources, creates CRDs, and adds to the operational surface area.

Which brings me to the resource question.

Resource consumption: the elephant in the cluster

Before you run a single pipeline, KubeFlow itself consumes approximately 8-12GB of RAM and 4-6 CPU cores across its system pods. This is for the platform components only — Istio (the ingress and service mesh layer), Argo (the workflow engine), the KFP API server, the KFP UI, the metadata server, MySQL (for pipeline metadata), MinIO (for artifacts), the notebook controller, the profile controller, cert-manager, and various admission webhooks.

On the AKS cluster at the industrial manufacturer, we had a 3-node cluster for the development environment. KubeFlow consumed roughly 40% of the cluster’s resources before we ran anything. The platform team looked at the Kubernetes dashboard, saw a dozen system pods they didn’t recognise consuming the equivalent of a full node, and asked me — reasonably — what the hell all this was for.

The honest answer was: most of it was for components we weren’t going to use. The team needed Pipelines and Notebooks. They did not need Katib, KFServing, the training operators, or Fairing. But removing components from a KubeFlow installation is not straightforward. The Kustomize manifests have interdependencies. Istio is assumed by multiple components. Pulling one piece out risks breaking the others.

We eventually built a minimal installation that included only Pipelines, Notebooks, and their dependencies. It took a week of trial and error to figure out the minimal viable set. There is no documentation for “I just want Pipelines.”

KubeFlow Pipelines: the core product

KFP is the component most teams actually want. You define ML workflows as directed acyclic graphs (DAGs) — data preprocessing, training, evaluation, deployment — and KFP orchestrates them as Argo Workflows on Kubernetes.

The KFP SDK lets you define pipelines in Python. You write functions, decorate them with @component, and connect them into a pipeline. The SDK compiles this into an Argo Workflow YAML that KFP submits to the cluster.

In principle, this is good. In practice, the SDK has issues that range from annoying to infuriating.

The compilation model

Every pipeline component runs in its own container. This means every step in your pipeline needs a container image with its dependencies. The SDK provides a @component decorator that can build lightweight containers on the fly, but for anything non-trivial — GPU dependencies, large libraries, custom packages — you’re building and maintaining Docker images for every pipeline step.

@component(
    base_image='python:3.8',
    packages_to_install=['pandas==1.2.0', 'scikit-learn==0.24.0']
)
def train_model(
    training_data: Input[Dataset],
    model: Output[Model],
    n_estimators: int = 100,
):
    import pandas as pd
    from sklearn.ensemble import RandomForestClassifier
    import pickle
    
    df = pd.read_csv(training_data.path)
    # ... training logic ...
    with open(model.path, 'wb') as f:
        pickle.dump(clf, f)

That looks clean. Now try adding a step that needs TensorFlow, CUDA drivers, and a custom C++ library. Your base_image becomes a 4GB monster, your build times are measured in minutes, and your pipeline definition is now tightly coupled to your container registry.

SDK version instability

The KFP SDK has changed its API between minor versions in ways that break existing pipelines. Between 0.5 and 1.0, the component specification format changed. The way you pass data between steps changed. The way you define pipeline parameters changed. I’ve had pipelines that worked on SDK 0.5.1 fail on 0.6.0 because a function signature changed.

The GitHub issues for KFP are a catalogue of migration pain. Issue after issue of people asking why their pipeline stopped compiling after a version bump. The response is usually “we changed the API, here’s the migration guide” — and the migration guide is a Google Doc with known gaps.

The YAML underneath

When you compile a KFP pipeline, the output is an Argo Workflow specification in YAML. For a pipeline with ten steps, this YAML can be 500+ lines. It includes container specs, volume mounts, parameter passing via files, init containers, and Argo-specific annotations.

I’ve had to debug these YAMLs directly when pipeline compilation produced something unexpected. Reading a compiled KFP pipeline YAML is harder than reading the Python that generated it. You’re debugging a code-generated artifact, which means you’re debugging two systems — the SDK and the generated output — simultaneously.

Multi-tenancy: Istio and isolation

KubeFlow’s multi-tenancy model is based on Kubernetes namespaces and Istio’s authorization policies. Each user gets a “profile” (a namespace with associated RBAC and Istio policies). Users can only see their own pipelines, notebooks, and experiments.

It sort of works. The namespace isolation is real. But the Istio dependency means:

1. Istio has to be running and healthy for any of the access control to function. Istio has its own operational overhead and failure modes. When Istio breaks — and it does — authentication and authorization break with it.

2. The profile controller that manages user namespaces is its own component with its own bugs. Creating a new profile involves creating a namespace, Istio authorization policies, RBAC bindings, and resource quotas. I’ve seen profile creation fail silently, leaving users with a namespace but no Istio policies, which means either no access or unrestricted access depending on the failure mode.

3. Shared resources — like the pipeline engine and artifact store — are accessible to all profiles. There’s no pipeline-level isolation. User A can’t see User B’s pipeline runs in the UI, but the underlying Argo Workflows and MinIO artifacts are in shared namespaces.

For an enterprise environment where the security team wants proper tenant isolation, KubeFlow’s model is insufficient. It’s UI-level isolation, not infrastructure-level isolation.

Debugging pipeline failures

When a KFP pipeline step fails, here’s the debugging journey:

1. KFP UI — shows the pipeline DAG with a red node. Click the node to see a summary. The summary says “Failed” with a truncated error message.

2. KFP step detail — click further to see the step’s inputs, outputs, and logs. The logs are often missing or incomplete because they’re streamed from the pod to MinIO, and if the pod crashed hard enough, the logs didn’t make it.

3. Argo UI — KFP runs on Argo, so you can look at the Argo Workflow for more detail. The Argo UI shows the same DAG with different metadata. Sometimes the Argo logs are more complete than the KFP logs. Sometimes they’re the same incomplete logs.

4. kubectl — eventually you kubectl logs the pod (if it still exists — completed pods get garbage collected). Or you kubectl describe pod to see events. This is where you actually find out what went wrong: OOMKilled, ImagePullBackOff, a volume mount failure, a permissions error.

Four UIs and a CLI to find out your pod ran out of memory. This is not a developer experience. This is an archaeological expedition.

The “just use Argo directly” realisation

About eight months in, I had a moment of clarity that I suspect many KubeFlow users eventually reach. I was debugging a pipeline failure, navigating between the KFP UI and the Argo UI, and I realised: everything KFP does that’s useful, it does through Argo Workflows.

Argo Workflows is a solid, well-maintained CNCF project. It orchestrates DAGs on Kubernetes. It has a UI, a CLI, an SDK, and a REST API. It handles retries, timeouts, parameter passing, artifact management, and conditional execution. It consumes a fraction of the resources that KubeFlow does.

If you strip away KubeFlow’s UI chrome and the KFP SDK’s Python-to-YAML compilation, what you’re left with is Argo Workflows plus a metadata store plus a container image build system. You can replicate the first with Argo directly, the second with MLflow, and the third with your existing CI/CD pipeline.

The value proposition of KFP over raw Argo is: a Python SDK that makes pipeline definition easier (debatable, given the compilation issues), a UI that tracks experiments and runs (useful, but not worth the platform overhead), and integration with other KubeFlow components (which you’re probably not using anyway).

What I’d recommend instead

In early 2021, if a team came to me and said “we need ML pipeline orchestration,” here’s what I’d tell them:

If you have fewer than five models in production: You don’t need a platform. You need MLflow for experiment tracking and model versioning, Airflow or Prefect for orchestration, and your existing CI/CD pipeline for deployment. This covers 80% of teams.

If you have a mature Kubernetes platform and a dedicated ML engineering team: Consider Argo Workflows directly, paired with MLflow for experiment tracking. You get DAG orchestration on Kubernetes without the KubeFlow overhead. Your ML engineers will need to write Argo Workflow specs, but they’re more readable than compiled KFP pipelines, and you have full control over what runs on your cluster.

If you’re a Google Cloud shop: AI Platform Pipelines is a managed KFP that eliminates the installation and infrastructure overhead. It’s KubeFlow as a service, which solves the worst problems (installation, resource management, upgrades). This is probably the only context where KFP makes sense — when someone else runs the platform.

If you’re on Azure: Azure Machine Learning has its own pipeline orchestration that’s tightly integrated with the Azure ecosystem. It’s not perfect, but it’s managed, and you don’t have to run Istio.

The deeper problem

KubeFlow’s fundamental issue is that it’s a Google-scale solution for a problem most organisations don’t have at Google scale. It was born out of Google’s internal ML infrastructure, donated to the CNCF, and developed by a community that’s heavy on contributions from large tech companies and light on feedback from the enterprises that are supposed to use it.

The result is a platform that’s conceptually ambitious and operationally punishing. It tries to be everything — pipeline orchestration, model serving, hyperparameter tuning, notebook management, experiment tracking — and it does none of them well enough to justify the complexity it introduces.

After twelve months, the teams I’ve worked with have either stripped KubeFlow down to just Pipelines and Notebooks (and fought constantly to keep it running), or abandoned it entirely in favour of simpler alternatives. The industrial manufacturer is now evaluating Prefect for orchestration and MLflow for tracking. The AI consultancy still runs KubeFlow on some projects but has stopped recommending it for new deployments.

The promise of end-to-end ML on Kubernetes is compelling. KubeFlow isn’t the delivery of that promise. It’s a reminder that “runs on Kubernetes” is not a product strategy, and that the distance between a CNCF sandbox project and enterprise production software is measured in years of unglamorous engineering that nobody wants to fund.

Twelve months in. Lesson learned. Moving on.