Istio: The Service Mesh Nobody Asked For

I’ve just come off a stint at a major media conglomerate. Big Kubernetes estate. Hundreds of microservices. And right there in the middle of the stack, like a tumour wrapped in YAML: Istio.

Nobody could tell me why it was there. Not really. When I asked the team — smart people, good engineers — the answers were variations of “it gives us observability” and “we need mTLS between services.” When I dug further, it turned out the observability they were using was Prometheus metrics they could have scraped without a sidecar, and the mTLS requirement came from a security review that had since been superseded by a network policy approach.

Istio was there because someone put it there. And nobody had the energy or the authority to take it out.

The promise

If you haven’t had the pleasure, Istio is a service mesh for Kubernetes. The pitch is compelling, at least on a conference slide:

Observability. Automatic distributed tracing, metrics, and logging for every service-to-service call. No code changes required.

Traffic management. Canary deployments, A/B testing, circuit breaking, fault injection, traffic mirroring. Fine-grained control over how requests flow through your system.

Security. Mutual TLS between all services, automatic certificate rotation, fine-grained access policies. Zero-trust networking without touching application code.

Policy enforcement. Rate limiting, access control, quota management. All declarative, all in the mesh.

It sounds brilliant. It sounds like the answer to every microservices problem you’ve ever had. And if you’re running a system at the scale and complexity of Google’s internal infrastructure — which is where the Envoy proxy at Istio’s heart came from — it might actually be.

But you’re not running Google’s infrastructure. Almost nobody is.

The reality

Here’s what Istio actually looked like in every organisation I’ve seen deploy it:

Resource overhead. Every pod gets an Envoy sidecar. Every single one. That’s an extra container consuming CPU and memory for every instance of every service in your cluster. At the media conglomerate, we estimated the sidecar fleet was consuming roughly 15% of total cluster resources. Fifteen percent of their compute bill, just for the mesh. For context, the Istio documentation itself acknowledges that each sidecar proxy uses around 0.5 vCPU and 50MB of memory at moderate load. Multiply that by a few hundred pods and you’re paying serious money for the privilege of routing traffic through a proxy.

Latency. Every service-to-service call now goes through two proxies — one on the sender, one on the receiver. Istio’s own performance benchmarks show added latency in the single-digit milliseconds range per hop. That sounds trivial until you have a request that traverses eight services. Now you’ve added 50-80ms of latency that didn’t exist before. For some workloads, that’s nothing. For others, it’s the difference between a snappy user experience and a sluggish one.

Operational complexity. Istio has its own control plane — Istiod (formerly Pilot, Mixer, Citadel, and Galley, because even the architecture couldn’t stop renaming things). It needs upgrading separately from your cluster. It has its own CRDs, its own configuration model, its own failure modes. When something goes wrong with service-to-service communication, you now have an additional layer to debug. Is it the application? The network? The sidecar? The control plane? The VirtualService configuration? The DestinationRule? Good luck.

Configuration sprawl. I counted the Istio-related YAML in that media company’s repository. Over twelve thousand lines. Twelve thousand lines of configuration for a system that most developers interacted with by copy-pasting a VirtualService manifest from another team and hoping for the best. Nobody understood the full configuration. Nobody could.

What they actually used it for

Here’s the bit that really gets me. After all that complexity, all that overhead, all that operational burden — what was the mesh actually doing?

Mutual TLS between services. That was it. That was the 95% use case.

Not canary deployments — they were doing blue-green deploys through their CI/CD pipeline, which worked fine. Not traffic mirroring — nobody had set it up. Not circuit breaking — the applications handled their own retry logic. Not distributed tracing — they had Jaeger, but it was configured independently of the mesh because the mesh-provided traces weren’t granular enough.

mTLS. Which, by the way, you can achieve with cert-manager and some application configuration, or with Kubernetes network policies if you’re in a trusted network, or with a dozen other approaches that don’t require bolting an entire service mesh onto your cluster.

This is a pattern I see constantly. Teams adopt a complex technology for one feature and then carry the full weight of everything else. It’s like buying a Swiss Army knife because you need a bottle opener and then wondering why your pocket is so heavy.

Resume-driven development

Let’s be honest about why Istio gets adopted. It’s not because someone did a thorough cost-benefit analysis. It’s not because the team evaluated alternatives and concluded that a full service mesh was the right tool for their specific problems. It’s because:

1. Conference talks. Every KubeCon from 2018 to 2020 was wall-to-wall service mesh content. If you went to KubeCon and came back without a service mesh on your roadmap, you weren’t paying attention (or you were, and you were the smart one in the room).

2. The CV arms race. “Experience with Istio and service mesh architecture” looks fantastic on a CV. “Decided we didn’t need a service mesh and used simpler alternatives” does not. The incentive structure rewards complexity, not appropriateness.

3. Vendor push. Every managed Kubernetes provider made it trivially easy to enable Istio. GKE had an Istio add-on. AWS had App Mesh (their own take, equally over-engineered). Azure had its own integration. When the platform makes it a checkbox, it looks like the default choice rather than a deliberate decision.

4. Fear of looking unsophisticated. If everyone else has a service mesh and you don’t, are you doing Kubernetes wrong? This is the enterprise version of keeping up with the Joneses, and it’s just as daft.

I’ve sat in architecture review meetings where the proposal was to add Istio to a cluster running twelve services. Twelve. With a team of four. When I pointed out that they could achieve their stated goals — observability and mTLS — with simpler tools, the response was essentially “but this is what modern Kubernetes looks like.” No. That’s what conference Kubernetes looks like. Production Kubernetes should look like whatever keeps the lights on without driving your team mad.

Linkerd existed

The thing that really stings is that a perfectly reasonable alternative existed the entire time. Linkerd — the other service mesh, the one that was actually designed to be simple.

Where Istio was built on Envoy (a general-purpose proxy with enormous configurability), Linkerd built its own purpose-built proxy in Rust. It was smaller, faster, and dramatically simpler to operate. The Linkerd vs Istio benchmarks consistently showed lower resource consumption and latency. The configuration model was simpler. The upgrade path was less painful.

But Linkerd didn’t have the marketing muscle. It didn’t have Google behind it. It didn’t have the conference presence. And crucially, it didn’t have the brand recognition that made architects feel like they were making a safe, defensible choice.

Nobody ever got fired for choosing Istio. Plenty of people quietly suffered because of it.

The sidecar pattern’s hidden costs

Beyond Istio specifically, the entire sidecar proxy model deserves scrutiny. The idea is elegant in theory: inject a proxy alongside every application container, and let the proxy handle cross-cutting concerns like security, observability, and traffic management. The application doesn’t need to know about any of it.

In practice:

Debugging is harder. When a request fails, you need to determine whether the failure happened in the application, the local sidecar, the remote sidecar, or the mesh control plane. Your logs now come from multiple containers per pod. Your network traces have extra hops. Your mental model of “service A calls service B” is now “service A calls local Envoy, which calls remote Envoy, which calls service B.” Every abstraction has a cost, and this one’s cost is legibility.

Startup ordering matters. The sidecar needs to be ready before the application starts making network calls. If the application boots faster than the sidecar (which happens), requests fail. Istio has workarounds for this, but the fact that you need workarounds for a sidecar interfering with application startup should give you pause.

Resource requests compound. Every pod now requests resources for two containers instead of one. Your cluster’s scheduling becomes less efficient. Your node packing gets worse. Your autoscaler makes decisions based on inflated resource numbers.

It’s worth noting that even the Istio project recognised these problems and introduced an ambient mesh mode that moves away from sidecars entirely. When the project itself starts backing away from its core architectural pattern, that tells you something.

What you should do instead

If you’re considering a service mesh in 2020 — and I know plenty of you are, because it’s on every architect’s FOMO list — here’s my actual advice:

Start with the problem, not the solution. What specifically do you need? mTLS? Use cert-manager or your cloud provider’s service identity. Observability? Use OpenTelemetry libraries in your applications — you’ll get better traces anyway. Traffic management? Your ingress controller probably handles 90% of what you need. Canary deployments? Argo Rollouts or Flagger work without a full mesh.

Count your services. If you have fewer than fifty services, you almost certainly don’t need a service mesh. The overhead isn’t worth it. If you have fewer than twenty, it’s actively harmful.

Count your team. A service mesh needs someone to own it. If you don’t have at least one person (ideally two) who deeply understand the mesh and can troubleshoot it, you’re creating a system that nobody can maintain. “The cluster team handles it” is not an ownership model. It’s a prayer.

Try Linkerd first. If you genuinely need mesh capabilities, start with the simpler option. You can always move to Istio later if Linkerd doesn’t meet your needs. You can’t easily go the other way.

Accept that simple is not unsophisticated. The best architecture is the simplest one that meets your requirements. A Kubernetes cluster with an nginx ingress controller, cert-manager for TLS, and some sensible application-level retry logic is a perfectly valid, production-ready setup. It’s not glamorous. It won’t get you a conference talk. But it works, it’s debuggable, and your team can maintain it at 3am.

The real lesson

Istio isn’t bad software. It’s impressive engineering solving a genuine problem — at a specific scale that most of us will never reach. The failure isn’t in the technology. It’s in the adoption pattern.

We keep doing this in our industry. We take a tool built for a specific context — Google-scale traffic management — and we adopt it universally because it’s popular. We did it with microservices (most systems should be a modolith). We did it with Kubernetes itself (most applications should be on a managed platform). And we’re doing it with service meshes.

The pattern is always the same: a genuinely good idea, built for a specific problem, at a specific scale, gets turned into a universal “best practice” by people who confuse sophistication with appropriateness.

If you’re reading this and you have Istio in your cluster, ask yourself honestly: do you need it? Not “could we theoretically use these features someday” but “is this solving a problem we actually have, today, that justifies the complexity and cost?”

If the answer is no, rip it out. Your team will thank you. Your cluster will thank you. Your cloud bill will thank you.

And your CV will survive. I promise.