· Charlie Holland · DevOps · 12 min read
Running the Elastic Stack on Kubernetes and Wishing You Hadn't
ECK just went GA and we're running Elasticsearch on Kubernetes in production. It works. It's also the most operationally demanding thing in the entire cluster, and we spend more time babysitting it than everything else combined.
ECK — Elastic Cloud on Kubernetes — went GA with version 1.1 about a month ago. We’ve been running it since the late betas at a major media conglomerate, where it’s handling logging and observability for a data platform that processes content from multiple global media properties. Elasticsearch, Kibana, Filebeat, and a handful of Logstash pipelines, all running on EKS.
It works. I want to be clear about that. ECK is a well-built Kubernetes operator. The Elastic team has done a genuinely impressive job of encoding Elasticsearch operational knowledge into a controller. But “it works” and “you should do this” are very different statements, and I’ve spent enough 3am mornings staring at shard allocation logs to have strong opinions about the gap between them.
This is a post about what running stateful workloads on Kubernetes actually looks like. Not the demo. Not the conference talk. The reality.
Why you’d do it
The pitch for running Elastic on Kubernetes is straightforward, and it’s not wrong:
Single platform. If everything else runs on Kubernetes, having Elastic there too means one set of tools, one deployment model, one monitoring approach, one team that knows one thing. The alternative — managing a separate fleet of EC2 instances for Elastic while everything else is containerised — feels wrong. It’s more infrastructure to manage, more deployment pipelines, more operational surface area.
Kubernetes-native operations. ECK gives you custom resources (Elasticsearch, Kibana, Beat) that you manage like any other Kubernetes object. kubectl get elasticsearch shows you your cluster health. GitOps workflows apply naturally. Changes go through the same review process as application deployments.
Self-service for dev teams. Need a new Kibana space? A new index template? A Filebeat configuration for your new service? It’s YAML in a repo, reviewed by the team, deployed by ArgoCD. No tickets, no waiting for the “Elastic admin” to click through the UI.
These are real benefits. They’re why we chose ECK. And for a while, in the honeymoon period when the cluster was small and the data volume was modest, it felt like the right call.
Then production happened.
Storage: the foundation of all your problems
Elasticsearch is, at its core, a storage system. It writes data to disk, reads it back, and everything about its performance is ultimately determined by I/O characteristics. Running it on Kubernetes means running it on PersistentVolumeClaims, which means running it on whatever your cloud provider’s block storage offers, which means learning more about EBS volume types than you ever wanted to know.
The IOPS problem
We run on AWS, which means our storage is EBS. The default gp2 volume type gives you a baseline of 3 IOPS per GB, bursting to 3,000 IOPS. That sounds fine until you do the maths.
A 500GB data node gets 1,500 baseline IOPS. During normal ingestion — which for us is around 50,000 events per second across the cluster — that’s adequate. During a shard rebalance, a snapshot, or a force merge? The IOPS burst budget depletes in minutes, and then you’re at baseline, and your ingestion pipeline backs up, and Filebeat starts reporting 429 Too Many Requests from Elasticsearch, and your dashboards go stale, and someone sends a Slack message asking why the logs are delayed.
The fix is io1 or io2 volumes with provisioned IOPS, which work beautifully but cost three to five times more than gp2. Or gp3, which AWS launched in December — wait, no, that hasn’t happened yet as I write this. We’re stuck with gp2 or io1.
Edit: we’ve moved our hot nodes to io1 with 5,000 provisioned IOPS each. The cost increase was significant. The 2am pages stopped. I’ll let you decide which matters more.
StorageClasses and the hot-warm-cold architecture
Elasticsearch best practice for logging workloads is a hot-warm architecture: recent data lives on fast storage (hot nodes), older data migrates to cheaper storage (warm nodes), and eventually data gets deleted or archived.
On Kubernetes, this means different StorageClasses for different node types:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: elastic-hot
provisioner: kubernetes.io/aws-ebs
parameters:
type: io1
iopsPerGB: "10"
fsType: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: elastic-warm
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
fsType: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumerThen each node set in the ECK spec references its StorageClass. This works, but it means you’re managing storage tiers through Kubernetes abstractions, and when something goes wrong — a volume that can’t attach because it’s in the wrong AZ, a PVC that can’t expand because the CSI driver doesn’t support it — you’re debugging at the intersection of Kubernetes storage, AWS EBS, and Elasticsearch, which is a particularly unpleasant Venn diagram.
Volume expansion: will it, won’t it
Your Elasticsearch data nodes will need more storage. This is a certainty, not a possibility. When that day comes, you’ll want to expand the PVCs. Whether you can depends on:
- Your StorageClass having
allowVolumeExpansion: true - Your CSI driver supporting expansion (the in-tree AWS EBS provisioner does; some CSI drivers don’t, or only support offline expansion)
- The filesystem supporting online resize (ext4 does, eventually, if the stars align)
We’ve had volume expansions work flawlessly and we’ve had them hang, requiring manual intervention to restart the node, let it mount the volume, and then trigger the filesystem resize. The Kubernetes documentation on volume expansion is honest about the beta status and caveats. If you’re planning to rely on this, test it thoroughly in a non-production environment first. We didn’t, and learned the hard way.
JVM tuning: a container-shaped nightmare
Elasticsearch runs on the JVM, and the JVM inside a container is a special kind of challenge. The fundamental tension is between what the JVM thinks it has, what the container limits allow, and what Elasticsearch actually needs.
The memory equation
Elasticsearch’s official guidance is: set the JVM heap (-Xms and -Xmx) to no more than 50% of available memory, and no more than ~30GB (to stay within the compressed ordinary object pointers threshold). The other 50% is for the filesystem cache, which Lucene relies on heavily.
In a container context, “available memory” means the container’s memory limit. So if your container has a 32GB memory limit, you’d set the heap to 16GB. Simple.
Except Elasticsearch also uses off-heap memory — for network buffers, thread stacks, direct byte buffers, the JVM itself. If your heap is 16GB and your container limit is 32GB, the off-heap usage might push total memory consumption to 33GB. The OOM killer has entered the chat.
We had this exact scenario. Data nodes set with a 32GB memory limit and a 16GB heap. Under heavy indexing load, off-heap usage grew, total RSS exceeded the limit, and Kubernetes killed the pod. At 3am. Repeatedly. The pod would restart, rejoin the cluster, start recovering shards (which is I/O intensive, which slowed everything else), and then get OOM-killed again thirty minutes later.
The fix:
containers:
- name: elasticsearch
resources:
requests:
memory: 34Gi
cpu: "4"
limits:
memory: 36Gi
cpu: "8"
env:
- name: ES_JAVA_OPTS
value: "-Xms16g -Xmx16g"That 4GB of headroom between heap and container limit is the tax you pay for running a JVM in a container. It’s not in any tutorial. You discover it the hard way.
GC pauses and the frozen node
Long garbage collection pauses can cause Elasticsearch to miss heartbeats, which causes the master to mark the node as having left the cluster, which triggers shard reallocation, which is I/O intensive, which slows GC further. It’s a death spiral.
We tune -XX:+UseG1GC explicitly and set -XX:G1HeapRegionSize=16m for our larger heaps, following guidance from various Elastic forum posts and this blog on GC tuning. Monitoring GC pause times through the node stats API is something I’d consider essential, not optional.
Node topology: YAML for days
A production Elasticsearch cluster isn’t just “some nodes.” It’s a carefully designed topology of different node roles, each with different resource requirements and failure domains.
For our setup, we run:
- 3 master-eligible nodes — small instances (4GB RAM, 2 CPU), dedicated to cluster management. Odd number for quorum.
- 6 data-hot nodes — large instances (36GB RAM, 8 CPU, io1 storage), handling recent data and all ingest.
- 4 data-warm nodes — medium instances (16GB RAM, 4 CPU, gp2 storage), holding older indices.
- 2 coordinating nodes — medium instances that handle search routing and aggregation, keeping load off data nodes.
- Kibana — 2 replicas behind a load balancer.
Each node type needs:
- Its own
nodeSetin the ECKElasticsearchspec - Specific resource requests and limits
- Anti-affinity rules to spread across availability zones
- Tolerations if using dedicated node pools (which you should)
- The correct StorageClass reference
- Environment variables for JVM settings
The full ECK manifest for this setup is 247 lines of YAML. Two hundred and forty-seven lines before we’ve configured a single index, ILM policy, or ingest pipeline. Here’s just the hot data nodes section to give you a flavour:
- name: data-hot
count: 6
config:
node.roles: ["data_hot", "data_content", "ingest"]
node.attr.data: hot
podTemplate:
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: logging
elasticsearch.k8s.elastic.co/statefulset-name: logging-es-data-hot
topologyKey: topology.kubernetes.io/zone
tolerations:
- key: "workload"
operator: "Equal"
value: "elastic"
effect: "NoSchedule"
containers:
- name: elasticsearch
resources:
requests:
memory: 34Gi
cpu: "4"
limits:
memory: 36Gi
cpu: "8"
env:
- name: ES_JAVA_OPTS
value: "-Xms16g -Xmx16g -XX:+UseG1GC"
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Gi
storageClassName: elastic-hotMultiply that by five node types and you start to understand why the full manifest is a small novel.
Upgrades: the shard shuffle
ECK handles Elasticsearch upgrades as rolling restarts — it takes down one node at a time, waits for the cluster to return to green, then proceeds to the next. In theory, this is correct and safe.
In practice, every time a data node goes down, Elasticsearch starts relocating its shards to other nodes. This is deliberate — it’s maintaining the replication factor. But shard relocation is I/O intensive. Every relocated shard is being copied across the network. While this is happening, query performance degrades because the remaining nodes are busy copying data instead of serving searches.
For a large cluster with hundreds of shards, an upgrade that “should” take thirty minutes takes three hours because each node replacement triggers a shard relocation storm that has to complete before the next node can be taken down.
The mitigation is to disable shard allocation before the upgrade and re-enable it after each node restarts:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "primaries"
}
}ECK doesn’t do this automatically. You can use pre-stop hooks or init containers to manage it, but it’s manual configuration that you need to know about and implement. The ECK documentation covers this, but it’s easy to miss, and the default behaviour — let Elasticsearch relocate shards freely during the upgrade — will make your upgrade slow and your cluster unhappy.
The alternative you don’t want to hear
Here’s the uncomfortable truth. Elastic Cloud exists. The managed service. Someone else runs Elasticsearch for you. Someone who knows Elasticsearch intimately, because they wrote it. They handle the JVM tuning, the shard allocation, the upgrades, the storage tiering, the monitoring, the lot.
At the media conglomerate, we did the maths. Our ECK setup consumed roughly 30% of the cluster’s compute capacity — the Elastic node pools, the storage costs, the additional monitoring overhead. We estimated the engineering time spent managing Elastic on Kubernetes at roughly two days per week across the team — configuration changes, debugging issues, capacity planning, handling the occasional crisis.
Two days per week of senior engineering time. That’s not free. If your engineers cost the business what senior engineers in London cost, that’s several thousand pounds a week in engineering time, plus the compute and storage costs of running it yourself.
Elastic Cloud for equivalent capacity would cost… less. Not dramatically less if you’re only counting the cloud bill, but once you add the engineering time, it’s not close.
The same is true of Amazon Elasticsearch Service (though note the recent trademark dispute and Elastic’s licence change — that’s a separate drama). The managed service exists. It’s not perfect. But the things it’s not perfect at are things you can live with. The things self-managed Elastic on Kubernetes isn’t perfect at are things that wake you up at 3am.
When self-managed makes sense
I’m not saying never run Elastic on Kubernetes. There are legitimate reasons:
- Data residency — you need Elasticsearch in a specific region or network that the managed service doesn’t cover.
- Customisation — you need plugins, configurations, or topologies that the managed service doesn’t support.
- Cost at extreme scale — if you’re running multiple petabytes, the managed service pricing gets eye-watering and self-managed starts to make sense economically even with the engineering overhead.
- You already have a platform team — if you’ve already invested in a Kubernetes platform with sophisticated operations, the marginal cost of adding Elastic is lower than building that capability from scratch.
We tick the last two boxes, which is why we’re still running ECK. But if we were starting today, knowing what I know about the operational burden, I’d push hard for the managed service and spend the engineering time on things that actually differentiate the product.
The broader lesson
This isn’t really about Elasticsearch. It’s about the seductive idea that Kubernetes can run everything and therefore should run everything.
Kubernetes is excellent at stateless workloads. HTTP APIs, workers, batch jobs — things that can be killed and restarted without consequence. The reconciliation model works brilliantly when your workloads are fungible.
Stateful workloads — databases, search engines, message brokers, anything with data on disk that matters — are a different game. Kubernetes can run them. ECK, the PostgreSQL operators, the various Kafka operators — they all work. But “works” isn’t the bar. The bar is “works well enough that the total cost of ownership, including engineering time, is lower than the alternatives.”
For most teams, for most stateful workloads, the managed service wins that calculation. Not because Kubernetes can’t do it, but because the operational complexity of doing it well is higher than people expect, and the engineering time has a cost that doesn’t show up on the cloud bill.
Run your stateless workloads on Kubernetes. Run your stateful workloads on managed services. And if someone tells you it’s simple to run Elasticsearch on Kubernetes, ask them who’s on call.
