Shift-Left Security Sounds Great Until You See the Pipeline

I’ve spent the last couple of years building multi-stage CI/CD pipelines for organisations that take security seriously — or at least say they do. A global marketing agency. A major pharma company. Both running containerised workloads on Kubernetes. Both with compliance requirements that would make your eyes water. Both absolutely committed to “shifting left.”

And both, on day one of turning on the security scanners, discovered that every single build failed.

Every. Single. One.

The gospel of shift-left

If you’ve been anywhere near a DevSecOps conference, a vendor pitch, or a LinkedIn post from someone with “Security Advocate” in their title, you’ve heard the sermon. Shift left — find security issues earlier in the development lifecycle, when they’re cheaper to fix. Don’t wait until production. Don’t wait until a pen test finds something embarrassing. Catch it in the pipeline, at build time, ideally before the code even gets merged.

The concept is genuinely sound. The IBM Systems Sciences Institute has been banging on about the cost curve of defects for decades — a bug found in design costs a fraction of one found in production. Apply that logic to security vulnerabilities and you get shift-left security. Find the dodgy dependency before it gets deployed. Spot the misconfigured container before it runs in production. Catch the SQL injection before it reaches a customer.

All sensible. All correct. And almost universally botched in practice.

Day one: everything is on fire

Here’s what actually happens when you “shift left” in a real enterprise.

Someone — usually a well-meaning security architect or a newly hired DevSecOps lead — decides it’s time. The tools get chosen. Trivy for container image scanning. Maybe Snyk for dependency analysis. Perhaps Aqua Security if the budget stretches. The tools are integrated into the CI/CD pipeline. A policy is set: fail the build on any HIGH or CRITICAL CVE.

Then the first build runs.

The base image — the one that’s been running in production for eighteen months without incident — has 47 known vulnerabilities. Twenty-three of them are rated HIGH or CRITICAL. Half have no fix available because they’re in upstream packages that the maintainers haven’t patched yet. The other half are in transitive dependencies — libraries that your application doesn’t use directly but that got pulled in because something else needed them, and you can’t remove them without rebuilding half the dependency tree.

The build fails. Every developer on the team gets a red pipeline. Nobody can deploy anything.

Now watch what happens next, because this is where the theology meets reality.

The exception list: security’s dirty secret

The security team can’t have every build failing. The business needs features shipped. The developers are furious. So the exceptions start.

“Okay, we’ll suppress CVE-2021-44228 on that image because we’ve confirmed the vulnerable code path isn’t reachable in our configuration.”

Fair enough. That’s a reasonable risk-based decision.

“And we’ll suppress CVE-2022-0778 because the fix requires upgrading OpenSSL, and that breaks three other things.”

Less ideal, but pragmatic.

“And we’ll suppress everything rated HIGH in the base image because we’re going to migrate to a new base image next quarter.”

Now we’re on a slippery slope.

“Actually, just suppress all the findings from the base image. We’ll deal with it later.”

And there it is. Three weeks in, the exception list is longer than the vulnerability list. The security scan runs, finds dozens of issues, suppresses them all, and the build passes. Everyone feels safe. The dashboards are green. The compliance report says “all images scanned.” Nobody mentions that “scanned” and “secure” are very different words.

I’ve watched this exact pattern play out at two separate organisations in the last eighteen months. It’s not an anomaly. It’s the default trajectory.

The tooling sprawl problem

It gets worse when you zoom out. The security scanning landscape is a confusing mess of overlapping categories, and most organisations end up with too many tools doing too many things, none of them well integrated.

SAST — Static Application Security Testing. Scans your source code for vulnerability patterns. Tools like SonarQube, Semgrep, Checkmarx. Good at finding certain classes of bugs. Terrible at understanding context. Generates false positives like a machine gun generates noise.

DAST — Dynamic Application Security Testing. Runs against your deployed application, poking at it like an automated pen tester. Tools like OWASP ZAP, Burp Suite. Useful but slow, and you need a running environment to test against, which means it’s not very “left” at all.

SCA — Software Composition Analysis. Scans your dependencies for known vulnerabilities. This is what Snyk, Trivy, and most of the container scanning tools do. It’s the most common form of shift-left security and the one most likely to drown you in noise.

IaC scanning — checks your Terraform, CloudFormation, and Kubernetes manifests for misconfigurations. Tools like Checkov, tfsec, kube-bench. Useful, but the rules are often generic and don’t account for your actual security posture.

Most enterprises end up with at least three of these categories, from different vendors, producing different report formats, with different severity ratings for the same underlying issue. Gartner calls this “tool sprawl” and recommends consolidation. I call it a mess.

The result: developers are expected to interpret and act on security findings from four different tools, each with their own dashboard, their own severity model, and their own definition of what constitutes a “critical” issue. Most developers — reasonably — give up trying to understand any of it and just focus on making the builds go green by whatever means necessary.

The security team can’t help

Here’s the bit that nobody wants to talk about: most enterprise security teams can’t actually interpret the output of the tools they mandated.

I’ve sat in meetings where a security architect mandated Trivy scanning for all container images, and then couldn’t explain what a CVE in libexpat actually meant for the application in question. Is the vulnerable code path reachable? Is the library loaded at runtime? Does the application even parse XML? They didn’t know. They just knew the severity was CRITICAL and the policy said CRITICAL means “fix or justify.”

This isn’t a dig at security professionals — they’re often understaffed, overworked, and dealing with an attack surface that’s genuinely enormous. But mandating tools without the capacity to help teams act on the findings isn’t security. It’s compliance theatre.

The NIST Cybersecurity Framework explicitly distinguishes between identifying risks and managing them. Identification without management is just anxiety with extra steps.

What actually works

I’m not saying don’t scan. Scan everything. But be intelligent about it.

Be ruthless about base images. This is the single highest-leverage thing you can do. Stop using ubuntu:latest or node:18 as your base image. Use distroless images from Google, or Alpine-based minimal images, or — better yet — build your own hardened base images that your platform team maintains and updates. A distroless Python image has a fraction of the attack surface of a standard Debian-based one. Fewer packages means fewer vulnerabilities means fewer findings means less noise. Simple.

At the pharma company, we cut the vulnerability count by over 70% just by switching base images. No code changes. No dependency updates. Just a smaller image with less rubbish in it.

Scan at the right stages — not just build. Build-time scanning catches what’s in the image when it’s built. But images sit in registries. Registries should be scanned continuously, because new CVEs are published daily against existing packages. And runtime scanning — tools like Falco — catches things that static analysis never will: unexpected process execution, anomalous network connections, container breakout attempts.

If you’re only scanning at build time, you’re checking the locks when you install the doors and never checking them again.

Make security the platform team’s problem. This is the big one. Individual developers should not be responsible for interpreting CVE reports and making risk decisions about transitive dependencies. That’s insane. The platform team should own the base images, the scanning configuration, the exception policy, and the triage process. Developers should get clear, actionable feedback: “this build failed because your code introduced a new HIGH vulnerability in a direct dependency. Here’s what it is and here’s how to fix it.” Not: “here are 200 findings, 195 of which are in the base image that you don’t control. Good luck.”

Set sensible policies. “Fail on any HIGH or CRITICAL” sounds rigorous. In practice it’s unworkable. A better policy: fail on any NEW vulnerability introduced by the application’s own dependencies. Track base image vulnerabilities separately, on a different cadence, managed by the platform team. Distinguish between “this CVE exists in a library that’s installed” and “this CVE is exploitable in the way we use this library.” That distinction matters enormously and almost nobody makes it.

Fix the feedback loop. Developers need to see security findings in the same place they see test results — in their PR, in their IDE, in their terminal. Not in a separate security dashboard that they never check. If the finding isn’t in the developer’s workflow, it doesn’t exist as far as the developer is concerned. Tools like Snyk and Semgrep understand this and integrate directly into pull requests. Use that.

The culture problem underneath

The deepest issue with shift-left security isn’t technical. It’s cultural.

“Shift left” implies that security was previously “right” — something that happened late in the process, owned by a specialised team, applied after the fact. Shifting it left means making it everyone’s problem earlier.

But most organisations haven’t actually changed who owns security. They’ve just moved when the security team’s tools run. The developers don’t feel ownership of security outcomes. The security team doesn’t feel ownership of developer experience. And the platform team is stuck in the middle, trying to make tools work that were designed for a world where security and development were separate disciplines.

The organisations that do this well — and I’ve seen a couple — treat security the same way they treat testing. Not as a gate at the end, not as someone else’s problem, but as an integral part of how software gets built. Developers write tests because they believe in the value of testing, not because a CI gate forces them to. The same needs to be true for security.

That’s a much harder shift than adding Trivy to a pipeline. It requires investment in developer education, sensible tooling, clear ownership, and — above all — a security team that sees itself as an enabler rather than a gatekeeper.

Stop pretending

If your shift-left security implementation consists of “add scanners to the pipeline, fail on critical, manage by exception” — you haven’t shifted left. You’ve added a speed bump that everyone’s learned to drive around.

The scanners are only as useful as the organisation’s ability to act on what they find. If nobody can interpret the findings, if the exception list grows without bound, if the security team mandates tools but can’t support teams in using them — then the whole exercise is compliance theatre with a DevOps label on it.

Shift left properly or don’t bother pretending. The vulnerabilities don’t care about your dashboards.