Stop Putting Secrets in Environment Variables

The twelve-factor app told everyone to put config in the environment, and a generation of engineers read that as "put the database password in the environment", which is not quite the same sentence. Config and secrets share a delivery mechanism in that document, and the convenience is real — os.environ["DATABASE_URL"] works the same on your laptop, in CI, and in the cluster, and it ships in every base image and every framework. So the password goes in an env var, the service starts, the tests pass, and everyone moves on.

The trouble is that the environment was never a private place. It feels private because you set it and you read it back, but a process's environment is closer to a noticeboard than a safe. It is inherited, dumped, logged, and inspected by half a dozen mechanisms that nobody put there on purpose, and most of them will hand your secret to anyone who can already read a file or run a command on the box. That is a lower bar than people assume.

How an environment variable actually leaks

Start with the obvious one. On Linux, a process's environment is readable through /proc, and the kernel will show it to you for any process owned by the same user — no root required.

$ cat /proc/$(pgrep -f my-service)/environ | tr '\0' '\n' | grep -i secret
DATABASE_PASSWORD=hunter2-but-real
STRIPE_SECRET_KEY=sk_live_...

That is not an exploit. That is a documented kernel interface working exactly as intended. Anyone who can get a shell as your service user — through a deploy hook, a debugging sidecar, a compromised dependency that runs at import time — can read every secret the process started with, in one command, with no privilege escalation at all. The environment is not a boundary; it is a convenience that the rest of the system treats as public.

It gets worse once you remember the environment is inherited. Every child you spawn gets a copy. The moment your service shells out — to git, to ffmpeg, to a healthcheck script, to that one bit of legacy tooling nobody has rewritten — the secret travels with it, and now it is sitting in that process's /proc/<pid>/environ too, governed by whatever that program does with its inputs. Shell out to something that logs its environment for debugging and you have published the password to a log aggregator without ever writing a line of logging code yourself.

Then there are the dumps. A crash handler that captures process state for a core dump captures the environment with it, and core dumps have a long history of being world-readable, shipped to error-reporting services, and attached to support tickets. APM agents helpfully snapshot the environment as "context". And in a container world, docker inspect will print the environment of any container to anyone in the docker group — which, on a lot of build hosts, is everyone — with no audit trail and no second thought.

$ docker inspect my-service | jq '.[0].Config.Env'
[
  "PATH=/usr/local/bin:...",
  "DATABASE_PASSWORD=hunter2-but-real",
  "STRIPE_SECRET_KEY=sk_live_..."
]

None of these are clever. That is the point. The leak surface of an environment variable is wide, passive, and almost entirely outside your control, and you usually discover it the day someone greps the log archive for the wrong reason.

The leak vectors, briefly

The common thread is that none of these were decisions. They are emergent properties of putting a secret somewhere designed to be inherited and introspected. You did not choose to ship the password to your APM vendor; the environment did it for you.

Mount it, don't inject it

The first and cheapest move is to stop treating the secret as a string and start treating it as a file. Mounting a secret as a file on a tmpfs — memory-backed, never written to disk — gives you something with actual access control: ownership, permissions, and a path you can lock down to a single user. A file at 0400 owned by the service user is not readable by a child process unless that child runs as the same user and you hand it the path. It does not get inherited. It does not show up in docker inspect. It is not in /proc/<pid>/environ. Kubernetes secret volumes, Docker secrets, and systemd's LoadCredential= all do exactly this, and the application change is usually one line — read a file instead of an env var.

This is not a security panacea; a file secret read into memory is still a secret in memory, and anyone with the access to read /proc/<pid>/environ can usually read /proc/<pid>/mem too if they try hard enough. But it closes every one of the passive, accidental leak paths above, and those are the ones that actually bite. You are trading a noticeboard for a locked drawer. The drawer can still be forced; it just is not lying open on the desk.

Short-lived credentials and the broker

The deeper problem is that a static secret in any location — env var, file, or otherwise — ages badly. It is valid until someone rotates it, which in practice means valid until the breach, because nobody rotates a database password that is working. The fix is not better storage for the long-lived secret; it is to stop having a long-lived secret.

This is where a broker earns its place. OpenBao and Vault can issue dynamic credentials — a database role created on demand, valid for an hour, revoked automatically when the lease expires. The service authenticates to the broker using its workload identity (a Kubernetes service account token, an instance identity document, a mounted certificate) and receives a fresh credential it never has to store anywhere durable. There is no static password to leak, because there is no static password. If a credential does leak, it is already expiring, and the audit log shows exactly which workload requested it and when.

That last part is the quiet win. Move secrets to a broker and rotation, revocation, and audit stop being things you build and start being things you get. Revoking access becomes a single API call instead of a coordinated redeploy. The audit trail — who fetched which secret, when — is a side effect rather than a project. You were never going to build that for an env var.

Migrating without a rewrite

None of this requires a heroic refactor, which is the usual objection and usually wrong. The cheap first step is the file mount: change os.environ["DB_PASSWORD"] to read a path, mount the secret on tmpfs, and you have closed the passive leak paths in an afternoon. The broker comes next, behind the same interface — a sidecar or agent fetches the dynamic credential and writes it to that same file, refreshing it on a lease, and the application still just reads a file. You can do this one service at a time, starting with whichever one currently has a production database password sitting in a Helm values.yaml in a Git repo. You know the one.

Treat the environment as public, because it effectively is. Anything you would not write on a whiteboard in a shared office does not belong in os.environ — and a process's environment is a good deal more public than that whiteboard, because the whiteboard at least does not get inherited by every child you spawn.