Scenario: The Token That Shipped
A generated Dockerfile built cleanly, passed review, and published a credential to every host that pulled it. The failure with no symptom at all.
The Symptom
There isn't one.
That's what makes this scenario different from the others in this phase. No slow pipeline, no lost data, no ten-second stall. The build succeeded. Tests passed. Review approved. The service ran in production for five weeks without a single anomaly.
The team found out when a security researcher emailed them a working npm token, extracted from a public image on their registry.
How It Got There
An engineer needed to install packages from a private npm registry during the build. They asked an AI assistant for a Dockerfile, got something reasonable, and shipped it:
FROM node:22-slim
WORKDIR /app
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc
COPY package.json package-lock.json ./
RUN npm ci
RUN rm -f .npmrc
COPY . .
CMD ["node", "server.js"]docker build --build-arg NPM_TOKEN=$NPM_TOKEN -t myapp .Read it the way the reviewer did. It takes the token as a build argument rather than hardcoding it. It writes the credential file, uses it, and deletes it on the next line. It doesn't appear anywhere in the final application. Nothing about it looks careless.
It is, nonetheless, completely broken — in two independent ways.
The Investigation
Anyone with the image can run this:
docker history --no-trunc myapp:latestCREATED BY SIZE
RUN |1 NPM_TOKEN=npm_xY3kP9mQ2vL8nR4tW6zA1bC5dE7fG0hJ /bin/sh … 2.1kB
The token is sitting in the layer history, in plaintext, readable by anyone who pulls the image.
And separately:
docker inspect myapp:latest | jq '.[0].Config.Env'Build arguments used in a RUN are recorded in the instruction that consumed them. The rm -f .npmrc removed the file — it did nothing about the record of the command that created it.
The Mechanism
Two distinct failures, and it's worth separating them because the fixes differ.
1. ARG values are recorded in image metadata. A build argument isn't a secret channel. Its value becomes part of the layer's CreatedBy string, which travels with the image forever. Deleting files has no bearing on this.
2. Deleting a file in a later layer doesn't remove it. Even without the ARG problem, if .npmrc had arrived via COPY, the rm on a later line would only add a whiteout marker hiding it from the merged view. The bytes remain in the earlier layer, downloaded on every pull and trivially extractable.
Both failures share one property that makes them dangerous: they produce no error. Docker considers this a correct build. The Dockerfile is valid, the image works, and every automated check passes. The only thing that surfaces it is someone deliberately looking at what the image contains.
The Fix
BuildKit secret mounts make the credential available to one RUN step and nowhere else — not in a layer, not in metadata, not in history:
# syntax=docker/dockerfile:1
FROM node:22-slim
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
npm ci
COPY . .
CMD ["node", "server.js"]docker build --secret id=npmrc,src=$HOME/.npmrc -t myapp .No rm needed, because nothing was ever written. Secrets mount at /run/secrets/<id> unless target= says otherwise. For private Git dependencies there's an SSH equivalent:
RUN --mount=type=ssh git clone git@github.com:me/private-repo.gitResponding to the Leak
The order matters more than the individual steps, because most of them don't reduce risk at all.
1. Rotate first. Revoke the token, issue a new one. This is the only step that actually makes the credential useless.
2. Then clean up. Delete the image tags, purge layers from the registry, rewrite Git history if the token was committed.
3. Then prevent. Fix the Dockerfile, add secret scanning to CI, fix .dockerignore.
The reason for that ordering is uncomfortable but important: steps 2 and 3 do nothing about copies already distributed. Every CI runner that cached the image, every host that pulled it, every developer who ran it locally still has the layer. You cannot un-publish bytes.
Treat a leaked credential as compromised the moment it was published, not when someone proves it was used. Rotation is cheap and reversible. The alternative is learning about it from your audit logs.
Check yourself
A team discovers an API key baked into an image three weeks ago. The image is in a private registry and has been pulled by several CI runners and a few developer machines. They delete the image tag. Is the incident closed?
The Broader Lesson: Generated Config Needs a Different Check
This scenario is in the roadmap specifically because of how the Dockerfile was produced. The assistant's output wasn't unreasonable — ARG plus cleanup is a pattern that appears in a great deal of older documentation and in plenty of real repositories. It was, in a narrow sense, idiomatic.
The gap was in how it was checked. The reviewer read it for correctness — does this build, does it install the packages, is the credential removed? Every answer was yes.
Nobody checked what the image contained.
That distinction generalises well beyond secrets, and it's the habit worth carrying:
| Instead of asking | Ask |
|---|---|
| Did the build succeed? | docker history — what's recorded in the image? |
| Does the config declare a volume? | docker inspect + df — is data landing on it? |
| Does it use exec form? | time docker stop — does it stop in under a second? |
| Does it look secure? | docker scout quickview — what does a scan say? |
Each takes seconds and tests the actual property rather than your reading of the file. As AI assistance makes it faster to produce configuration, the verification step becomes the part that carries the weight — not because generated config is worse, but because more of it exists and none of it arrives with the author's context.
In an Interview
"How do you handle secrets in Docker?" is extremely common, and the mediocre answer is "use environment variables, not hardcoded values."
What's being tested
- Do you distinguish build-time from runtime secrets? They're different problems with different solutions, and conflating them is the most common weakness.
- Do you understand layer immutability? The delete-doesn't-remove point demonstrates real mechanical understanding.
- Do you know what to do after a leak? Rotation-first shows incident experience rather than theory.
A strong answer
Split the question immediately: "There are two problems here — credentials needed during the build, and credentials needed at runtime."
Build-time: never ARG or COPY, because both persist in image metadata and layers. Use BuildKit secret mounts, which expose the credential to a single RUN and never write it to a layer.
Runtime: environment variables work and are widely used, but they're visible in docker inspect, inherited by child processes, and routinely captured by crash reporters. Files mounted from a secret store are better — many images support the _FILE suffix convention for exactly this.
Then close on where secrets should come from: a secret manager gives you access control, audit and rotation, which a .env file gives you none of.
Follow-ups
"What if the secret is created and deleted inside a single RUN?" Technically it isn't in that layer's result — but it's still the wrong approach. It breaks the moment someone splits the instruction for readability, and it doesn't address build cache or logs. Secret mounts remove the class of problem instead of relying on a trick.
"How would you detect this in CI?" Three layers. Secret scanning on the repository; a check on the built image itself, since scanning source alone misses anything injected at build time; and BuildKit's own build checks, which include a SecretsUsedInArgOrEnv rule that flags exactly this Dockerfile — it detects credentials passed via ARG or ENV and points you at secret mounts instead. Knowing that the builder will warn you about this unprompted is a good detail to have.
Check yourself
Which pair correctly matches the secret type to its appropriate mechanism?
Next Scenario
This failure was silent. The next one is the opposite — a single opaque error line that stops a service dead in production while it runs perfectly on every developer machine.