feat(catalog): pin every app image to a content digest - #398
Merged
Conversation
A tag is a mutable pointer. `voidic/route96:latest` is whatever that name resolved to the last time a node pulled it, so a publisher, or anyone who takes their registry account, could change the code running in every deployment of that app by re-pushing, with nobody here approving anything. Six of the seven catalog entries were on `:latest` or a branch tag, and only haven pinned a version at all. Kubernetes made that likelier rather than less likely: `imagePullPolicy` is inferred from the reference and defaults to `Always` for a `:latest` tag, so re-pushed bytes arrived on the next pod creation. Pin all 12 images as `repo:tag@sha256:...`, keeping the tag in front so a diff stays readable while the digest is what the runtime honours. Resolve with `buildx imagetools` so the pin is the multi-arch index digest rather than one platform's manifest. Add `Compose::validate_pinned_images`, refusing a service or init image without a well-formed digest, wired into admin app create/update and the `compose-validate` CLI. It is an admission rule rather than part of `validate`, following `validate_declarations`: the operator re-parses the stored compose on every reconcile, so enforcing it there would take apps admitted before the rule offline over something they cannot fix. Set `imagePullPolicy: IfNotPresent` explicitly on every app container and init container. With a digest a cached layer set can only be the right one, and re-pulling only adds a way for a restart to fail while a registry is down or rate-limiting. Add `scripts/catalog-digest-drift.sh` and a weekly workflow job to resolve each pinned tag and report the ones that have moved, so an upstream release arrives as something to review, test and re-pin. It does not run on pull_request: drift is a property of the registry that day, not of the change under review. This buys immutability, not trust. An image that is hostile when it is pinned stays pinned hostile, a pinned base receives no CVE fixes until somebody bumps it, and a pin does nothing for an image that fetches its own code at boot.
v0l
force-pushed
the
feat/pin-catalog-image-digests
branch
from
August 28, 2026 15:13
d357de3 to
f7f8131
Compare
v0l
enabled auto-merge
August 28, 2026 15:16
The new digest-pin admission rule rejects every compose the admin app tests post, which is why the e2e job fails while the workspace tests (run with --exclude lnvps_e2e) pass. Pins each fixture image and adds a case asserting an unpinned image is refused with 400.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every catalog image was on a mutable tag. Six of seven entries used
:latestor a branch tag; onlyhavenpinned a version, and even a version tag can be re-pushed. That means a publisher, or anyone who takes their registry account, could change the code running in every deployment of that app without anyone here approving anything.Kubernetes made it likelier rather than less likely.
imagePullPolicyis inferred from the reference and defaults toAlwaysfor a:latesttag, which the operator never overrode, so re-pushed bytes reached running deployments on the next pod creation.This came out of reviewing #397, where the same class of problem is far worse (that image clones
mainand installs dependencies at boot). Pinning is the general fix for the rest of the catalog; it is necessary but not sufficient there, which is why that PR is asked to bake its app into the image instead.What
repo:tag@sha256:..., tag kept in front so a diff stays readable while the digest is what the runtime honours. Resolved withdocker buildx imagetools inspect --format '{{.Manifest.Digest}}', which returns the multi-arch index digest.docker manifest inspect -vreturns the per-platform manifest digest and would have silently pinned every entry to amd64.Compose::validate_pinned_images()refuses a service orinit:image without a well-formed@sha256:<64 lower-case hex>, wired into admin app create/update and thecompose-validateCLI (so the existing CI step overcatalog/*.yamlenforces it). It is an admission rule rather than part ofvalidate(), following thevalidate_declarationsprecedent: the operator re-parses the stored compose on every reconcile, so enforcing it there would take apps admitted before the rule offline over something they cannot fix.imagePullPolicy: IfNotPresentset explicitly on every app container and init container, instead of inheriting a policy from whether the tag happens to read:latest.scripts/catalog-digest-drift.shplus a weekly workflow job resolves each pinned tag and reports the ones that have moved, so an upstream release arrives as something to review, test and re-pin. Not run onpull_request: drift is a property of the registry that day, not of the change under review, and a red check nobody can fix in their branch is a check people learn to ignore.What this does not do
It buys immutability, not trust. An image that is hostile when it is pinned stays pinned hostile; a pinned base receives no CVE fixes until somebody bumps it; and a pin does nothing for an image that fetches its own code at boot. The containment for a hostile image remains the per-deployment namespace, the missing service account token and the egress NetworkPolicy.
Decisions worth reviewing
postgres:17-alpine,redis:7-alpineandmariadb:11were deliberately floating minor tags. Pinning freezes their patch and CVE updates, and the weekly drift report is what turns that into a decision rather than neglect. It does add recurring work.PATCHto any existing catalog app will be refused until its compose is pinned.Validation
cargo test --workspace --exclude lnvps_e2e -- --test-threads=1green. New tests: pinned accept, unpinned service, unpinned init, four malformed-digest shapes, the admission-only guarantee, admin field validation, and an operator assertion that both container kinds set the pull policy.cargo fmt --checkclean; clippy warning count unchanged (332 before and after).compose-validategreen on all seven documents.scripts/app-catalog-test.shstartsstrfryandroute96from their digest references, HTTP 200 on ingress, so the pins resolve and run rather than merely parsing.scripts/catalog-digest-drift.shreports all 12 pins current.