Skip to content

feat(catalog): pin every app image to a content digest - #398

Merged
v0l merged 2 commits into
masterfrom
feat/pin-catalog-image-digests
Aug 28, 2026
Merged

feat(catalog): pin every app image to a content digest#398
v0l merged 2 commits into
masterfrom
feat/pin-catalog-image-digests

Conversation

@v0l

@v0l v0l commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Why

Every catalog image was on a mutable tag. Six of seven entries used :latest or a branch tag; only haven pinned a version, and even a version tag can be re-pushed. That means a publisher, or anyone who takes their registry account, could change the code running in every deployment of that app without anyone here approving anything.

Kubernetes made it likelier rather than less likely. imagePullPolicy is inferred from the reference and defaults to Always for a :latest tag, which the operator never overrode, so re-pushed bytes reached running deployments on the next pod creation.

This came out of reviewing #397, where the same class of problem is far worse (that image clones main and installs dependencies at boot). Pinning is the general fix for the rest of the catalog; it is necessary but not sufficient there, which is why that PR is asked to bake its app into the image instead.

What

  • All 12 images pinned as repo:tag@sha256:..., tag kept in front so a diff stays readable while the digest is what the runtime honours. Resolved with docker buildx imagetools inspect --format '{{.Manifest.Digest}}', which returns the multi-arch index digest. docker manifest inspect -v returns the per-platform manifest digest and would have silently pinned every entry to amd64.
  • Compose::validate_pinned_images() refuses a service or init: image without a well-formed @sha256:<64 lower-case hex>, wired into admin app create/update and the compose-validate CLI (so the existing CI step over catalog/*.yaml enforces it). It is an admission rule rather than part of validate(), following the validate_declarations precedent: the operator re-parses the stored compose on every reconcile, so enforcing it there would take apps admitted before the rule offline over something they cannot fix.
  • imagePullPolicy: IfNotPresent set explicitly on every app container and init container, instead of inheriting a policy from whether the tag happens to read :latest.
  • scripts/catalog-digest-drift.sh plus a weekly workflow job resolves each pinned tag and reports the ones that have moved, so an upstream release arrives as something to review, test and re-pin. Not run on pull_request: drift is a property of the registry that day, not of the change under review, and a red check nobody can fix in their branch is a check people learn to ignore.

What this does not do

It buys immutability, not trust. An image that is hostile when it is pinned stays pinned hostile; a pinned base receives no CVE fixes until somebody bumps it; and a pin does nothing for an image that fetches its own code at boot. The containment for a hostile image remains the per-deployment namespace, the missing service account token and the egress NetworkPolicy.

Decisions worth reviewing

  • postgres:17-alpine, redis:7-alpine and mariadb:11 were deliberately floating minor tags. Pinning freezes their patch and CVE updates, and the weekly drift report is what turns that into a decision rather than neglect. It does add recurring work.
  • Stored apps are untouched, so the first PATCH to any existing catalog app will be refused until its compose is pinned.

Validation

  • cargo test --workspace --exclude lnvps_e2e -- --test-threads=1 green. New tests: pinned accept, unpinned service, unpinned init, four malformed-digest shapes, the admission-only guarantee, admin field validation, and an operator assertion that both container kinds set the pull policy.
  • cargo fmt --check clean; clippy warning count unchanged (332 before and after).
  • compose-validate green on all seven documents.
  • scripts/app-catalog-test.sh starts strfry and route96 from their digest references, HTTP 200 on ingress, so the pins resolve and run rather than merely parsing.
  • scripts/catalog-digest-drift.sh reports all 12 pins current.

@v0l v0l added api User-facing or admin API changes app-catalog labels Aug 28, 2026
A tag is a mutable pointer. `voidic/route96:latest` is whatever that name
resolved to the last time a node pulled it, so a publisher, or anyone who
takes their registry account, could change the code running in every
deployment of that app by re-pushing, with nobody here approving anything.
Six of the seven catalog entries were on `:latest` or a branch tag, and only
haven pinned a version at all.

Kubernetes made that likelier rather than less likely: `imagePullPolicy` is
inferred from the reference and defaults to `Always` for a `:latest` tag, so
re-pushed bytes arrived on the next pod creation.

Pin all 12 images as `repo:tag@sha256:...`, keeping the tag in front so a
diff stays readable while the digest is what the runtime honours. Resolve
with `buildx imagetools` so the pin is the multi-arch index digest rather
than one platform's manifest.

Add `Compose::validate_pinned_images`, refusing a service or init image
without a well-formed digest, wired into admin app create/update and the
`compose-validate` CLI. It is an admission rule rather than part of
`validate`, following `validate_declarations`: the operator re-parses the
stored compose on every reconcile, so enforcing it there would take apps
admitted before the rule offline over something they cannot fix.

Set `imagePullPolicy: IfNotPresent` explicitly on every app container and
init container. With a digest a cached layer set can only be the right one,
and re-pulling only adds a way for a restart to fail while a registry is
down or rate-limiting.

Add `scripts/catalog-digest-drift.sh` and a weekly workflow job to resolve
each pinned tag and report the ones that have moved, so an upstream release
arrives as something to review, test and re-pin. It does not run on
pull_request: drift is a property of the registry that day, not of the
change under review.

This buys immutability, not trust. An image that is hostile when it is
pinned stays pinned hostile, a pinned base receives no CVE fixes until
somebody bumps it, and a pin does nothing for an image that fetches its own
code at boot.
@v0l
v0l force-pushed the feat/pin-catalog-image-digests branch from d357de3 to f7f8131 Compare August 28, 2026 15:13
@v0l
v0l enabled auto-merge August 28, 2026 15:16
The new digest-pin admission rule rejects every compose the admin app
tests post, which is why the e2e job fails while the workspace tests
(run with --exclude lnvps_e2e) pass. Pins each fixture image and adds a
case asserting an unpinned image is refused with 400.
@v0l
v0l merged commit b547ebd into master Aug 28, 2026
19 checks passed
@v0l
v0l deleted the feat/pin-catalog-image-digests branch August 28, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api User-facing or admin API changes app-catalog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant