Run app deployment backups as Kubernetes Jobs - #409
Open
v0l wants to merge 3 commits into
Open
Conversation
A run covers one deployment at a point in time; every service that declares a `backup:` method contributes one artifact, and each artifact is one Job and one uploaded object, because two services back up with different images and different volumes and cannot share a pod. A `volume:` backup tars the claim mounted read-only, and is pinned by pod affinity to the node already holding it: a ReadWriteOnce claim can only be mounted where it is already attached, and without the affinity the Job schedules anywhere and sits in Multi-Attach error until its deadline kills it. A `command:` backup runs the dump in an init container on the app's own image -- only that image has `pg_dumpall` or `mariadb-dump` -- with the service's resolved env, so a generated password is available to it exactly as the running app has it. The uploader is a separate container because the app's image is not guaranteed to contain an HTTP client. Scheduling is here rather than in a CronJob because a CronJob fires with nothing to write the run down in and no way to hold an upload URL that has not expired. The operator mints the row, signs a PUT for that one key, and hands the Job a URL through a Secret-backed env var -- not a command line, where every other process in the pod could read it -- then deletes the Secret with the Job. Due-ness is measured from the last run, so a deployment that was down through several occurrences gets one catch-up rather than one per missed slot. Every app on the same daily schedule comes due in the same minute, so Jobs in flight are capped per cluster. The pending rows are the queue, so a run that waits a sweep loses nothing. A Job that reports success with nothing in the bucket is recorded as failed: the alternative is offering the customer a restore point that does not exist. Retention deletes the object before tombstoning the row, and keeps the row when that delete fails, or the object is orphaned with nothing pointing at it. Backups are opt-in per operator: no `backups:` config, nothing scheduled and nothing run, which is how every operator behaved until now. The bucket credentials can come from the environment, because the config file lives in a ConfigMap and a bucket key does not belong in one.
The dump does not run inside the service's container: it runs in the backup Job's pod, on the same image. `pg_dumpall` and `mariadb-dump` left to their defaults connect to a unix socket that only exists inside the service's own container, so both catalog entries would have failed on their first scheduled run with a missing-socket error. Both now address the service over the network (`-h db`), which is also why the rule is worth writing down next to the grammar rather than only in the field's doc comment: it is not what a compose file looks like it does. route96 and buzz also get the schedule the feature exists for -- daily at 03:00 UTC, keeping a week of restore points.
The unit tests pin our SigV4 output against AWS's published vector. That proves the arithmetic and nothing about whether a server accepts what we send: the canonical request has to match byte for byte, and a difference in path encoding, query ordering or the signed-headers list surfaces as an opaque 403 at the bucket, not as a wrong hex string in CI. So the presigned round trip now runs against the `rustfs` service already in the e2e compose -- the same S3 implementation the app catalog ships to customers. Upload, size, download with the signed filename, delete, and delete again, because retention re-runs against objects that are already gone. Two properties are worth asserting against a server rather than against our own belief, because they are what make it safe to hand a URL to a pod running in the customer's namespace: an upload URL cannot be replayed as a read, cannot be pointed at another deployment's key, and stops working when it expires. The operator test goes one further and runs the uploader's *actual command line* in the image the Job names, against the same server, then downloads the artifact and checks it really is the archive it claims to be. The builder tests assert what the script says; only this catches a busybox `tar` flag or a `curl` invocation a real S3 server refuses -- either of which is a backup that silently never existed until somebody tried to restore it. `ObjectStore::create_bucket` exists for this, and for provisioning a fresh store. It is deliberately not on the operator's startup path: creating buckets is not a permission the operator's key should need in order to write objects into a bucket somebody else made. Both suites skip themselves when the environment is unset, so a plain `cargo test --workspace` still needs nothing. `run-e2e.sh` checks rustfs is answering before running them, since a suite that skips itself would otherwise pass the run without a single upload happening.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Increment 6c of app-deployment backups (
work/app-deployments.md), after #403 (schema) and #405(signing). This is the part that actually captures data. The customer API is 6d, and restore is a
later increment.
Backups are opt-in per operator: with no
backups:config block, nothing is scheduled and nothingruns, which is how every operator has behaved until now.
How a run executes
A run covers one deployment at one point in time. Every service declaring a
backup:methodcontributes one artifact, and each artifact is one Job and one object, because two services back
up with different images and different volumes and cannot share a pod.
volume:tars the claim mounted read-only, pinned by pod affinity to the node alreadyholding it. A ReadWriteOnce claim can only be mounted where it is already attached; without the
affinity the Job schedules anywhere and sits in
Multi-Attach erroruntil the deadline kills it.command:runs the dump in an init container on the app's own image, since only that imagehas
pg_dumpallormariadb-dump, with the service's resolved env so a generated password isavailable to it exactly as the running app has it. The uploader is a separate container because
the app's image is not guaranteed to contain an HTTP client.
The pod is hardened like the app's own: no service-account token, no service links, read-only
root, dropped capabilities, and it runs as the uid that owns the data (a read-only mount gets no
fsGroupremap, so any other uid would tar an unreadable tree).Why the operator schedules, not a CronJob
A CronJob fires with nothing to write the run down in and no way to hold an upload URL that has not
expired. The operator mints the row, signs a
PUTfor that one key, and hands the Job the URLthrough a Secret-backed env var rather than a command line where every other process in the pod
could read it, then deletes the Secret with the Job. Due-ness is measured from the last run, so a
deployment down through several occurrences gets one catch-up run rather than one per missed slot.
Jobs in flight are capped per cluster (default 3). Every app on the same daily schedule comes due
in the same minute, and each Job reads a whole volume on a node that is also serving apps. The
pending rows are the queue, so a run that waits a sweep loses nothing.
Failure handling
is offering the customer a restore point that does not exist.
or the object is orphaned in the bucket with nothing pointing at it.
real restore point in favour of one that does not exist yet.
Catalog fix worth flagging
The dump does not run inside the service's container, so
pg_dumpall/mariadb-dumpleft totheir defaults connect to a unix socket that exists only there. Both catalog entries would have
failed on their first scheduled run. They now address the service over the network (
-h db), therule is documented next to the grammar, and both apps carry the schedule the feature exists for:
daily at 03:00 UTC, keeping a week.
Operations
backups:block in the operator config (endpoint, bucket, credentials, uploader image, URLexpiry, Job deadline, concurrency). Credentials can come from
LNVPS_BACKUP_ACCESS_KEY/LNVPS_BACKUP_SECRET_KEY, because the config file lives in a ConfigMap.batch/jobs(get/create/patch/delete) and Secretdelete, both in the per-namespacerole, not cluster-wide.
GRANT INSERT, UPDATE ON lnvps.app_deployment_backup.Testing
10 new unit tests over the pure builders: both capture shapes, read-only claim + node affinity,
the URL never reaching a command line, pod hardening and deadlines, staging size, artifact and key
naming, shell quoting of a catalog argv, the retention window, and the concurrency cap. The
cluster-touching functions are untested here, consistent with the rest of this crate.
cargo test --workspace --exclude lnvps_e2e -- --test-threads=1, clippy and fmt are clean, andevery
catalog/*.yamlstill validates.