feat(preload): hold a schema's images on every worker ahead of its launches

README.md

@@ -300,6 +300,31 @@ Workers:
       show every worker, its two health states, how long it has been in the
       reachable one and why, and how many instances it holds
 
+Preloads (when a schema's images fit on every worker, pull them ahead of its launches):
+  preload-add <schema>
+      pull every image the schema launches onto every worker, including
+      workers added later, without holding any worker out of placement: a
+      worker serves without pulling once preload-list shows it ready.
+      Cleanup is then corkd's, not docker-reaper's, for as long as the
+      schema is preloaded: after its pulls, each worker deletes the images
+      of its challenges that no current build launches, though not before
+      pulling a version that landed meanwhile. An image a container still
+      uses (an instance the update has yet to stop, a launch it overtook)
+      is tagged <repo>:stale-<tag> instead, for a later pass to delete
+      once nothing uses it. The workers' image store
+      has to hold the whole set plus a second version of what one update
+      redeploys, with docker-reaper's image eviction off. What the preload
+      stops covering -- the schema removed with remove-schema, a challenge
+      dropped from it -- may stay on the workers unless another preloaded
+      schema shares the challenge, and corkd's log lists those images
+  preload-remove <schema>
+      go back to pulling on demand: the workers pull no more of the
+      schema's images, the ones already pulled stay, and the ones that may
+      remain uncleaned are listed: turn docker-reaper's image eviction back
+      on, or remove them by hand
+  preload-list
+      each preloaded schema, and how many of its images each worker holds
+
 Base image pins:
   pin-list
       show the pinned base images and how many challenges use each
@@ -387,6 +412,59 @@ changes.
 Either way, your challenges are then built and ready to run. Send HTTP
 requests for start/stop with the appropriate build IDs and watch them run.
 
+Workers hold challenge images in one of two modes, chosen per schema by
+whether its images fit on a worker:
+
+- **On demand**, the default. A worker pulls an image the first time it
+  launches it, so the set being served never has to fit on one box. This is the
+  mode for a large or growing catalogue; its cost is a pull on each worker's
+  first launch of each image. Cleanup is out of band: corkd removes no images
+  from a worker, and docker-reaper evicts what has gone unused.
+- **Preloaded**. When a schema's images fit on every worker,
+  `cork preload-add <schema>` has every worker pull all of them ahead of time,
+  and `preload-list` shows when each worker holds them. A worker that shows
+  `ready` serves the schema's launches without pulling. Until then launches on
+  it pull as they would on demand: a worker just added takes launches while it
+  catches up, and the new version of a redeployed challenge is pulled by each
+  worker once its build lands, after any pulls the worker already has under
+  way.
+
+  Cleanup is corkd's own, not docker-reaper's or an operator's, for as long as
+  a challenge is preloaded. After its pulls, each worker deletes the images of
+  it that no current build launches, such as the version a redeploy replaced
+  or a seed the schema dropped; a version that lands while the worker pulls is
+  pulled before its predecessor goes. The delete is never forced. A redeploy stops the old version's instances, or restarts them
+  on the new one; an image a container still uses -- an instance the redeploy
+  has yet to reach, a launch it overtook -- is tagged `<repo>:stale-<tag>`
+  instead, and a later pass deletes it once nothing uses it. A redeployed
+  challenge mostly shares its layers with the version it replaces, so these
+  deletes are small; while one runs, the worker's container creates and
+  removes, and its image checks and lists, wait for it.
+
+  A preload ends with `preload-remove` or with its schema's deletion; an update
+  that leaves the schema empty, such as a converge that failed, keeps it for
+  the schema's return. What leaves preloading is not tracked down: deleting a
+  preloaded schema, or dropping a challenge from it, may leave its images on
+  the workers, unless another preloaded schema shares the challenge. corkd logs
+  the images that may remain, from its database, without asking the workers
+  what they hold. Stale tags left behind that way are listed by
+  `docker images --format '{{.Repository}}:{{.Tag}}' | grep ':stale-'`, and the
+  rare race of a delete with a launch can leave an untagged image, which
+  `docker image prune` removes.
+
+  Size the workers' image store for the whole set, plus a second version of
+  whatever one update redeploys (the new version is pulled before the old one
+  goes), and turn docker-reaper's image eviction off there, since corkd decides
+  what stays.
+
+The modes are per schema but eviction is per worker: a worker with eviction off
+also keeps every image an on-demand schema has it pull, and an on-demand schema
+sharing a challenge with a preloaded one has its old versions of that challenge
+removed while the preload lasts. `preload-remove` leaves the schema's images in
+place and lists the ones that may remain uncleaned; turn docker-reaper's image
+eviction back on for those workers, or remove them by hand, since nothing else
+will.
+
 To build or run challenges by hand, use `cork`'s `build`, `start` and
 `stop` — they act on one orchestrator directly. Note that a manual build
 belongs to no schema, so it has no destination and no build plane will route
@@ -410,6 +488,7 @@ Every setting also answers to its pre-rename `CMGR_` name (`CMGR_DB` for `CORK_D
 | CORK_DB_WAL              | Enable WAL journaling for SQLite                                                                                                   | on                                                                       |
 | CORK_LOGGING             | Log verbosity for `corkd` and `cork-build`: `debug`, `info`, `warn`, `error` or `disabled`. There is no flag for it on `corkd`, so this is the only way to raise an orchestrator's logging; `cork-build --verbose` is the flag form and wins over it. A value that does not parse is warned about and not fatal | info |
 | CORK_CONCURRENT_LAUNCHES | Launch slots per daemon (1-16), and as many teardown slots: network creation plus container starts, which dockerd serializes internally on either firewall backend; no gain measured past 2 | 2                                                                        |
+| CORK_PRELOAD_PULLS       | How many preload pulls run at once across the fleet (each worker pulls one at a time); see `cork preload-add` | 8 |
 | CORK_WORKER_POLL_INTERVAL | How often each worker is probed (telemetry and docker on the same tick)                                                           | 5s                                                                       |
 | CORK_WORKER_POLL_TIMEOUT | Telemetry poll timeout; must be under the probe interval (clamped to half of it otherwise)                                         | 250ms                                                                    |
 | CORK_WORKER_PING_TIMEOUT | Docker `/_ping` timeout; clamped the same way. Looser than the telemetry one — dockerd under load answers more slowly than a cached verdict, and there is TLS in the path | 1s                          |
@@ -441,7 +520,7 @@ Every setting also answers to its pre-rename `CMGR_` name (`CMGR_DB` for `CORK_D
 Everything above is read by the shared `cork` library, so `corkd` and
 `cork-build` accept the same surface — but each ignores the other's half, and
 says so at startup rather than silently. A build plane runs nothing, so
-`CORK_CONCURRENT_LAUNCHES`, `CORK_PORTS`, `CORK_INTERFACE`,
+`CORK_CONCURRENT_LAUNCHES`, `CORK_PRELOAD_PULLS`, `CORK_PORTS`, `CORK_INTERFACE`,
 `CORK_ENABLE_DISK_QUOTAS`, `CORK_PRUNE_AGE` and the `CORK_WORKER_*`
 tunables are inert on `cork-build`; an orchestrator on an external build
 plane builds nothing, so `CORK_DIR`, `CORK_BASE_PINS`, `DOCKER_HOST` and

cmd/cork/main.go

@@ -88,6 +88,12 @@ func main() {
 		exitCode = workerDownCommand(c, cmdArgs)
 	case "worker-list":
 		exitCode = workerListCommand(c, cmdArgs)
+	case "preload-add":
+		exitCode = preloadAddCommand(c, cmdArgs)
+	case "preload-remove":
+		exitCode = preloadRemoveCommand(c, cmdArgs)
+	case "preload-list":
+		exitCode = preloadListCommand(c, cmdArgs)
 	case "pin-list":
 		exitCode = pinListCommand(c, cmdArgs)
 	case "pin-refresh":
@@ -182,6 +188,31 @@ Workers:
       daemon; load, from its telemetry agent), how long it has been in that
       state, why, and how many instances it holds
 
+Preloads (when a schema's images fit on every worker, pull them ahead of its launches):
+  preload-add <schema>
+      pull every image the schema launches onto every worker, including
+      workers added later, without holding any worker out of placement: a
+      worker serves without pulling once preload-list shows it ready.
+      Cleanup is then corkd's, not docker-reaper's, for as long as the
+      schema is preloaded: after its pulls, each worker deletes the images
+      of its challenges that no current build launches, though not before
+      pulling a version that landed meanwhile. An image a container still
+      uses (an instance the update has yet to stop, a launch it overtook)
+      is tagged <repo>:stale-<tag> instead, for a later pass to delete
+      once nothing uses it. The workers' image store
+      has to hold the whole set plus a second version of what one update
+      redeploys, with docker-reaper's image eviction off. What the preload
+      stops covering -- the schema removed with remove-schema, a challenge
+      dropped from it -- may stay on the workers unless another preloaded
+      schema shares the challenge, and corkd's log lists those images
+  preload-remove <schema>
+      go back to pulling on demand: the workers pull no more of the
+      schema's images, the ones already pulled stay, and the ones that may
+      remain uncleaned are listed: turn docker-reaper's image eviction back
+      on, or remove them by hand
+  preload-list
+      each preloaded schema, and how many of its images each worker holds
+
 Base image pins:
   pin-list
       show the pinned base images and how many challenges use each

cmd/cork/preloads.go

@@ -0,0 +1,116 @@
+package main
+
+import (
+	"fmt"
+	"io"
+	"net/url"
+	"os"
+	"time"
+)
+
+// PreloadInfo mirrors cork.PreloadInfo; cork/preload.go is the source of truth.
+type PreloadInfo struct {
+	Schema  string          `json:"schema"`
+	Images  int             `json:"images"`
+	Workers []PreloadWorker `json:"workers"`
+}
+
+type PreloadWorker struct {
+	IP        string           `json:"ip"`
+	Reachable string           `json:"reachable"`
+	Held      int              `json:"held"`
+	Pulling   string           `json:"pulling"`
+	Failed    []PreloadFailure `json:"failed"`
+	LastPass  *time.Time       `json:"last_pass"`
+}
+
+type PreloadFailure struct {
+	Image string `json:"image"`
+	Error string `json:"error"`
+}
+
+func preloadAddCommand(c *client, args []string) int {
+	if len(args) != 1 {
+		fmt.Fprintln(os.Stderr, "usage: preload-add <schema>")
+		return USAGE_ERROR
+	}
+	var resp struct {
+		WorkersConfigured *bool `json:"workers_configured"`
+	}
+	if err := c.doJSON("POST", "/preloads", map[string]string{"schema": args[0]}, &resp); err != nil {
+		return runtimeError(err)
+	}
+	if resp.WorkersConfigured != nil && !*resp.WorkersConfigured {
+		fmt.Println("Preloaded, but no worker is registered: workers pull the set as they are added.")
+		return NO_ERROR
+	}
+	fmt.Println("Preloading on every worker; preload-list shows how far each has got.")
+	return NO_ERROR
+}
+
+func preloadRemoveCommand(c *client, args []string) int {
+	if len(args) != 1 {
+		fmt.Fprintln(os.Stderr, "usage: preload-remove <schema>")
+		return USAGE_ERROR
+	}
+	var resp struct {
+		Released []string `json:"released"`
+	}
+	if err := c.doJSON("DELETE", "/preloads/"+url.PathEscape(args[0]), nil, &resp); err != nil {
+		return runtimeError(err)
+	}
+	printReleased(os.Stdout, resp.Released)
+	return NO_ERROR
+}
+
+// printReleased lists what corkd no longer cleans up, as it knows from its
+// database: no worker was asked what it holds.
+func printReleased(out io.Writer, released []string) {
+	if len(released) == 0 {
+		fmt.Fprintln(out, "Pulling on demand again; other preloaded schemas still want every image it preloaded.")
+		return
+	}
+	fmt.Fprintf(out, "Pulling on demand again. corkd no longer cleans up these %d image(s), which may remain on the workers, as may older ones of the same repositories, until eviction or an operator removes them:\n", len(released))
+	for _, name := range released {
+		fmt.Fprintf(out, "  %s\n", name)
+	}
+}
+
+func preloadListCommand(c *client, args []string) int {
+	var infos []PreloadInfo
+	if err := c.doJSON("GET", "/preloads", nil, &infos); err != nil {
+		return runtimeError(err)
+	}
+	printPreloads(os.Stdout, infos, time.Now())
+	return NO_ERROR
+}
+
+func printPreloads(out io.Writer, infos []PreloadInfo, now time.Time) {
+	if len(infos) == 0 {
+		fmt.Fprintln(out, "no schema is preloaded")
+		return
+	}
+	for _, info := range infos {
+		fmt.Fprintf(out, "%s: %d image(s)\n", info.Schema, info.Images)
+		for _, w := range info.Workers {
+			progress := fmt.Sprintf("%d/%d", w.Held, info.Images)
+			state := ""
+			switch {
+			case w.Pulling != "":
+				state = "pulling " + w.Pulling
+			case info.Images == 0: // a schema left with no builds
+				state = "no images"
+			case w.Held == info.Images && w.LastPass != nil:
+				state = "ready"
+			}
+			checked := "not checked yet"
+			if w.LastPass != nil {
+				checked = fmt.Sprintf("checked %s ago", now.Sub(*w.LastPass).Round(time.Second))
+			}
+			fmt.Fprintf(out, "  %-15s  %-12s  %9s  %-16s  %s\n", w.IP, w.Reachable, progress, checked, state)
+			for _, f := range w.Failed {
+				fmt.Fprintf(out, "  %-15s  failed %s: %s\n", "", f.Image, f.Error)
+			}
+		}
+	}
+}

cmd/cork/preloads_test.go

@@ -0,0 +1,58 @@
+package main
+
+import (
+	"regexp"
+	"strings"
+	"testing"
+	"time"
+)
+
+// preload-list is what an operator watches before launches start: "ready" has
+// to mean every image held, on a worker that has actually been checked.
+func TestPrintPreloads(t *testing.T) {
+	now := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
+	checked := now.Add(-90 * time.Second)
+	infos := []PreloadInfo{{
+		Schema: "event",
+		Images: 3,
+		Workers: []PreloadWorker{
+			{IP: "10.0.0.1", Reachable: "ok", Held: 3, LastPass: &checked},
+			{IP: "10.0.0.2", Reachable: "ok", Held: 1, Pulling: "r/web:s1", LastPass: &checked,
+				Failed: []PreloadFailure{{Image: "r/pwn:s1", Error: "manifest unknown"}}},
+			{IP: "10.0.0.3", Reachable: "unresponsive"},
+			{IP: "10.0.0.4", Reachable: "ok", Held: 3}, // a count from no pass of its own
+		},
+	}}
+
+	var out strings.Builder
+	printPreloads(&out, infos, now)
+	lines := strings.Split(strings.TrimRight(out.String(), "\n"), "\n")
+	if len(lines) != 6 {
+		t.Fatalf("got %d lines, want 6:\n%s", len(lines), out.String())
+	}
+	for i, want := range []string{
+		`^event: 3 image\(s\)$`,
+		"3/3 .* checked 1m30s ago .* ready$",
+		"1/3 .* pulling r/web:s1$",
+		"failed r/pwn:s1: manifest unknown$",
+		"unresponsive .* 0/3 .* not checked yet",
+		`10\.0\.0\.4 .* 3/3 .* not checked yet\s*$`,
+	} {
+		if !regexp.MustCompile(want).MatchString(lines[i]) {
+			t.Errorf("line %d = %q, want it to match %q", i, lines[i], want)
+		}
+	}
+
+	// A schema left with no builds keeps its preload, and is not "ready".
+	out.Reset()
+	printPreloads(&out, []PreloadInfo{{Schema: "emptied", Workers: []PreloadWorker{{IP: "10.0.0.1", Reachable: "ok", LastPass: &checked}}}}, now)
+	if got := out.String(); strings.Contains(got, "ready") || !strings.Contains(got, "no images") {
+		t.Errorf("an emptied schema shows:\n%s", got)
+	}
+
+	out.Reset()
+	printPreloads(&out, nil, now)
+	if got := out.String(); got != "no schema is preloaded\n" {
+		t.Errorf("with nothing preloaded: %q", got)
+	}
+}

cmd/corkd/main.go

@@ -120,6 +120,8 @@ func main() {
 	http.HandleFunc("/instances/", s.instanceHandler)
 	http.HandleFunc("/workers", s.workersHandler)
 	http.HandleFunc("/workers/", s.workerHandler)
+	http.HandleFunc("/preloads", s.preloadsHandler)
+	http.HandleFunc("/preloads/", s.preloadHandler)
 	http.HandleFunc("/pins", s.pinsHandler)
 	http.HandleFunc("/schemas", s.schemaHandler)
 	http.HandleFunc("/schemas/", s.existingSchemaHandler)
@@ -437,6 +439,43 @@ Workers:
       variables.  See https://docs.docker.com/engine/reference/commandline/cli/
       for specific details.
 
+Preloads:
+  Workers hold challenge images in one of two modes, chosen per schema by
+  whether its images fit on a worker. On demand, the default, a worker pulls
+  an image the first time it launches it, so the set being served never has
+  to fit on one box, and cleanup is out of band: corkd removes no images, and
+  docker-reaper evicts what goes unused. A preloaded schema (POST /preloads
+  {"schema": "<name>"}, or cork preload-add) instead has every image it
+  launches pulled onto every worker ahead of its launches, including workers
+  added later. Each worker's preload pulls one image at a time, in its own
+  order, and at most CORK_PRELOAD_PULLS (default 8) run at once across the
+  fleet, without holding back placement; launches still pull their own. A
+  worker checks again when a schema is preloaded, as each build lands, and
+  every five minutes; once GET /preloads shows it holding every image, its
+  launches of the schema do not wait on the registry. Until then they pull
+  as on demand: a worker just added catches up while it serves, and a
+  redeployed challenge's new version is pulled by each worker once its build
+  lands, after any pulls already under way there. Its workers need an image
+  store that holds the whole set plus a second version of what one update
+  redeploys, with docker-reaper's image eviction off.
+
+  A preloaded schema's cleanup is corkd's own. While a challenge is preloaded,
+  each worker deletes, after its pulls, the images of it that no current
+  build launches, pulling first any version that landed meanwhile. The
+  delete is never forced: an image a container still uses, such as an
+  instance the redeploy has yet to stop or a launch it overtook, is tagged
+  <repo>:stale-<tag> instead and deleted by a later pass once nothing uses
+  it. A worker's removals stop at the first that times
+  out, and a preload call that times out counts against nothing. A preload
+  ends with DELETE /preloads/<name> or with its schema's deletion. What an
+  operation takes out of preloading -- the schema deleted, a challenge
+  dropped from it -- may stay on the workers unless another preloaded schema
+  shares the challenge, and corkd logs the images that may remain, read from
+  its database rather than from the workers. DELETE /preloads/<name> returns
+  the schema to pulling on demand, so the workers pull no more of it, leaves
+  its images in place, and answers with the ones that may remain uncleaned:
+  {"released": [<image>, ...]}.
+
 `, os.Args[0])
 }