Remap all sidecar images across Docker and Kubernetes

README.md

@@ -49,7 +49,10 @@ backend:
 
 This setting does not control how you get the long-running `oz-agent-worker` image (see [Image releases and pinning](#image-releases-and-pinning)). It controls only the task and sidecar images the worker uses to run tasks.
 
-`backend.docker.sidecar_image` overrides the warp-agent sidecar image reference sent by the server (e.g. `docker.io/warpdotdev/warp-agent:latest`); set this when the worker host cannot pull directly from Docker Hub and must use an internal registry mirror or pull-through cache instead. This only affects the warp-agent sidecar (mounted at `/agent`), not any additional sidecars. When using this override, you are responsible for keeping your mirror in sync with `docker.io/warpdotdev/warp-agent` — the server normally sends the correct version-matched image per task, so a stale mirror may cause version incompatibility.
+`backend.docker.sidecar_image` is a legacy shorthand for mapping the official
+`docker.io/warpdotdev/warp-agent` repository to a custom image. Prefer
+`sidecar_image_mappings` for new configurations. Do not combine the two options.
+The [remapping version rules](#remapping-sidecar-images) apply to both forms.
 
 ```yaml
 worker_id: "my-worker"
@@ -58,6 +61,61 @@ backend:
     sidecar_image: "my-registry.io/warpdotdev/warp-agent:latest"
 ```
 
+### Remapping sidecar images
+
+Docker and Kubernetes both support `sidecar_image_mappings` under their backend
+configuration. Keys are source **repositories without a tag or digest**; values
+are destination image references. The mapping applies to every assigned sidecar:
+the main agent, coding-agent CLI/ACP images, computer use, and any other additional
+sidecar. It does not remap the task's base image, Kubernetes preflight image, or
+service containers configured in `pod_template`.
+
+```yaml
+worker_id: "my-worker"
+backend:
+  docker: # Use kubernetes here for the same remapping behavior.
+    sidecar_image_mappings:
+      docker.io/warpdotdev/warp-agent: "registry.internal/warp-agent"
+      docker.io/warpdotdev/warp-claude-cli-sidecar: "registry.internal/claude:approved"
+      docker.io/warpdotdev/warp-xvfb-sidecar: "registry.internal/computer-use@sha256:<digest>"
+```
+
+- When the destination has no tag or digest, the assigned tag/digest is preserved:
+  `docker.io/warpdotdev/warp-agent:v123` becomes
+  `registry.internal/warp-agent:v123`.
+- When the destination includes a tag and/or digest, that destination version
+  replaces the assigned version entirely. For example, any version of the Claude
+  image above becomes `registry.internal/claude:approved`. Replace `<digest>` with
+  the full digest of your mirrored computer-use image before using this config.
+- Matching is by normalized repository, not prefix or pattern:
+  `warpdotdev/warp-agent` and `docker.io/warpdotdev/warp-agent` identify the same
+  source. Duplicate normalized sources, versioned source keys and malformed
+  references are rejected at startup.
+- Mappings act once on the assigned repositories. A mapped destination is not
+  remapped again. Unmatched images and mount options are unchanged; no sidecars
+  are injected.
+- Legacy `sidecar_image` and Kubernetes `coding_cli_sidecars` settings are
+  converted into mappings at startup. `sidecar_image` maps
+  `docker.io/warpdotdev/warp-agent`; each `coding_cli_sidecars` harness maps both
+  `docker.io/warpdotdev/warp-{harness}-cli-sidecar` and
+  `docker.io/warpdotdev/warp-{harness}-acp-sidecar` to its configured image.
+  Non-empty `sidecar_image_mappings` cannot be combined with either legacy option.
+  These aliases use the same version rules as explicit mappings.
+- **Compatibility change:** legacy options now match only official Warp
+  repositories. They no longer override arbitrary source images by mount path or
+  inject coding-CLI sidecars missing from the assignment. Use explicit mappings
+  for non-Warp repositories; the server must assign a sidecar for it to be used.
+
+For Helm installations, configure the equivalent
+`kubernetesBackend.sidecarImageMappings` map in your values file.
+
+Docker sidecar pulls use credentials from the worker's Docker configuration
+(including credential helpers), just like task-image pulls. Make that
+configuration available to the worker process when using private mirrors.
+Kubernetes uses `imagePullSecrets` from `backend.kubernetes.pod_template`.
+Operators remain responsible for mirroring the selected versions and ensuring
+that pinned custom images are compatible with the server and agent.
+
 ### Image releases and pinning
 
 Production self-hosted workers should pin an immutable image version instead of relying on `latest`.
@@ -215,8 +273,8 @@ Notes:
 - `unschedulable_timeout` controls the Pod-age threshold for failing an unschedulable task; it defaults to `10m` to allow node autoscaling, and `0s` disables that fail-fast behavior
 - `image_pull_policy` defaults to `IfNotPresent`
 - the worker owns a set of names in the task container's environment and writes them last, so an entry with one of these names in `task_env` or `pod_template` is overwritten rather than honoured: `OZ_RUN_ID`, `OZ_WORKER_BACKEND`, `OZ_WORKSPACE_ROOT`, `OZ_ENVIRONMENT_FILE`, and the `WARP_`-prefixed name of each (`WARP_RUN_ID` and so on). Pick different names for operator-supplied variables. Warp reserves the `WARP_` names against managed secrets too, but a worker's own config is yours, so nothing stops you setting them — they just will not survive
-- `sidecar_image` overrides the warp-agent sidecar image reference sent by the server (e.g. `docker.io/warpdotdev/warp-agent:latest`); set this when cluster nodes cannot pull directly from Docker Hub and must use an internal registry mirror or pull-through cache instead. This only affects the warp-agent sidecar (mounted at `/agent`), not any additional sidecars. When using this override, you are responsible for keeping your mirror in sync with `docker.io/warpdotdev/warp-agent` — the server normally sends the correct version-matched image per task, so a stale mirror may cause version incompatibility
-- `coding_cli_sidecars` maps a harness config name (e.g. `claude`, `codex`) to a custom Docker image that will be mounted as the coding CLI sidecar for runs using that harness. When set, the worker replaces the server-provided sidecar image (or injects a new entry if the server did not send one) at the standard mount path `/mnt/{harness}-cli-sidecar`. Use this when your cluster uses a custom or internal Claude Code binary wrapper instead of the Warp-provided image. Example:
+- `sidecar_image` is a legacy shorthand for mapping the official warp-agent repository to a custom image, with the same [remapping rules](#remapping-sidecar-images) as Docker
+- `coding_cli_sidecars` is a legacy shorthand mapping a harness config name (e.g. `claude`, `codex`) to a custom image for its official Warp CLI and ACP repositories. It affects only assigned sidecars; it does not inject missing sidecars or match by mount path. Neither legacy option can be combined with non-empty `sidecar_image_mappings`. Example:
 
 ```yaml
 backend:

charts/oz-agent-worker/templates/configmap.yaml

@@ -30,6 +30,10 @@ data:
         {{- end }}
         {{- if .Values.kubernetesBackend.sidecarImage }}
         sidecar_image: {{ .Values.kubernetesBackend.sidecarImage | quote }}
+        {{- end }}
+        {{- with .Values.kubernetesBackend.sidecarImageMappings }}
+        sidecar_image_mappings:
+{{ toYaml . | indent 10 }}
         {{- end }}
         {{- if .Values.kubernetesBackend.setupCommand }}
         setup_command: |-

charts/oz-agent-worker/values.yaml

@@ -85,6 +85,10 @@ kubernetesBackend:
   useImageVolumes: null
   preflightImage: ""
   sidecarImage: ""
+  # Mapping from Warp-provided sidecar images to mirrored repositories. If the destination
+  # repository includes a tag or digest, that takes precedence over any server-provided
+  # version.
+  sidecarImageMappings: {}
   setupCommand: ""
   teardownCommand: ""
   extraLabels: {}

internal/common/image_remapping.go

@@ -0,0 +1,65 @@
+package common
+
+import (
+	// Register the OCI digest algorithms used by reference parsing.
+	_ "crypto/sha256"
+	_ "crypto/sha512"
+	"fmt"
+	"sort"
+	"strings"
+
+	"github.com/distribution/reference"
+)
+
+// ImageRemapper matches normalized repositories, independently of tags and digests.
+type ImageRemapper struct {
+	repositories map[string]reference.Named
+}
+
+func NewImageRemapper(mappings map[string]string) (*ImageRemapper, error) {
+	remapper := &ImageRemapper{repositories: make(map[string]reference.Named, len(mappings))}
+	sources := make([]string, 0, len(mappings))
+	for source := range mappings {
+		sources = append(sources, source)
+	}
+	sort.Strings(sources)
+	for _, source := range sources {
+		from, err := reference.ParseNormalizedNamed(source)
+		if err != nil {
+			return nil, fmt.Errorf("invalid source repository %q: %w", source, err)
+		}
+		if !reference.IsNameOnly(from) {
+			return nil, fmt.Errorf("source repository %q must not include a tag or digest", source)
+		}
+		if _, exists := remapper.repositories[from.String()]; exists {
+			return nil, fmt.Errorf("duplicate source repository %q after normalization", source)
+		}
+		to, err := reference.ParseNormalizedNamed(mappings[source])
+		if err != nil {
+			return nil, fmt.Errorf("invalid destination for repository %q: %w", source, err)
+		}
+		remapper.repositories[from.String()] = to
+	}
+	return remapper, nil
+}
+
+// Remap applies one mapping. A pinned destination replaces the source version;
+// otherwise the source's tag and digest are carried over to the new repository.
+func (r *ImageRemapper) Remap(image string) string {
+	if r == nil || len(r.repositories) == 0 {
+		return image
+	}
+	from, err := reference.ParseNormalizedNamed(image)
+	if err != nil {
+		return image
+	}
+	repository := reference.TrimNamed(from).String()
+	to, ok := r.repositories[repository]
+	if !ok {
+		return image
+	}
+	if !reference.IsNameOnly(to) {
+		return to.String()
+	}
+	return to.String() + strings.TrimPrefix(from.String(), repository)
+}

internal/common/image_remapping_test.go

@@ -0,0 +1,63 @@
+package common
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestImageRemapper(t *testing.T) {
+	digest := "sha256:" + strings.Repeat("a", 64)
+	otherDigest := "sha256:" + strings.Repeat("b", 64)
+	for _, tc := range []struct {
+		name, source, destination, image, want string
+	}{
+		{"digest preserved", "warpdotdev/warp-agent", "mirror.example/agent", "warpdotdev/warp-agent@" + digest, "mirror.example/agent@" + digest},
+		{"tag and digest preserved", "warpdotdev/warp-agent", "mirror.example/agent", "warpdotdev/warp-agent:v123@" + digest, "mirror.example/agent:v123@" + digest},
+		{"tag pin replaces source digest", "warpdotdev/warp-agent", "mirror.example/agent:approved", "warpdotdev/warp-agent:v123@" + digest, "mirror.example/agent:approved"},
+		{"tag and digest pin", "warpdotdev/warp-agent", "mirror.example/agent:approved@" + otherDigest, "warpdotdev/warp-agent@" + digest, "mirror.example/agent:approved@" + otherDigest},
+		{"implicit latest", "warpdotdev/warp-agent", "mirror.example/agent", "warpdotdev/warp-agent", "mirror.example/agent"},
+		{"registry port", "source.example:5000/agent", "mirror.example:5001/agent", "source.example:5000/agent:v1", "mirror.example:5001/agent:v1"},
+		{"default library repository", "busybox", "mirror.example/busybox", "docker.io/library/busybox:1.36", "mirror.example/busybox:1.36"},
+		{"exact repository match", "warpdotdev/warp-agent", "mirror.example/agent", "warpdotdev/warp-agent-extra:v123", "warpdotdev/warp-agent-extra:v123"},
+		{"invalid assignment left for backend validation", "warpdotdev/warp-agent", "mirror.example/agent", "invalid image", "invalid image"},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			remapper, err := NewImageRemapper(map[string]string{tc.source: tc.destination})
+			if err != nil {
+				t.Fatal(err)
+			}
+			if got := remapper.Remap(tc.image); got != tc.want {
+				t.Fatalf("Remap(%q) = %q, want %q", tc.image, got, tc.want)
+			}
+		})
+	}
+}
+
+func TestImageRemapperRejectsInvalidMappings(t *testing.T) {
+	for _, mappings := range []map[string]string{
+		{"": "mirror.example/agent"},
+		{"bad source": "mirror.example/agent"},
+		{"warpdotdev/warp-agent:latest": "mirror.example/agent"},
+		{"warpdotdev/warp-agent@sha256:" + strings.Repeat("a", 64): "mirror.example/agent"},
+		{"warpdotdev/warp-agent": ""},
+		{"warpdotdev/warp-agent": "https://mirror.example/agent"},
+		{"warpdotdev/warp-agent": "mirror.example/agent@sha256:bad"},
+		{"warpdotdev/warp-agent": "mirror.example/agent", "docker.io/warpdotdev/warp-agent": "other.example/agent"},
+	} {
+		if _, err := NewImageRemapper(mappings); err == nil {
+			t.Fatalf("expected invalid mapping error for %v", mappings)
+		}
+	}
+}
+
+func TestImageRemapperDoesNotChainMappings(t *testing.T) {
+	mappings := map[string]string{"source.example/agent": "mirror.example/agent", "mirror.example/agent": "other.example/agent"}
+	remapper, err := NewImageRemapper(mappings)
+	if err != nil {
+		t.Fatal(err)
+	}
+	mappings["source.example/agent"] = "mutated.example/agent"
+	if got := remapper.Remap("source.example/agent:v1"); got != "mirror.example/agent:v1" {
+		t.Fatalf("Remap = %q, want single-pass immutable mapping", got)
+	}
+}

internal/config/config.go

@@ -57,9 +57,10 @@ type DockerConfig struct {
 	// Kubernetes backend's image_pull_policy: Always, IfNotPresent, Never. Unlike the
 	// Kubernetes backend, an omitted value defaults to Always here, preserving the Docker
 	// backend's original unconditional-pull behavior for existing installations.
-	ImagePullPolicy string     `yaml:"image_pull_policy" validate:"omitempty,oneof=Always Never IfNotPresent"`
-	SidecarImage    string     `yaml:"sidecar_image" validate:"omitempty,no_whitespace"`
-	Environment     []EnvEntry `yaml:"environment" validate:"dive"`
+	ImagePullPolicy      string            `yaml:"image_pull_policy" validate:"omitempty,oneof=Always Never IfNotPresent"`
+	SidecarImage         string            `yaml:"sidecar_image" validate:"omitempty,no_whitespace"`
+	SidecarImageMappings map[string]string `yaml:"sidecar_image_mappings"`
+	Environment          []EnvEntry        `yaml:"environment" validate:"dive"`
 }
 
 // DirectConfig holds direct-backend-specific configuration.
@@ -82,6 +83,7 @@ type KubernetesConfig struct {
 	UseImageVolumes       *bool             `yaml:"use_image_volumes"`
 	PreflightImage        string            `yaml:"preflight_image" validate:"omitempty,no_whitespace"`
 	SidecarImage          string            `yaml:"sidecar_image" validate:"omitempty,no_whitespace"`
+	SidecarImageMappings  map[string]string `yaml:"sidecar_image_mappings"`
 	SetupCommand          string            `yaml:"setup_command"`
 	TeardownCommand       string            `yaml:"teardown_command"`
 	ExtraLabels           map[string]string `yaml:"extra_labels"`
@@ -90,13 +92,9 @@ type KubernetesConfig struct {
 	TTLSecondsAfterFinish *int32            `yaml:"ttl_seconds_after_finished"`
 	WorkspaceSizeLimit    string            `yaml:"workspace_size_limit"`
 	UnschedulableTimeout  *string           `yaml:"unschedulable_timeout"`
-	// CodingCLISidecars maps harness config name (e.g. "claude", "codex") to a custom
-	// Docker image. When set, the worker overrides the server-provided sidecar image for
-	// that harness (or injects a new sidecar entry if the server did not send one),
-	// mounting it at /mnt/{harness}-cli-sidecar. Use this to bring your own Claude Code
-	// wrapper or custom harness binary instead of the Warp-provided sidecar image.
-	// The custom image must place the harness binary in the path where the Warp agent
-	// entrypoint expects it (e.g. /usr/local/bin for most sidecar layouts).
+	// CodingCLISidecars is a compatibility alias for mappings from the official
+	// Warp CLI and ACP repositories for each harness to a custom image.
+	// It cannot be combined with SidecarImageMappings and does not inject sidecars.
 	CodingCLISidecars map[string]string `yaml:"coding_cli_sidecars"`
 	// PodTemplate holds a raw Kubernetes PodSpec that is merged with the worker's
 	// required fields at runtime. Declarative task Job configuration such as

internal/worker/docker.go

@@ -47,7 +47,6 @@ type DockerBackendConfig struct {
 	// empty value defaults to PullPolicyAlways, preserving the backend's original
 	// unconditional-pull behavior for installations that don't set it.
 	ImagePullPolicy string
-	SidecarImage    string
 }
 
 type sidecarVolumeState struct {
@@ -676,8 +675,8 @@ func (b *DockerBackend) prepareSidecars(ctx context.Context, dockerClient *clien
 
 		log.Infof(ctx, "Preparing additional sidecar: image=%s, mount=%s", sidecar.Image, sidecar.MountPath)
 
-		// Additional sidecar images are public, so no auth is needed when a pull is required.
-		if err := b.prepareImage(ctx, sidecar.Image, "", nil); err != nil {
+		authStr := b.getRegistryAuth(ctx, sidecar.Image)
+		if err := b.prepareImage(ctx, sidecar.Image, authStr, nil); err != nil {
 			return nil, fmt.Errorf("failed to prepare additional sidecar image %s: %w", sidecar.Image, err)
 		}
 

internal/worker/kubernetes.go

@@ -97,7 +97,6 @@ type KubernetesBackendConfig struct {
 	ImagePullPolicy       string
 	UseImageVolumes       *bool
 	PreflightImage        string
-	SidecarImage          string
 	SetupCommand          string
 	TeardownCommand       string
 	NoCleanup             bool
@@ -107,10 +106,6 @@ type KubernetesBackendConfig struct {
 	TTLSecondsAfterFinish *int32
 	WorkspaceSizeLimit    *resource.Quantity
 	UnschedulableTimeout  *time.Duration
-	// CodingCLISidecars maps harness config name (e.g. "claude", "codex") to a custom
-	// Docker image for the coding CLI sidecar. When non-empty for a task's harness,
-	// the worker overrides or injects the sidecar at /mnt/{harness}-cli-sidecar.
-	CodingCLISidecars map[string]string
 	// TaskEnv contains runtime-only env overrides from CLI -e/--env. Declarative
 	// task container env in file or Helm config should be set in PodTemplate.
 	TaskEnv map[string]string