diff --git a/specs/REMOTE-3452/PRODUCT.md b/specs/REMOTE-3452/PRODUCT.md new file mode 100644 index 0000000..5b12bc4 --- /dev/null +++ b/specs/REMOTE-3452/PRODUCT.md @@ -0,0 +1,35 @@ +# Self-hosted Kubernetes network policies +## Summary +Enforce the server-resolved egress policy for each self-hosted Kubernetes execution. Refuse restricted executions when enforcement is unavailable. Preserve the existing behavior for executions without a policy. +- Work item: [REMOTE-3452](https://linear.app/warpdotdev/issue/REMOTE-3452/self-hosted-kubernetes-network-policy-support). +- Status: [approved by Ben Navetta in the origin thread](https://warpdev.slack.com/archives/C0A3CQ638NQ/p1792589633164009). This PR does not implement the feature. +- The companion [TECH.md](TECH.md) defines the implementation and validation criteria. +## Behavior +1. **Policy presence controls enforcement.** An absent or null resolved policy adds no worker restriction. A present policy with zero rules denies all policy-controlled egress. Do not treat an empty policy as absent. A policy declaration that the server combines with system allowances is not an empty resolved policy. +2. **The server owns policy meaning.** Enforce the resolved ALLOW rules supplied for this execution. Do not reload agent or Factory declarations in the worker. Do not add package repositories, arbitrary internal networks, wildcards, port restrictions, or new system hosts. Leave ingress unchanged. Preserve the server's existing feature gate and nil-policy behavior. +3. **Operators must opt in.** Set `backend.kubernetes.network_policy.enabled` to `true` to enable support; its default is `false`. All network-policy settings belong under `backend.kubernetes.network_policy`, with DNS settings under `dns`. The Helm chart supplies the persistent restricted-Pod default-deny baseline. An operator may enable support only after the documented cluster prerequisites are met and the baseline is enforced. The option declares that the operator has verified the cluster; it is not an automatic CNI security certification. +4. **Use the available enforcement engine.** Use Cilium's DNS-aware policy when its policy API is available. Otherwise, use native Kubernetes NetworkPolicy and resolve each allowed DNS name to IP addresses once before launch. Do not refresh that snapshot during the execution. Do not downgrade to native policy after a Cilium permission or application failure. +5. **Use cluster DNS without requiring ordinary in-cluster configuration.** Autodetect resolver addresses when the worker uses its own cluster's in-cluster configuration. Explicit resolver settings override discovery. Out-of-cluster workers require explicit addresses. Operators must align worker and task DNS views and configure overrides for NodeLocal or custom DNS topologies. Both enforcement modes grant network destinations, not HTTP Host or TLS identity. A permitted IP can serve more than one DNS name. The native snapshot can become stale. Trusted DNS access is not a general data-loss-prevention guarantee. +6. **Fail instead of weakening isolation.** Restricted executions fail with an actionable terminal error on unsupported workers, disabled support, invalid policy, DNS-resolution failure, missing permissions, or policy application failure. Docker, direct, and command backends do not support this feature. Never run the task unrestricted as a fallback. Capacity rejection and a disconnected worker retain their existing queue behavior. +7. **Upgrade the pool before enabling restricted assignments.** This change does not add capability negotiation or server-side worker-version filtering. Old workers ignore unknown JSON fields and can therefore ignore a policy. Operators must upgrade every worker in the target pool before the server sends policy-bearing assignments. Do not claim fail-closed compatibility with old workers or mixed old/new pools. +8. **Protect the entire execution within the operator-configured boundary.** Create or validate the policy and durably register the execution before creating its ordinary unsuspended Job. Isolation covers setup, init containers, regular containers, sidecars, teardown, and debug-idle time. Worker control-plane traffic and node image pulls are outside the task-Pod network boundary. Reuse one immutable policy when its final native Kubernetes or Cilium enforcement content matches, within that resource kind and namespace. Native DNS answers matter through the IP rules they produce. A shared policy fingerprint identifies enforcement content; existing run/execution IDs identify its consumers. Reserve `warp-` metadata for Warp, not operator labels. +9. **Trust the operator-managed cluster boundary.** Enforcement requires an enforcing CNI, an already-enforced restricted-Pod default-deny baseline, trusted DNS, and no other policies that broaden restricted Pods' egress. Operators remain responsible for task credentials and container settings. The worker must not enable host networking, privileged mode, or added `NET_ADMIN`/`SYS_ADMIN` capabilities itself. Preserve operator-supplied settings and do not reject a run solely because the operator chose these options. Such choices can defeat network-policy isolation and fall outside its guarantee. Do not add another opt-in gate or per-execution connectivity probes. +10. **Keep isolation during uncertainty.** Attach a consumer-aware finalizer when creating the policy. Worker shutdown, a lost watch, a logical run outcome, a cancellation acknowledgment, and main-container exit do not authorize removal. Pending launches, active Pods, and Jobs that can still create Pods all count as consumers. Fence registration against deletion so cleanup cannot remove the policy between registration and Job creation. Reuse is not proof of CNI readiness on a new node. +11. **Separate execution cleanup from object retention.** Release a consumer only when its Job cannot start more Pods and all its containers have stopped. Automatically delete the shared policy only after zero execution consumers remain. With `--no-cleanup`, retain terminal Job/Pod objects for inspection but release their consumers. Retained terminal objects alone must not keep a policy. Deliberately restarting retained work requires a newly registered execution. +12. **Recover policy cleanup after disruption.** A replacement worker reconciles durable registrations and policies, including executions that completed while the original worker was absent. Recover a matching Job after an ambiguous launch response. Retry cleanup on API errors. Retain unresolved launch reservations and policies when launch or termination cannot be verified. Do not use an in-memory reference count or expire an ambiguous reservation on a timer. Document verified operator recovery for stuck finalizers and the unsupported-CNI limitations. +## Decisions from the interview +- Ben chose Cilium when available and one-time DNS-to-IP resolution otherwise. He accepted snapshot staleness for the expected run duration. +- Ben required terminal failure when the policy cannot be applied. +- Ben chose operator prerequisites and explicit opt-in through `network_policy.enabled`, with DNS settings in `network_policy.dns`. +- Ben preferred removing policies after completion under `--no-cleanup`. Completion means terminal execution, not only a finished agent conversation. +- Ben explicitly excluded capability negotiation. A coordinated operator-managed upgrade is required instead. +- In review, Ben requested in-cluster DNS autodetection, a Helm-provided baseline, reserved `warp-` labels, existing run/execution identities, and policy creation before Job creation. He accepted operator-controlled weakening of container settings without worker rejection. +- Ben selected shared immutable fingerprint policies, a consumer-aware finalizer, durable launch registration, and automatic safe cleanup. Run/execution identity belongs on each registration, not on the shared policy ID. +- In review, Ben chose content-addressing based only on the final native Kubernetes or Cilium configuration. Existing authenticated routing is sufficient; no additional assignment scope is required. +## Out of scope +- Docker, direct, and custom-command policy enforcement. +- Changes to agent/Factory policy composition, inheritance, `extends`, or the system allowance list. +- Native DNS refresh, strict application-host enforcement, an egress proxy, or new wildcard/CIDR/port policy syntax. +- A full execution reattach/finalization service, CNI installation, or automatic cluster certification. +- Capability negotiation, worker-version dispatch filtering, or an equivalent compatibility handshake. +- Ingress controls, protection from a malicious cluster administrator, and rendered UI changes. diff --git a/specs/REMOTE-3452/TECH.md b/specs/REMOTE-3452/TECH.md new file mode 100644 index 0000000..ba5b383 --- /dev/null +++ b/specs/REMOTE-3452/TECH.md @@ -0,0 +1,222 @@ +# Self-hosted Kubernetes network policy enforcement +## Summary +Implement the behaviors in [PRODUCT.md](PRODUCT.md) across the worker and warp-server. This is a spec-only PR. Approval authorizes later implementation, not deployment or a feature-flag change. +## Context +Inspected source commits: +- Worker: `266a4bea06605f46b3bef978dbd6298fcb24dade`. +- Server: `e8ce83577401d37056ab85122b5b63c459ec0b25`. +- Protocol repository: `30d54b549cf87930f3f071b38ade82bf13305ba2`. Self-hosted assignments use mirrored JSON, not protobuf. No warp-proto-apis change is required. +Current data flow and implementation anchors: +- [Server `dispatch/worker.go:33-38`](https://github.com/warpdotdev/warp-server/blob/e8ce83577401d37056ab85122b5b63c459ec0b25/logic/ai/ambient_agents/dispatch/worker.go#L33-L38) carries `SandboxRequirements.NetworkPolicy`. [Dispatcher `:1862-1875`](https://github.com/warpdotdev/warp-server/blob/e8ce83577401d37056ab85122b5b63c459ec0b25/logic/ai/ambient_agents/dispatcher.go#L1862-L1875) resolves it under `WorkerNetworkPoliciesEnabled`. +- [Server `networking/network_policy.go:28-116`](https://github.com/warpdotdev/warp-server/blob/e8ce83577401d37056ab85122b5b63c459ec0b25/logic/ai/ambient_agents/networking/network_policy.go#L28-L116) already implements REMOTE-3510: declaration union, deny subtraction, system allowances, and nil when neither scope declares a policy. +- [Server `model/types/network_policy.go:11-48`](https://github.com/warpdotdev/warp-server/blob/e8ce83577401d37056ab85122b5b63c459ec0b25/model/types/network_policy.go#L11-L48) defines the resolved rules. [Host validation](https://github.com/warpdotdev/warp-server/blob/e8ce83577401d37056ab85122b5b63c459ec0b25/util/network_hosts.go#L28-L60) accepts normalized literal DNS names and IPv4/IPv6 addresses, not URLs, wildcard hosts, or CIDRs. +- [Server `selfhosted/worker.go:148-198`](https://github.com/warpdotdev/warp-server/blob/e8ce83577401d37056ab85122b5b63c459ec0b25/logic/ai/ambient_agents/workers/selfhosted/worker.go#L148-L198) receives requirements but does not forward the policy. Assignment construction is in [server `websocket.go:838-850`](https://github.com/warpdotdev/warp-server/blob/e8ce83577401d37056ab85122b5b63c459ec0b25/logic/ai/ambient_agents/workers/selfhosted/websocket.go#L838-L850). +- [Worker `internal/types/messages.go:120-149`](https://github.com/warpdotdev/oz-agent-worker/blob/266a4bea06605f46b3bef978dbd6298fcb24dade/internal/types/messages.go#L120-L149) mirrors the assignment. [Worker `worker.go:660-734`](https://github.com/warpdotdev/oz-agent-worker/blob/266a4bea06605f46b3bef978dbd6298fcb24dade/internal/worker/worker.go#L660-L734) claims before backend provisioning. `prepareTaskParams` transfers assignment fields to `TaskParams`. +- JSON decoding does not reject unknown fields. An old worker can silently ignore an added policy field. Ben explicitly excluded capability negotiation; coordinated upgrade is a prerequisite, not a guarantee supplied by this protocol. +- [Worker `kubernetes.go:360-419`](https://github.com/warpdotdev/oz-agent-worker/blob/266a4bea06605f46b3bef978dbd6298fcb24dade/internal/worker/kubernetes.go#L360-L419) creates Jobs, retains failures, and cleans successful Jobs. [Cancellation and shutdown `:571-626`](https://github.com/warpdotdev/oz-agent-worker/blob/266a4bea06605f46b3bef978dbd6298fcb24dade/internal/worker/kubernetes.go#L571-L626) distinguish deleting a Job from preserving it across worker disruption. +- [Worker `kubernetes_api.go:133-248`](https://github.com/warpdotdev/oz-agent-worker/blob/266a4bea06605f46b3bef978dbd6298fcb24dade/internal/worker/kubernetes_api.go#L133-L248) requests asynchronous stop and background deletion. Neither operation confirms that Pods have stopped. +- Config wiring spans `internal/config/config.go`, `main.go:197-309`, `internal/worker/kubernetes.go:93-126`, and Helm `values.yaml` / `templates/configmap.yaml`. The chart's `templates/role.yaml` has no policy permissions. +- `loadKubernetesClientConfig`, `kubernetesTaskJobName`, and `taskExecutionID` in `internal/worker/kubernetes.go` already select clients and name execution Jobs. Reuse the run/execution identities for launch consumers, not for the shared rule fingerprint. +- Existing authenticated server routing offers work only to the worker's own team and host. [Server `SelfHostedSlug` parsing](https://github.com/warpdotdev/warp-server/blob/e8ce83577401d37056ab85122b5b63c459ec0b25/logic/ai/ambient_agents/workers/selfhosted/websocket.go#L168-L195) identifies that routing scope. No additional assignment scope field or worker-side scope validation is required. +## Proposed changes +### 1. Operator configuration and supported boundary +- Add one `backend.kubernetes.network_policy` block. Its `enabled` field defaults to `false`; its `dns` field contains all DNS settings. Mirror it in Helm as `kubernetesBackend.networkPolicy.enabled` and `kubernetesBackend.networkPolicy.dns`. +- Add nested `NetworkPolicy` settings to both `KubernetesConfig` and `KubernetesBackendConfig`, with `Enabled` and `DNS` fields. Wire the complete nested value through `mergeConfig`. Do not retain separate support or DNS settings alongside this block. +- This is a file/Helm option. Do not add a global CLI boolean that can imply support on Docker, direct, or command backends. +- Under `backend.kubernetes.network_policy.dns`, define: + - `server_ips`: optional explicit resolver IPv4/IPv6 addresses; default empty means autodetect for an in-cluster client. Helm: `kubernetesBackend.networkPolicy.dns.serverIPs`. + - `namespace`: DNS Pod namespace; default `kube-system`. Helm: `kubernetesBackend.networkPolicy.dns.namespace`. + - `pod_labels`: DNS Pod labels; default `{"k8s-app": "kube-dns"}`. Helm: `kubernetesBackend.networkPolicy.dns.podLabels`. +- The default file configuration is: +```yaml +backend: + kubernetes: + network_policy: + enabled: false + dns: + server_ips: [] + namespace: kube-system + pod_labels: + k8s-app: kube-dns +``` +- Explicit `server_ips` take precedence. Otherwise, when `loadKubernetesClientConfig` selected `rest.InClusterConfig`, parse the worker's `/etc/resolv.conf` nameserver entries. Kubernetes supplies these addresses for ordinary `ClusterFirst` Pods. Deduplicate valid IPv4/IPv6 addresses; do not use loopback or unspecified addresses as task DNS peers. Do not infer the target cluster's DNS from the worker's local resolver when an explicit kubeconfig or out-of-cluster client was selected. +- Do not add a `kube-system/kube-dns` Service API lookup or cross-namespace Service RBAC. That Service's ClusterIPs can differ from the resolver that Pods actually use, notably with NodeLocal DNS. Document the Service lookup as an operator diagnostic, not the worker's autodetection algorithm. +- When a policy needs hostname rules, require nonempty effective resolver addresses and valid DNS namespace/labels. Missing, unreadable, or unusable autodetection fails that restricted assignment with instructions to set `server_ips`. Do not silently fall back to a public resolver. Empty/IP-only policies need no DNS exception. +- The native resolver sends bounded UDP/TCP lookups to the effective addresses. Use those addresses for native DNS grants and the configured namespace/labels for DNS Pod selection. The task's final DNS configuration must use the same trusted DNS view; do not overwrite an operator's `dnsPolicy` or `dnsConfig`. +- V1's zero-configuration case is ordinary Pod-network tasks with cluster DNS. NodeLocal DNS can expose a node-local address instead of a DNS Pod endpoint; custom forwarding can expose a different DNS view. Autodetection does not configure those topologies or prove Cilium DNS-proxy compatibility. Document explicit resolver/selector and task-DNS overrides plus operator CNI validation for those cases. Do not synthesize broad DNS grants, a Cilium `host` allowance, or a new attestation gate. Apply the same operator responsibility to service meshes and secondary interfaces. +- Add the persistent native egress default-deny baseline to `charts/oz-agent-worker/templates/networkpolicy.yaml`. Select only `warp-network-restricted=true`, set `policyTypes: [Egress]`, and set `egress: []`; do not change ingress. Render it even while support is disabled so operators can install and verify it before enabling workers. Disabled workers never assign that label. Non-Helm operators install the equivalent documented manifest. Do not create this baseline moments before each task. +- The operator must verify that the CNI has handled the baseline before enabling support. Kubernetes exposes no portable policy-enforced readiness status. A Helm success or successful policy POST does not satisfy this prerequisite. +- The supported CNI/version must also retain enforcement for a policy with `deletionTimestamp` set while its finalizer remains. Require the live tests below for both providers. Kubernetes retains the API object; that alone does not guarantee dataplane behavior. Reusing an existing policy does not prove propagation to a newly scheduled node or health after a CNI restart. +- The baseline must not select unrestricted task Pods. Other native, Cilium, or administrative policies must not widen restricted Pods' egress. Admission must preserve the worker's selector labels and the task's network boundary. +- Recommend that task identities cannot create Pods, modify labels/policies, access a privileged daemon, or escape to host networking. The task ServiceAccount remains separate from the worker's policy-management ServiceAccount. Operators own credentials that weaken this boundary. +- Worker-generated Pod/container settings must not enable `hostNetwork`, privileged mode, or added `NET_ADMIN`/`SYS_ADMIN` capabilities. Keep operator-supplied settings through the existing Pod-template merge, including regular/init containers. Do not reject them or add a separate waiver setting. Document that they can bypass enforcement; `hostNetwork` policy behavior is CNI-dependent. Root sidecar-copy init containers remain supported; root UID alone is not privileged mode. +- Do not run connectivity probes per task. Real-CNI tests validate the implementation; production enforcement still relies on the operator's attestation. +### 2. Wire contract and admission +- Add `network_policy,omitempty` to both `TaskAssignmentMessage` types. The server field is `*types.NetworkPolicy`; the worker defines equivalent policy/rule structs in `internal/types/messages.go`. +- Rely on existing authenticated team/host offer routing. Do not add scope transport to assignments or `TaskParams`, scope-specific rejection errors, or control-plane identity plumbing to the Kubernetes backend. +- Preserve `rules`, `hosts`, `action`, and `description`. `action` is the JSON string `ALLOW`. Descriptions do not change enforcement. +- Omit the policy for nil requirements. Preserve a non-nil empty policy as an object. Accept both empty and null `rules` arrays as zero rules; an absent/null policy remains unrestricted. +- Pass the existing requirements pointer through `SelfHostedWorker.Process`, `SendTaskAndWaitForClaim`, and `sendTaskToWorker`. Do not independently resolve declarations in the assignment builder. Transfer it through `prepareTaskParams` to `TaskParams.NetworkPolicy`. +- Do not add advertised capabilities, new handshake headers, worker-version routing, or a server-side opt-in lookup. Leave connection selection unchanged. +- Every worker in a policy-bearing pool must run the upgraded worker. Its local opt-in determines whether it can apply the received policy. A disabled or unsupported upgraded worker rejects restricted assignments; the server cannot enforce this check on old workers. +- Preserve existing disconnected-worker availability/claim-timeout behavior and capacity requeue behavior. +- Validate the received policy before claiming or provisioning: known action, nonempty hosts per rule, and the server's canonical literal-host grammar. Do not silently skip bad rules, invalid addresses, or unknown actions. +- Extend `TaskRejectedMessage` with optional `execution_id` and `code`. New worker rejections echo the assignment execution ID. Codes `network_policy_unsupported` and `network_policy_invalid` are terminal; absent code retains legacy capacity behavior. +- Convert those two rejection codes to non-retryable user errors in the server claim path. Require a matching pending execution and selected connection for coded policy rejections. A late rejection must not finalize a successor execution. +- A disabled/unsupported upgraded worker receiving a policy rejects it through this coded path without invoking its backend. Old workers cannot supply this protection; document the coordinated rollout requirement. +- Keep the current early `task_claimed` timing for accepted work. Actual policy API calls happen after claim, so DNS/API latency does not consume the existing ten-second claim window. +- After claim, report enforcement failures through `task_failed`, not `task_rejected`. Add bounded failure reason `network_policy` to worker metrics and the server's mirrored failure classification. +- Classify this reason as `FAILED`, with automatic retries and new-execution retries disabled. The current generic self-hosted user-error constructor marks failures retryable; special-case this reason rather than inheriting that behavior. +### 3. Provider selection and translation +- Add a small provider boundary in `internal/worker/kubernetes_network_policy.go`: discover provider, compile canonical rules, create/get policy, CAS-patch metadata, list managed policies, and delete with UID/resourceVersion preconditions. Keep the native typed client and a Cilium dynamic client behind this boundary. Put durable registration and cleanup reconciliation in `internal/worker/kubernetes_network_policy_lifecycle.go`. +- Discover `cilium.io/v2` / `ciliumnetworkpolicies` with API discovery when support is enabled. Use a dynamic client for that resource; do not add a full Cilium agent dependency. +- A definite missing group/resource selects native `networking.k8s.io/v1` / `networkpolicies`. A discovery timeout, authorization error, or partial discovery failure is not evidence of absence. +- If the Cilium API exists, select Cilium. A Cilium create/get/list/patch/delete failure must not trigger native fallback. A CRD's existence is not proof that Cilium enforcement is active; the operator attests that prerequisite. +- Discover the provider for each accepted restricted assignment before resource creation. Initialization failure reports terminal `network_policy` failure. Unrestricted assignments do not require policy discovery or new policy permissions. +- Explicitly set `policyTypes: [Egress]` for native policies. For Cilium, set `enableDefaultDeny.egress: true` and omit ingress rules/default-deny changes. +- Use the exact shared-fingerprint reserved selector described below, never an empty selector or the worker's entire pool. +- For native policies, compile literal IPv4 and IPv6 addresses to exact `/32` and `/128` `ipBlock` destinations. Resolve hostname rules once using an injectable resolver. Include all returned A/AAAA addresses; deduplicate and sort them. +- Any required hostname lookup error or zero-address answer fails the execution before its Job is created. Do not emit a partial allow list after resolution failure. +- For Cilium, compile literal addresses to exact `toCIDR` destinations and hostname strings to `toFQDNs.matchName`. Do not use match patterns or widen to `world`, `cluster`, or `host` entities. +- Host rules have no port constraint. Do not restrict task traffic to HTTPS merely because a system host usually uses port 443. +- If hostname rules exist, add DNS UDP/TCP port 53 access only to the configured DNS service: + - Native: separate exact resolver-IP peers and a combined DNS namespace/Pod selector. This covers Service-IP and endpoint policy evaluation without allowing all Pods or all namespaces. + - Cilium: `toEndpoints` matching the configured DNS namespace/labels, with port 53 DNS-proxy rules for the allowed query names. +- With zero resolved rules, emit no DNS exception and no allow rule. Never serialize an empty peer/rule object that means allow-all. +- Do not resolve Cilium hostname rules to worker-side IP snapshots. Cilium manages DNS-to-IP tracking. Treat CNAME chains according to Cilium's documented DNS handling; do not add unrelated hostname grants. +- Cluster Service DNAT and Cilium-managed endpoint identity can deny traffic that an IP rule names. Document these provider limits. Never solve reachability by granting an entire cluster/network. Acceptance tests use controlled external destinations and the declared DNS service. +### 4. Canonical shared policies +- Content-address the final native NetworkPolicy `spec` or Cilium policy `spec` after translation and native DNS resolution. Canonicalize the complete enforcement content, including selectors and egress/ingress settings, as deterministic JSON. Normalize API defaults and IP spellings; sort and deduplicate unordered sets. Do not put non-enforcing descriptions in the rendered spec. +- The generated policy name and `warp-network-policy-id` selector value depend on the digest. Compile with one fixed placeholder, the string `POLICY_ID`, for only that selector value, hash the canonical spec, then substitute the digest. Never send the placeholder to Kubernetes. On reuse, first verify the actual generated selector equals the expected digest, then apply the same placeholder normalization for comparison. Do not exclude other selector fields. +- Hash only this canonical enforcement content. Do not salt it with tenant/pool, server identity, compiler version, or DNS source settings. Exclude all object metadata, including the generated name, bookkeeping labels/annotations, finalizers, and consumer ledger. Resource kind and namespace bound the API lookup; they are not additional digest inputs. +- Native DNS resolution still runs for each launch before lookup. Answers affect identity only through the final IP/DNS rules. Equal rendered specs reuse even when source hostnames or unused settings differ; changed rendered CIDRs produce a different policy. Do not refresh an existing shared policy. +- Cilium identity includes its final exact FQDN, CIDR, and DNS-proxy rules. Do not add worker-side DNS snapshots; Cilium tracks answers without changing those rules. +- Set `policyID` to lowercase, unpadded base32 of the full SHA-256 canonical-spec digest (52 characters). Name the policy `warp-np-` (60 characters). Use bookkeeping labels `warp-network-managed=true` and `warp-network-policy-id=`; neither label is a separate identity input. +- The policy selects exactly `warp-network-restricted=true` and `warp-network-policy-id=` in its namespace. Add those labels to each consumer's Job and Pod template. Matching final content may protect multiple executions, including executions launched by different workers sharing the namespace. +- Store the canonical enforcement content in `warp-network-policy-canonical` for diagnosis and the versioned consumer ledger in `warp-network-policy-consumers`. These annotations do not determine enforcement identity or replace live-spec validation. The ledger changes; the selector and enforcement spec do not. Add finalizer `warp.dev/network-policy-consumers` in the original create request. Never attach a per-Job owner reference. +- Reserve task label keys beginning `warp-` for Warp. Reject such keys in operator-supplied task extra labels, including Helm input. Worker-owned `warp-` annotations take precedence over conflicting extra annotations. Do not rename unrelated existing `oz-*` metadata. +- Create the complete policy atomically, with its finalizer and first launch registration in the same request. On a create response, ambiguous create response, or `AlreadyExists`, GET the object in the selected resource kind/namespace. Verify its actual generated selector and compare the complete live enforcement content with the intended canonical spec. Check the managed label, finalizer, and valid ledger for lifecycle safety, not as extra identity dimensions. Ignore server metadata/status in the content comparison. +- `AlreadyExists` permits reuse only after that full match. A name/hash/annotation match alone is insufficient. Do not adopt an unmanaged object, repair a mismatched spec, add a missing finalizer after creation, or overwrite rules. Report a terminal `network_policy` error and make zero Job-create calls on any conflict. +- For reuse, combine content validation with registration: CAS-patch the exact policy UID/resourceVersion that was validated. A concurrent spec/metadata change must fail the CAS and trigger a fresh read and validation. Trusted operators/admission must not modify the spec after registration. +- The persistent, pre-enforced baseline supplies startup isolation while allow rules propagate. Reuse reduces policy creation churn, but does not replace that baseline or prove CNI readiness. Do not add a sleep or treat a policy POST/GET as an enforcement acknowledgment. +### 5. Durable registration and policy-before-Job launch +- Keep nil-policy launches unchanged. Restricted launches follow `compile -> create/validate policy and register consumer -> create ordinary unsuspended Job -> bind Job UID`. Do not add a prepared/suspended/resume or time-based abort protocol. +- Identify each consumer by the full tuple `(runID=TaskID, executionID=taskExecutionID(params))`. Store both full IDs, expected `kubernetesTaskJobName`, launcher process incarnation, and optional bound Job UID. The containing object's UID scopes the ledger; capture it from the validated create/GET response rather than trying to put a server-generated UID in the initial create body. Use one process-incarnation value per worker process to distinguish a crashed launcher from its replacement; do not invent a per-execution UUID. +- Persist registrations in the policy's annotation ledger, not an in-memory count or a separate unfenced ConfigMap. CAS all ledger updates by UID/resourceVersion. Append only to a matching, nonterminating policy with the finalizer present. The initial create installs its first entry atomically; subsequent consumers register by CAS before any Job request. +- Only the launcher that successfully inserts a new registration can submit its Job-create request. Serialize duplicate assignments by run/execution identity. An existing entry is not authorization for another process to create or retry a Job. A successor execution has a new consumer tuple even when it reuses the same policy. +- If a create/registration response is ambiguous, recover its exact entry and launcher incarnation from the API before proceeding. Another launcher's entry is not a successful registration for this process. A replacement reconciles it but does not take over an ambiguous launch. +- Bound the complete encoded annotations below Kubernetes' 256 KiB annotation limit. If an additional registration or canonical spec will not fit, fail before Job creation. Never evict live registrations, split an unfenced ledger, or silently fall back to a per-execution policy. +- Registration and collector DELETE must use the same policy resourceVersion fence. If registration wins, a collector's zero-consumer DELETE precondition fails. If DELETE wins, registration observes `deletionTimestamp` or conflicts and must not submit a Job against that UID. +- A registration that won before deletion remains a consumer, even if its Job has not appeared. Its original launcher may complete the already-authorized create against that UID; the finalizer must retain the policy. New consumers must not join a terminating policy. +- If the matching policy is already terminating before registration, wait for that UID to disappear and retry create within the provisioning budget. Do not remove its finalizer, join it, mutate its name/content, or create a second selector variant to bypass cleanup. A long-lived or ambiguous consumer can therefore block a new identical-policy launch; surface that cause as a terminal failure when the budget expires. +- After registration, create the Job with existing `backoffLimit: 0` and `restartPolicy: Never`. Add full `warp-run-id`, `warp-execution-id`, and `warp-network-policy-uid` annotations to the Job and Pod template. Validate full identity and selector on the returned Job; Job `AlreadyExists` is not permission to overwrite or relaunch. +- Submit at most one Job-create attempt per registration. Do not retry an ambiguous POST. GET the expected name and adopt only a matching run/execution, policy UID, and reserved selector. Record the resulting Job UID by ledger CAS. A name conflict, unknown result, or failed UID binding retains the registration and policy; report failure and request stop for any confirmed matching Job. +- Give restricted provisioning a two-minute context budget. Policy validation/registration failure makes zero Job-create calls. Do not launch task code solely to report a worker error. +- The owning launcher may remove an unbound registration only after it has irrevocably ended provisioning and proves no Job create was submitted, or received a definite API rejection with no possible successful create. A deadline, cancellation, connection loss, one `NotFound`, or old registration age is not such proof. Never release one while an authorized/in-flight create can still succeed. +- A bound registration authorizes no further creates. Do not recreate a deleted/terminal execution from that entry. Keep existing completed-assignment suppression and server execution IDs; this feature does not add a durable global execution-replay service. +- Never delete the policy or drop a registration in a generic `ExecuteTask` defer. Failure/cancellation can return before Kubernetes stops containers. +### 6. Consumer-aware finalization and recovery +- Reconcile at worker startup, every 30 seconds while support is enabled, and immediately after terminal observations. List both provider kinds with `warp-network-managed=true` in the configured namespace, including terminating policies. Check all consumers of each shared policy, not only the current worker's executions. Do not filter policy reuse or cleanup by pool/server identity. A provider change must not hide old policies from cleanup. No server secret or full run reattachment is required. +- Kubernetes supports same-namespace Job owner references on native and Cilium policies. Do not use them for this shared policy. Background deletion removes the owner first; foreground deletion waits for blocking dependents but does not order sibling policy deletion after Pod termination. Multiple owner references still do not count pending launches. A finalizer on the Job alone can delay Pod garbage collection and cannot clean policy state for retained terminal Jobs. +- The policy's qualified finalizer retains its API object after DELETE; it does not discover consumers automatically. Reconcile its durable ledger and all matching restricted Jobs/Pods. Pending registrations, nonterminal/restartable Jobs, live Pods, and unconfirmed termination all prevent finalizer removal. +- Recover an unbound entry by GET of its expected Job name. Verify full IDs, policy UID, and selectors, then CAS-bind the Job UID. If no matching Job can be established, retain the reservation. Do not delete it because its worker disconnected, its process restarted, its execution failed logically, or a timeout elapsed. +- For a bound entry, re-read the Job by name and compare the full UID. If it exists, require `Complete` or `Failed`. A nonterminal Job with zero Pods can still launch work and is a consumer, including suspended or controller-retryable Jobs introduced outside the ordinary worker path. +- List Pods by captured Job controller UID and verify owner references. Require terminal phase and no running/waiting regular, init, native-sidecar, or ephemeral container. Keep the registration for Pending, Running, Unknown, missing status, or unconfirmed termination. Job terminal-condition timing differs across Kubernetes versions; a Job condition alone is insufficient. +- For an absent Job, require all Pods of its captured UID to be terminal or absent. Do not infer physical termination from force deletion or a partitioned node. A name-reused Job is not adopted as the old consumer; inspect its full identity and retain the policy on conflict. Operator cluster fencing remains outside the worker's guarantee. +- Independently list Jobs/Pods matching the shared selector before finalization. A live/restartable or unidentifiable matching object blocks deletion even if a ledger entry is missing. A confirmed terminal retained Job/Pod does not block it. Report unexplained live objects; never repair the ledger by assuming they are stopped. +- Remove a stopped consumer by CAS while preserving all other entries. Normal success/failure and `NoCleanup` use the same stop predicate. Preserve terminal inspection objects, failed-Job retention, and TTL behavior. A still-running sidecar/debug-idle process remains a consumer. +- When the ledger is empty and no matching execution consumer remains, automatically request policy DELETE with the captured UID/resourceVersion preconditions. On a conflict, start the check again. An external DELETE can set `deletionTimestamp` earlier; keep the finalizer until the same zero-consumer predicate is satisfied. +- Once terminating, take fresh API observations and re-read the ledger before removing only `warp.dev/network-policy-consumers` by UID/resourceVersion CAS. Never remove other finalizers. The terminating-policy registration ban plus pending-launch entries closes the list-then-create race; a last Pod scan alone does not. +- On cancellation, background-delete the Job as today and retain its registration through Pod grace periods. Continue reconciliation after the task context ends. On worker shutdown, preserve live Jobs, policies, and reservations. Concurrent replacement workers use the same API CAS fence, not local reference counts. +- Retry API errors with existing Kubernetes backoff and a detached bounded cleanup context. `NotFound` for the expected policy UID is success. Permission failures, stale reads, malformed metadata, unexplained matching consumers, or an unknown consumer-ledger schema retain the finalizer and emit actionable diagnostics with policy UID and consumer IDs. +- A replacement automatically cleans confirmed-terminal consumers and recovers matching unbound Jobs. A crash before Job creation, or a Job that completed and disappeared before UID binding, can leave an irreducibly ambiguous reservation. Do not claim automatic liveness for that case. +- Document operator recovery: fence every launcher that could still submit the execution's Job; establish that no create remains in flight; inspect expected Job names, full IDs, UIDs, and Pods; confirm stopped/fenced nodes; then CAS-remove only the verified stale entries and let normal finalization run. Removing the finalizer directly is a break-glass action outside the guarantee. +- If no reconciler runs, or API rights/CNI behavior prevent progress, policies can remain terminating indefinitely. A deleting shared policy cannot be resurrected; same-name recreation waits for actual removal. Do not promise automatic repair of missing enforcement, force-removed finalizers, or operator-mutated policies. +### 7. Helm, documentation, and rollout +- When `kubernetesBackend.networkPolicy.enabled` is `true`, add a namespaced Role rule for `networking.k8s.io` `networkpolicies`: `create`, `get`, `list`, `patch`, `delete`. Its omitted/default value is `false`. +- Also grant `cilium.io` `ciliumnetworkpolicies` with those verbs. An RBAC rule for a missing resource is harmless; Helm must render without the CRD installed. Do not install a Cilium CRD or ClusterRole. +- Existing Job patch rights support the existing stop path. Existing Pod get/list/watch rights support termination checks. The API discovery behavior uses the cluster's standard discovery permissions; surface denial rather than granting cluster administration. DNS autodetection requires no additional API permissions. +- Update `templates/configmap.yaml`, values, config validation, and merge tests together for the single nested `networkPolicy.enabled` / `networkPolicy.dns` Helm block and `network_policy.enabled` / `network_policy.dns` file block. Include the baseline from section 1 in both disabled/enabled renders and with `rbac.create=false`; baseline creation is separate from worker RBAC. Do not condition its existence on a per-run assignment. +- Document the Helm/non-Helm baseline, DNS autodetection/overrides, additive policies, operator-controlled security weakening, and the fact that node image pulls are not Pod egress. Explain immutable fingerprint reuse, native per-launch snapshots, registration limits, automatic zero-consumer cleanup, terminating-policy launch failures, and the verified stuck-reservation/finalizer recovery procedure. +- Publish a tested Kubernetes/CNI version matrix with enforcement-while-terminating results. Require revalidation on CNI upgrades. Do not describe arbitrary native CNIs or Cilium versions as supported solely because the API accepts their policies. +- Publish the upgraded worker/chart first. With policy-bearing dispatch disabled for the target pool, upgrade every connected replica to an immutable supported image, verify cluster prerequisites, and explicitly set the local opt-in. +- Only then deploy or enable the assignment-plumbing change that sends non-nil policies. Coordinate with the existing server feature gate and policy configuration; do not add a new negotiation mechanism or assume the global gate is pool-scoped. +- If old and new worker replicas coexist while policy-bearing dispatch is enabled, old replicas can execute unrestricted. This is an unsupported, insecure rollout state, not a case the new server detects. Rollback requires stopping policy-bearing assignment before reintroducing old binaries. +- Preserve the existing `WorkerNetworkPoliciesEnabled` gate. Turning it off intentionally restores nil-policy dispatch for new executions; it is not a secure workaround for unsupported workers. +- Drain existing restricted Jobs, durable reservations, and policies before removing policy RBAC, uninstalling the chart/baseline, or disabling reconciliation. Preserve handling for in-use consumer-ledger schema versions during upgrades; never remove live objects/finalizers or mutate shared rules as a rollback shortcut. +## Decisions and alternatives +- **Cilium first, native snapshot second — Ben's choice.** Cilium tracks DNS changes but requires its API/CNI. Native policy works with more CNIs but freezes A/AAAA answers and shares IP-level allowances. Rejecting all non-Cilium clusters and building a proxy were not chosen. +- **No native refresh — Ben's choice.** It reduces controller complexity for short-lived runs. Long/debug-idle executions can lose reachability or retain stale allowed IPs; the spec does not hide that trade-off. +- **Operator attestation — Ben's choice.** Disabled-by-default opt-in makes the prerequisite explicit. Per-run probes add latency and can confuse an unreachable target with correct enforcement; they do not prove universal isolation. +- **No capability negotiation — Ben's explicit scope limit.** A handshake could protect mixed-version pools, but is not part of this change. Coordinated upgrade avoids that protocol expansion and leaves old-worker safety dependent on the rollout prerequisite. +- **Terminal enforcement failures — Ben's choice.** Existing capacity requeue semantics cannot represent permanent incompatibility. Coded preclaim rejection and non-retryable postclaim failure preserve this distinction. +- **Execution-bound cleanup — Ben's preference, clarified.** Retained Job objects do not require retained policies after all execution stops. Deleting on agent conversation completion is unsafe for idle processes and sidecars. +- **Automatic in-cluster DNS, explicit overrides elsewhere — review decision.** Read the Pod resolver configuration without adding Service-read RBAC. A `kube-dns` Service lookup can miss NodeLocal/custom DNS. Discovery reduces ordinary setup; it does not certify alternate DNS topologies. +- **Operator-controlled container settings — Ben's choice.** Preserve explicit operator weakening instead of rejecting it. Worker-generated settings remain safe by default. The policy guarantee cannot cover host-network or privileged escape paths chosen by the operator. +- **One network-policy configuration block — Ben's review decision.** Keep `enabled` and `dns` under `network_policy` in the file and `networkPolicy` in Helm. Separate sibling support/DNS settings were not chosen. Enabling support requires explicit `enabled: true`; omission remains disabled. +- **Existing authenticated routing — Ben's review decision.** Workers receive offers for their own team and host. Additional scope transport/validation and local control-plane identity plumbing duplicate that boundary and are not part of this change. +- **Shared immutable content-addressed policy — Ben's choice.** Per-execution policies simplify ownership but create/delete one object per launch. Share by the final enforcement spec, with generated self-references normalized as described above. Separate scope/compiler/DNS-input salts prevent reuse of identical content without changing enforcement and were not chosen. Native per-launch resolution remains necessary because it determines the rendered IP rules. Never mutate a shared allow list for one execution. +- **Policy before Job; run/execution consumer identity — Ben's choice.** Rule fingerprints deliberately replace run/execution IDs as shared policy IDs. Existing run/execution pairs still identify launch registrations and Jobs. No fresh per-execution UUID, suspended Job, or resume protocol is required. +- **Consumer-aware finalizer and durable registration — Ben's choice.** A policy finalizer plus Pod scanning alone races an in-flight launch. A small CAS ledger on that same policy atomically fences registration and deletion without a new CRD/database. In-memory refcounts and an unfenced separate ConfigMap lose that guarantee. The ledger trades bounded metadata capacity/contention and possible safe leaks for a smaller controller. +- **Automatic zero-consumer cleanup — Ben's choice.** Permanent policy caching avoids finalization races but retains obsolete snapshots. Job ownership does not order policy-after-Pod deletion and is unsuitable for shared rules. The selected reconciler deletes after all consumers stop, including under `NoCleanup`; ambiguous launches can instead require verified operator recovery. Age-based reservation expiry was rejected because a delayed Job request can still succeed. +## Assumptions +- **Assumption:** Ordinary in-cluster worker and task Pods use the same cluster-DNS view. Operators can supply explicit resolver/selector settings for out-of-cluster workers or differing DNS topologies; autodetection cannot verify this equivalence. +- **Assumption:** Provider initialization can use standard API discovery. Missing discovery permission is a failure, not permission to downgrade. +- **Assumption:** Operators can hold policy-bearing dispatch until all replicas in a target pool are upgraded. The opt-in cannot compensate for an old replica that ignores the field. +- **Assumption:** A policy-local, bounded annotation ledger is proportionate to expected concurrent execution counts. Metadata exhaustion fails closed; introducing a scalable external registry is deferred. +- **Assumption:** Full task reattachment and durable global replay detection are unnecessary for confirmed-terminal policy cleanup. Persisted policy/Job/Pod metadata is sufficient. Ambiguous unbound reservations can require launcher/node fencing and verified operator cleanup. +- **Assumption:** The two-minute provisioning budget and 30-second sweep interval are initial internal constants, not new product settings. +- No unanswered interview decisions remain. These implementation assumptions remain visible for spec review. +## Out of scope +Apply the product non-goals. Do not change server policy resolution, introduce a new policy language, promise strict hostname/DLP isolation, or implement general-purpose sandbox hardening. Do not add capability negotiation, CNI health certification, a per-node probe service, or a full worker execution-recovery protocol. +## Validation criteria +These are required implementation checks, not results claimed by this spec-only PR. New test names below are proposed acceptance-test names. +### Worker tests +- `TestNetworkPolicyAssignmentJSON`: nil/null/absent versus `{}`/empty/null rules; ALLOW wire string; IPv4/IPv6/hostname fields; and description preservation. A policy-bearing assignment needs no additional team/host scope field. Covers product behaviors 1–2 and 7–8. +- `TestNetworkPolicyConfigAndBackendAdmission`: omitted block and omitted `enabled` both default to disabled; explicit `backend.kubernetes.network_policy.enabled` opts in; nested `NetworkPolicy.Enabled` / `NetworkPolicy.DNS` survive config loading and `mergeConfig`. Cover non-Kubernetes backends and local opt-in rejection. Assert no new scope transport/validation, handshake/header, or server worker-version filter. Covers 3, 6–7, 9. +- `TestNetworkPolicyDNSConfiguration`: read `backend.kubernetes.network_policy.dns`; verify explicit-IP precedence, ordinary in-cluster resolv.conf autodetection, dual-stack/deduplication, malformed/empty/unreadable resolver data, no loopback/unspecified peers, and no local autodetection with an out-of-cluster/explicit-kubeconfig client. Verify no Service lookup or added Service RBAC. Missing usable DNS fails hostname policies, not empty/IP-only policies. Document NodeLocal/custom-DNS fixtures as topology overrides, not universal autodetection. Covers 3 and 5–6. +- `TestNetworkPolicyTemplateAndMetadata`: worker-generated settings do not enable the bypass options in behavior 9; explicit operator Pod/container options survive the merge and do not trigger policy rejection. Reserved `warp-` operator labels fail configuration validation. Full run/execution and policy UID annotations survive on Job/Pod metadata. Consumer tuple conflicts fail rather than overwrite. Covers 8–9. +- `TestNetworkPolicyPreclaimRejection`: unsupported/invalid policy sends execution-scoped coded rejection, makes no backend/resource call, and does not consume one-shot acceptance or leak a concurrency slot. Covers 6–7. +- `TestKubernetesNetworkPolicyProviderSelection`: present Cilium, definitely absent API, discovery denied/partial/timeout, and Cilium permission/create failure. Only definite API absence uses native. Covers 4 and 6. +- `TestNativeNetworkPolicyTranslation`: multiple A/AAAA results, canonical exact CIDRs, duplicate hosts, DNS TCP/UDP peers, lookup failure/no answers, and no refresh. Covers 1–2, 4–5. +- `TestCiliumNetworkPolicyTranslation`: exact names, literal CIDRs, restricted DNS query rules, no ingress changes, no broad entity grants, and no unintended port restrictions on host grants. Empty policy must deny all without DNS. Covers 1–2, 4–5. +- `TestKubernetesNetworkPolicyFingerprint`: reordered/duplicate sets normalize identically; different final enforcement fields change identity. Native sources that render identical IP/DNS rules reuse; changed rendered CIDRs do not. Cilium hashes its exact rendered rules, not DNS answers. Changes only to compiler identity, tenant/pool/server identity, unused DNS inputs, descriptions, finalizers, or ledger metadata leave the digest unchanged. Verify placeholder substitution is stable, no placeholder reaches the API, and altered actual selectors fail validation. Assert the 52-character ID and 60-character name. Covers 4–5 and 8. +- `TestKubernetesNetworkPolicySharedReuse`: concurrent launches, including different workers sharing a namespace, create one matching immutable policy and distinct consumer entries. The same spec has the same digest across namespaces; resource kind/namespace still select distinct API objects. Validate create/AlreadyExists content, API default normalization, admission mutations, digest/spec conflicts, missing finalizer, UID replacement, and resourceVersion races. Diagnostic annotations cannot substitute for live-spec comparison or create a separate identity gate. Only matching content can be registered; no rule overwrite or per-Job owner reference is allowed. Covers 6 and 8. +- `TestKubernetesRestrictedLaunchOrdering`: policy plus finalizer/registration precedes ordinary Job creation; failure makes zero Job-create calls. Cover initial-create/registration ambiguity, duplicate launchers, wrong launcher incarnation, Job identity conflict, ambiguous Job POST, deadline, and failed UID binding. Assert at most one Job POST per registration and no suspend/resume state. Covers 6 and 8–10. +- `TestKubernetesNetworkPolicyRegistrationFinalizerRace`: deterministically interleave last-consumer cleanup, new registration, DELETE, an accepted-but-delayed Job POST, and finalizer removal. If registration wins, the stale DELETE conflicts. If DELETE wins, no new consumer can join that UID. A pre-deletion pending registration retains the terminating policy until its matching Job stops or launch absence is proven. Covers 8 and 10–12. +- `TestKubernetesNetworkPolicyCleanup`: multiple consumers, success/failure, `NoCleanup`, regular/native/ephemeral sidecars, Pending/Unknown Pods, nonterminal zero-Pod/restartable Jobs, background cancellation, TTL-deleted Jobs, name reuse, and matching live objects without ledger entries. Never finalize before the last consumer stops; retained terminal objects alone do not block it. Covers 10–12. +- `TestKubernetesNetworkPolicyRecovery`: worker death before/after registration, before/after Job POST, and before UID binding; lost watches; concurrent replacement workers; absent/ambiguous Jobs; provider changes; malformed/unknown schemas; forbidden API calls; and annotation size limits. Recover matching UIDs, retain unresolved reservations indefinitely without timer release, and exercise documented fenced operator recovery. Assert no global in-memory refcount or automatic recreation of a bound execution. Covers 10–12. +- Run `go test ./...`, `go vet ./...`, `go build ./...`, and the repository CI's formatting/lint checks. +### Server tests +- Extend `selfhosted/websocket_test.go` and `worker_test.go` for exact requirements passthrough, nil/empty preservation, and unchanged authenticated team/host routing and unrestricted dispatch. Assert no additional assignment scope field or scope-specific rejection is required. +- Test coded preclaim rejection fencing by execution and selected connection. A late message cannot reject a successor. Capacity and legacy uncoded rejections still requeue. +- Add dispatcher and Temporal offer tests that assert `FAILED` and non-retryability after an upgraded worker's coded policy rejection. Disconnected/capacity cases retain existing behavior. +- Test `network_policy` postclaim failure as a non-retryable user failure. Verify it does not inherit the generic constructor's retry flag. +- Run `go test ./logic/ai/ambient_agents/workers/selfhosted`, `go test ./logic/ai/ambient_agents/networking`, and `go test ./logic/ai/ambient_agents` in warp-server. +### Helm and live Kubernetes tests +- Run `helm lint ./charts/oz-agent-worker --set worker.workerId=ci-worker --set image.tag=ci`. +- Run `helm template oz-agent-worker ./charts/oz-agent-worker --namespace agents --set worker.workerId=ci-worker --set image.tag=ci` for the default block, explicit `--set kubernetesBackend.networkPolicy.enabled=false`, explicit `--set kubernetesBackend.networkPolicy.enabled=true`, and enabled support with `--set rbac.create=false`. For enabled renders, test automatic DNS and explicit `kubernetesBackend.networkPolicy.dns.serverIPs`, `.namespace`, and `.podLabels` overrides. Verify rendered file settings all live under `backend.kubernetes.network_policy`; absent/default enablement remains false. Every render must include the namespaced baseline with exactly `warp-network-restricted=true`, egress-only default deny, and no DNS grant. Disabled support adds no policy RBAC; enabled support grants no cluster-scoped or Service permissions. +- Add integration-tagged `TestKubernetesNetworkPolicyWithCNI` and `TestKubernetesNetworkPolicyTerminatingWithCNI` using the existing `OZ_K8S_TEST_KUBECONFIG` disposable-cluster convention. Run `go test -tags=integration ./internal/worker -run 'Test(KubernetesNetworkPolicyWithCNI|KubernetesNetworkPolicyTerminatingWithCNI|JobExpirationWithController)' -count=1`. +- Run it separately against an enforcing Cilium cluster and an enforcing native-policy cluster without the Cilium API. Record cluster/CNI versions and provider selection. Fake Kubernetes clients cannot prove enforcement. +- Use controlled external allowed/forbidden endpoints with distinct IPv4 addresses and, on dual-stack clusters, distinct IPv6 addresses. Use controlled DNS with multiple records. Assert allowed reachability and blocked direct-IP/hostname destinations from setup and task containers. +- Verify the empty resolved policy blocks DNS and external egress. Verify a nil policy does not acquire restricted labels or a consumer registration. +- Run the worker in-cluster with no explicit resolver addresses and verify ordinary cluster-DNS discovery. Repeat with an explicit override and with an out-of-cluster worker; assert the latter requires the override for hostname policies. +- Remove policy-create permission and assert terminal failure before task code runs. Restore it and confirm a new execution succeeds. Verify a present Cilium API with denied Cilium rights does not use native fallback. +- Launch two executions with identical rules and verify one policy, two registrations, and no spec changes. Complete one execution while the other runs; verify enforcement and the remaining consumer. With `--no-cleanup`, stop the last execution and verify policy deletion while terminal Job/Pod objects remain. +- For native policies, change the controlled A/AAAA answer between launches and verify a new fingerprint with the new snapshot. Keep the first execution running and verify its old snapshot remains unchanged. For Cilium, verify same-rule reuse while DNS tracking remains provider-managed. +- Exercise cancellation with a slow sidecar shutdown, worker termination before Job completion, replacement reconciliation, ambiguous Job responses, and crashes before UID binding. Confirm safe automatic cleanup where proof exists and retained reservations where it does not. Exercise the documented fenced operator recovery for the latter. +- In `TestKubernetesNetworkPolicyTerminatingWithCNI`, DELETE the shared policy while a registered Pod is active and hold it terminating. For each recorded supported CNI/version, verify its UID/spec remain and both allowed reachability and forbidden-IP/hostname blocking continue. Restart the worker/reconciler and, in a separate fixture, the CNI agent while terminating; repeat probes before removing the last consumer. A CNI that drops enforcement on deletionTimestamp fails support, even if its API object persists. +- Exercise registration versus finalization with a delayed Job-create request against the real API server. Confirm no Job can appear after its protection disappears. For new consumers of a terminating fingerprint, verify wait/recreate after removal or a bounded terminal failure, not attachment to the old UID. +- The production opt-in is an operator attestation; these tests do not claim that an API-only check can detect a non-enforcing production CNI. No UI/computer-use verification is required. +- Verify the rollout instructions against a mixed-version fixture: an old JSON decoder ignores the new field. Record this as the known compatibility limit, not a passing security scenario. Require evidence that all target replicas use the upgraded image before enabling policy-bearing dispatch. +## References +- [Kubernetes NetworkPolicy semantics, asynchronous handling, and host-network limitations](https://kubernetes.io/docs/concepts/services-networking/network-policies/). +- [Cilium DNS-based egress policies](https://docs.cilium.io/en/stable/security/dns/). +- [Job terminal conditions and Kubernetes version differences](https://kubernetes.io/docs/concepts/workloads/controllers/job/#terminal-job-conditions). +- [Native sidecar lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/). +- [Pod resolver configuration and DNS policies](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/). +- [NodeLocal DNSCache and its differing resolver addresses](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/). +- [Owner references are supported across object kinds in one namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/). +- [Foreground/background garbage collection and blocking dependents](https://kubernetes.io/docs/concepts/architecture/garbage-collection/#cascading-deletion). +- [Finalizers retain API objects, cannot be added after deletion starts, and require qualified names](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/). +- [Cilium v1.18.3 CNP upsert/delete handling](https://github.com/cilium/cilium/blob/v1.18.3/pkg/policy/k8s/cilium_network_policy.go) imports on upsert and removes on a delete event. [Resource event generation](https://github.com/cilium/cilium/blob/v1.18.3/pkg/k8s/resource/resource.go#L536-L559) produces Delete when the object is absent from the store. This source evidence motivates testing retained terminating objects; it is not a guarantee for every CNI/version or a substitute for the live checks.