Skip to content

[REMOTE-3452] Enforce Kubernetes egress policies for self-hosted executions - #174

Draft
warp-agent-staging[bot] wants to merge 3 commits into
mainfrom
factory/remote-3452-network-policy-worker
Draft

warp-agent-staging[bot] wants to merge 3 commits into
mainfrom
factory/remote-3452-network-policy-worker

Conversation

@warp-agent-staging

@warp-agent-staging warp-agent-staging Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Implement restricted self-hosted Kubernetes assignments so an opted-in worker installs immutable, shared enforcement before any task Job can start. This follows the approved REMOTE-3452 spec at 6c4e205; that docs-only PR is unchanged. Pair with warp-server #20459. No flags, deployment, capability negotiation, or server policy resolution change is included here.

Review guide

Read kubernetes_network_policy.go for provider translation/fingerprints, then kubernetes_network_policy_lifecycle.go for the policy-local CAS ledger and finalization. Focus on the single Job POST, ambiguous reservations, UID/resourceVersion races, and retained terminal objects. Operator prerequisites, recovery and the measured CNI matrix are in the README.

What changed

  • Preserve absent/null versus present-empty policies. Unsupported/disabled workers and invalid rules reject before claiming, with execution-scoped codes. Provisioning failures use terminal network_policy classification.
  • Add disabled-by-default nested file/Helm opt-in and DNS settings. Render the persistent egress-only restricted baseline regardless of opt-in; add only namespaced native/Cilium policy rights when enabled.
  • Prefer a definitely present Cilium API; errors never downgrade. Native policies snapshot all resolved addresses per launch; Cilium tracks exact FQDNs. Empty policies grant neither DNS nor external egress, including the explicit egress: [] required by the Cilium CRD.
  • Hash only canonical final enforcement content. Create the finalizer and first consumer atomically; register other consumers using UID/resourceVersion CAS before one unsuspended Job POST. Disable client-go's implicit replay of Job POSTs and native/Cilium policy CREATE/PATCH writes; recover each committed reservation from its exact API identity. Bind only a fully matching Job UID.
  • Reconcile both provider kinds at startup, every 30 seconds and on terminal notifications. Retain ambiguous launches and all potentially live containers; delete only after all consumers stop, including under NoCleanup. A terminating fingerprint can block new identical launches instead of bypassing protection.

Validation

Passed locally, before the final gofmt -s -w pass:

  • go test ./... -count=1; targeted policy/config/Helm tests; policy lifecycle go test -race.
  • go vet ./..., go build ./..., golangci-lint run (default and integration build tags; 0 issues).
  • Four focused committed-write regressions (native/Cilium × CREATE/registration PATCH) failed before the correction and pass afterward. Each sends HTTP 504 with Retry-After: 0 after committing and asserts exactly one CREATE/PATCH, exact reservation recovery, and zero Job POSTs from registration alone.
  • helm lint ./charts/oz-agent-worker --set worker.workerId=ci-worker --set image.tag=ci; automated default/disabled/enabled/external-RBAC/DNS-override render checks.
  • Real Kubernetes v1.32.2 / Cilium 1.18.3, single-node kind, controlled external IPv4 endpoints and CoreDNS: setup/main allowed and forbidden probes, persistent baseline, empty-policy DNS denial, two-consumer reuse, NoCleanup cleanup, delayed authorized Job POST after DELETE, and continued terminating-policy enforcement before/during/after Cilium agent restart. Existing Job-controller expiry/TTL/retention tests also passed. The README records fixture inputs and the exact reproducible command: go test -tags=integration ./internal/worker -run 'Test(KubernetesNetworkPolicyWithCNI|KubernetesNetworkPolicyTerminatingWithCNI|JobExpirationWithController)' -count=1 -v.

Blocking native acceptance: native Calico 3.29.3 / Kubernetes v1.32.2 remains BLOCKED / NOT VERIFIED. The iptables fixture could not program rpfilter on this sandbox kernel. The documented eBPF/VXLAN alternative also failed: its connect-time BPF program could not load (Invalid argument). Disabling only the optional connect-time load balancer and enabling packet-path NAT made calico-node Ready, but the real suite's unrestricted external control failed and node-side DNS UDP writes returned invalid argument. Only controller expiry/TTL passed; native allow/deny, empty-policy/DNS, and terminating-policy/restart acceptance did not. A compatible isolated native-CNI cluster without the Cilium API, working declared DNS/external IPv4, and test-namespace/CNI-restart permissions is required. No scope removal or acceptance waiver is requested.

Default kindnet is not enforcing. IPv6/multi-node dataplane, in-cluster autodetection/NodeLocal/custom-DNS topology, live RBAC revocation, native DNS snapshot rotation, slow-sidecar cancellation/worker-crash and fenced operator-recovery live fixtures remain unverified; unit/API tests are not substituted for those claims. No rollout is authorized, and no arbitrary native/Cilium version is claimed supported.

The post-correction credential-free Cilium and failed native logs are attached as supplemental artifacts 01a15eff-499f-74b7-a55c-3524c8f8c6d7 and 01a15eff-4b35-7318-9f72-620eed63a2a2. Complete logs are reproduced below for shareable evidence. Logs/kubeconfigs are not committed.

Real-Cilium post-correction run: PASS (193.833 seconds)
=== RUN   TestJobExpirationWithController
=== RUN   TestJobExpirationWithController/ttl
15:01:39.000 INF Requested expiration of abandoned Kubernetes Job abandoned-zvmvm; retaining it according to its TTL
=== RUN   TestJobExpirationWithController/retained
15:01:46.000 INF Requested expiration of abandoned Kubernetes Job abandoned-86l9b; retaining it according to its TTL
--- PASS: TestJobExpirationWithController (8.50s)
    --- PASS: TestJobExpirationWithController/ttl (6.22s)
    --- PASS: TestJobExpirationWithController/retained (2.27s)
=== RUN   TestKubernetesNetworkPolicyWithCNI
    kubernetes_network_policy_integration_test.go:199: Kubernetes=v1.32.2 CNI=cilium-1.18.3 provider=cilium allowed=172.18.0.12 forbidden=172.18.0.13
    kubernetes_network_policy_integration_test.go:272: shared enforcement, setup/main probes, retained Jobs and zero-consumer deletion passed
--- PASS: TestKubernetesNetworkPolicyWithCNI (145.47s)
=== RUN   TestKubernetesNetworkPolicyTerminatingWithCNI
    kubernetes_network_policy_integration_test.go:276: Kubernetes=v1.32.2 CNI=cilium-1.18.3 provider=cilium allowed=172.18.0.12 forbidden=172.18.0.13
    kubernetes_network_policy_integration_test.go:378: terminating UID retained enforcement before/during/after CNI restart and fresh API reconciliation
--- PASS: TestKubernetesNetworkPolicyTerminatingWithCNI (39.86s)
PASS
ok  	github.com/warpdotdev/oz-agent-worker/internal/worker	193.833s
Native Calico eBPF fixture: BLOCKED / NOT VERIFIED
=== RUN   TestJobExpirationWithController
=== RUN   TestJobExpirationWithController/ttl
14:59:44.000 INF Requested expiration of abandoned Kubernetes Job abandoned-psllm; retaining it according to its TTL
=== RUN   TestJobExpirationWithController/retained
14:59:51.000 INF Requested expiration of abandoned Kubernetes Job abandoned-5cv92; retaining it according to its TTL
--- PASS: TestJobExpirationWithController (8.49s)
    --- PASS: TestJobExpirationWithController/ttl (6.22s)
    --- PASS: TestJobExpirationWithController/retained (2.27s)
=== RUN   TestKubernetesNetworkPolicyWithCNI
    kubernetes_network_policy_integration_test.go:199: Kubernetes=v1.32.2 CNI=calico-3.29.3-ebpf provider=native allowed=172.18.0.12 forbidden=172.18.0.13
    kubernetes_network_policy_integration_test.go:206: probe Job failed: {Conditions:[{Type:FailureTarget Status:True LastProbeTime:2026-10-21 14:59:58 +0000 UTC LastTransitionTime:2026-10-21 14:59:58 +0000 UTC Reason:BackoffLimitExceeded Message:Job has reached the specified backoff limit} {Type:Failed Status:True LastProbeTime:2026-10-21 14:59:58 +0000 UTC LastTransitionTime:2026-10-21 14:59:58 +0000 UTC Reason:BackoffLimitExceeded Message:Job has reached the specified backoff limit}] StartTime:2026-10-21 14:59:52 +0000 UTC CompletionTime:<nil> Active:0 Succeeded:0 Failed:1 Terminating:0xdc405cafcc4 CompletedIndexes: FailedIndexes:<nil> UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],} Ready:0xdc405cafcc0}
--- FAIL: TestKubernetesNetworkPolicyWithCNI (6.26s)
=== RUN   TestKubernetesNetworkPolicyTerminatingWithCNI
    kubernetes_network_policy_integration_test.go:276: Kubernetes=v1.32.2 CNI=calico-3.29.3-ebpf provider=native allowed=172.18.0.12 forbidden=172.18.0.13
    kubernetes_network_policy_integration_test.go:279: resolve network policy host allowed.remote3452.test: no complete DNS answer (lookup allowed.remote3452.test on 172.18.0.1:53: write udp 172.18.0.3:50133->10.96.0.10:53: write: invalid argument)
--- FAIL: TestKubernetesNetworkPolicyTerminatingWithCNI (0.01s)
FAIL

@warp-agent-staging

Copy link
Copy Markdown
Contributor Author

This PR was generated with Warp.

Comment @warp-staging-factory on this PR to send it follow-up work.

View run View conversation View on Slack

@warp-agent-staging warp-agent-staging Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

This adds worker-enforced Kubernetes egress policies. Native-provider acceptance remains blocked.

Concerns

  • Native enforcement and terminating-policy/CNI-restart tests have not passed on an enforcing native cluster without the Cilium API. The local Calico fixtures could not establish a working dataplane, so the native fallback's isolation guarantee remains unvalidated. Run the approved live suite in a compatible isolated environment and record provider versions, allow/deny, empty-policy/DNS, and termination/restart results before acceptance.

Verdict

Checks: build pass, tests pass (unit/race/Cilium; native acceptance blocked), CI pending, visual proof n/a

Found: 0 critical, 1 important, 0 suggestions, 0 nits

Responding as wilson: View in factory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants