ci: run the agent e2e suite on kind for each pull request (MK8S-450)

.github/workflows/agent-ci.yaml

@@ -50,3 +50,31 @@ jobs:
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
       - name: Build the container image
         run: make -C agent docker-build
+
+  e2e:
+    runs-on: ubuntu-24.04
+    # Without a limit, a cluster that never gets ready holds the runner for
+    # the 6 hour default.
+    timeout-minutes: 30
+    env:
+      # Taken from the official image, so Renovate can bump the tag and the
+      # digest together. kubectl supports a cluster one minor version away,
+      # so it may move before KIND_VERSION in agent/Makefile.
+      # renovate: datasource=docker depName=registry.k8s.io/kubectl
+      KUBECTL_VERSION: v1.37.1@sha256:b7cab618e281b1ee7484e7b706a96e2135fbb6e072c2a573a7dab4e87d7f2385
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - name: Set up Go
+        uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+        with:
+          go-version-file: agent/go.mod
+          cache-dependency-path: agent/go.sum
+      - name: Install kubectl
+        run: |
+          id=$(docker create --platform linux/amd64 "registry.k8s.io/kubectl:${KUBECTL_VERSION}")
+          sudo docker cp "$id":/bin/kubectl /usr/local/bin/kubectl
+          docker rm "$id"
+          kubectl version --client
+      - name: Run the e2e tests
+        run: make -C agent test-e2e

CONTRIBUTING.md

@@ -29,10 +29,10 @@ in [agent/README.md](agent/README.md#deploying).
 
 ## Development environment
 
-The agent needs Go 1.26+. controller-gen, kustomize, envtest and golangci-lint
-are downloaded into `agent/bin/` at pinned versions by the `Makefile`, so those
-need no manual install. `kubectl` and `kind` do: the deploy and e2e targets
-expect them on your `PATH`.
+The agent needs Go 1.26+. controller-gen, kustomize, envtest, golangci-lint and
+kind are downloaded into `agent/bin/` at pinned versions by the `Makefile`, so
+those need no manual install. `kubectl` does: the deploy and e2e targets expect
+it on your `PATH`. CI pins it in `.github/workflows/agent-ci.yaml`.
 
 One version escapes the pinning. The `logcheck` plugin of the custom
 golangci-lint build resolves to `latest` in `agent/.custom-gcl.yml`, so a new
@@ -74,7 +74,7 @@ make -C rpm rpm EL=9        # build both RPMs into rpm/_build/
   per-resource failures with `utilerrors.NewAggregate`, so one bad resource
   does not stall the others.
 - **Tests**: Go's standard `testing` for the packages, Ginkgo + Gomega with
-  envtest for the reconciler, and a kind-based smoke test under `test/e2e`.
+  envtest for the reconciler, and a kind-based e2e suite under `test/e2e`.
   A behaviour that can fail on a node deserves a test that reproduces it, not
   only a unit test of the helper it calls: a partial extraction, a missing
   mount, a manual deletion.

agent/AGENTS.md

@@ -30,7 +30,7 @@ internal/controller/ the node reconciler, the filesystem watcher, the labels
 internal/cache/      the on-disk store: state, extraction, garbage collection
 internal/puller/     pulling an image and streaming its layers
 config/              CRD, RBAC and DaemonSet manifests (kustomize)
-test/e2e/            kind-based smoke test, builds and deploys for real
+test/e2e/            kind-based e2e suite, builds, deploys and pulls for real
 ```
 
 ## Generated files
@@ -57,12 +57,17 @@ and several kustomizations.
 make fmt vet     # both run as prerequisites of test, listed here for clarity
 make test        # unit tests plus the envtest suite
 make lint        # custom golangci-lint build, logcheck included
-make test-e2e    # needs kind and an idle cluster name, see below
+make test-e2e    # needs kubectl and an idle cluster name, see below
 ```
 
-`make test-e2e` creates the cluster `image-cache-agent-test-e2e`, builds the
-image, loads it and deploys the manifests. It needs `kind` and `kubectl` on
-your `PATH`; neither is downloaded by the `Makefile`.
+`make test-e2e` creates the two-node cluster `image-cache-agent-test-e2e`
+(`test/e2e/kind-config.yaml`), builds the image, loads it and deploys the
+manifests. It deploys two registries in the cluster, one plain HTTP and one
+signed by a throwaway CA, so a run pulls `registry:3.1.2` from Docker Hub
+(`registryVersion` in `test/e2e/e2e_test.go`, pinned by digest). It also
+builds `imagecachectl` and runs it on the worker. The `Makefile` downloads
+`kind` into `bin/`, but `kubectl` has to be on your `PATH`. CI runs it on
+every pull request, in the `e2e` job of `.github/workflows/agent-ci.yaml`.
 
 A green `make lint` right after a dependency bump is not proof. Run
 `bin/golangci-lint cache clean` first, for the reason

agent/DESIGN.md

@@ -382,7 +382,12 @@ The agent never writes ImageCache resources (no status, no finalizers).
 - envtest: the full reconciler with a fake puller, from resource lifecycle to
   node labels and on-disk state: failure paths, adoption of a seeded
   directory, and entries the agent did not write.
-- A minimal kind-based e2e smoke test, with no registry infrastructure: CRD
-  installed, agent running, node labelled `pending` for a resource with an
-  unreachable source, label cleared on deletion, and garbage collection of
-  root-owned directories, which needs `DAC_OVERRIDE`.
+- A kind-based e2e suite on a control plane and a worker: CRD installed,
+  agent running on both nodes, garbage collection of root-owned directories,
+  which needs `DAC_OVERRIDE`, and node labels cleared on deletion. A source
+  nobody listens on leaves the node `pending`. Two registries run in the
+  cluster: a pull over plain HTTP lands on the selected node only, and a
+  registry signed by a private CA is refused until the shipped CA patch is
+  applied. `imagecachectl` seeds two directories on the worker: the agent
+  adopts the one holding the image its resource names, with no extraction,
+  and replaces the other.

agent/Makefile

@@ -63,40 +63,45 @@ vet: ## Run go vet against code.
 test: manifests generate fmt vet setup-envtest ## Run tests.
 	KUBEBUILDER_ASSETS="$(shell "$(ENVTEST)" use $(ENVTEST_K8S_VERSION) --bin-dir "$(LOCALBIN)" -p path)" go test $$(go list ./... | grep -v /e2e) -coverprofile cover.out
 
-# TODO(user): To use a different vendor for e2e tests, modify the setup under 'tests/e2e'.
-# The default setup assumes Kind is pre-installed and builds/loads the Manager Docker image locally.
-# kubectl kuberc is disabled by default for test isolation; enable with:
+# The e2e suite runs on a kind cluster. It builds the manager image and loads it
+# into kind, so that image is never pushed. kubectl kuberc is disabled by default
+# for test isolation; enable with:
 # - KUBECTL_KUBERC=true
-# CertManager is installed by default; skip with:
-# - CERT_MANAGER_INSTALL_SKIP=true
 KIND_CLUSTER ?= image-cache-agent-test-e2e
+# The node count of test/e2e/kind-config.yaml. A cluster left over with another
+# count would make the node assertions fail for a reason unrelated to the agent.
+KIND_NODES = 2
 
 .PHONY: setup-test-e2e
-setup-test-e2e: ## Set up a Kind cluster for e2e tests if it does not exist
-	@command -v $(KIND) >/dev/null 2>&1 || { \
-		echo "Kind is not installed. Please install Kind manually."; \
-		exit 1; \
-	}
-	@case "$$($(KIND) get clusters)" in \
-		*"$(KIND_CLUSTER)"*) \
-			echo "Kind cluster '$(KIND_CLUSTER)' already exists. Skipping creation." ;; \
-		*) \
-			echo "Creating Kind cluster '$(KIND_CLUSTER)'..."; \
-			$(KIND) create cluster --name $(KIND_CLUSTER) ;; \
-	esac
+setup-test-e2e: kind ## Set up a Kind cluster for e2e tests if it does not exist
+	@if $(KIND) get clusters | grep -qx '$(KIND_CLUSTER)'; then \
+		nodes=$$($(KIND) get nodes --name $(KIND_CLUSTER) | wc -l); \
+		if [ "$$nodes" -ne $(KIND_NODES) ]; then \
+			echo "Kind cluster '$(KIND_CLUSTER)' has $$nodes nodes, the e2e suite needs $(KIND_NODES)." \
+				"Delete it with 'make cleanup-test-e2e'."; \
+			exit 1; \
+		fi; \
+		echo "Kind cluster '$(KIND_CLUSTER)' already exists. Skipping creation."; \
+	else \
+		echo "Creating Kind cluster '$(KIND_CLUSTER)'..."; \
+		$(KIND) create cluster --name $(KIND_CLUSTER) --config test/e2e/kind-config.yaml; \
+	fi
 
 .PHONY: test-e2e
 test-e2e: setup-test-e2e manifests generate fmt vet ## Run the e2e tests. Expected an isolated environment using Kind.
 # Tear the cluster down even when the tests fail, keeping their exit code:
 # setup-test-e2e reuses an existing cluster, so a leftover one would silently
-# run the next attempt against dirty state.
-	KIND=$(KIND) KIND_CLUSTER=$(KIND_CLUSTER) go test -tags=e2e ./test/e2e/ -v -ginkgo.v; \
-	  status=$$?; \
+# run the next attempt against dirty state. The `|| status=` keeps the shell
+# alive past a failure: .SHELLFLAGS has -e. The 25m timeout replaces go
+# test's 10m default, which a failing suite can reach before Ginkgo prints
+# which spec failed, and stays under the 30m of the CI job.
+	status=0; \
+	  KIND=$(KIND) KIND_CLUSTER=$(KIND_CLUSTER) go test -tags=e2e ./test/e2e/ -timeout 25m -v -ginkgo.v || status=$$?; \
 	  $(MAKE) cleanup-test-e2e; \
 	  exit $$status
 
 .PHONY: cleanup-test-e2e
-cleanup-test-e2e: ## Tear down the Kind cluster used for e2e tests
+cleanup-test-e2e: kind ## Tear down the Kind cluster used for e2e tests
 	@$(KIND) delete cluster --name $(KIND_CLUSTER)
 
 .PHONY: lint
@@ -194,7 +199,7 @@ $(LOCALBIN):
 
 ## Tool Binaries
 KUBECTL ?= kubectl
-KIND ?= kind
+KIND ?= $(LOCALBIN)/kind
 KUSTOMIZE ?= $(LOCALBIN)/kustomize
 CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
 ENVTEST ?= $(LOCALBIN)/setup-envtest
@@ -215,6 +220,10 @@ ENVTEST_K8S_VERSION ?= $(shell v='$(call gomodver,k8s.io/api)'; \
   printf '%s\n' "$$v" | sed -E 's/^v?[0-9]+\.([0-9]+).*/1.\1/')
 
 GOLANGCI_LINT_VERSION ?= v2.12.2
+# The kind release also pins the Kubernetes version of the e2e cluster, through
+# its default node image.
+# renovate: datasource=github-releases depName=kubernetes-sigs/kind
+KIND_VERSION ?= v0.33.0
 .PHONY: kustomize
 kustomize: $(KUSTOMIZE) ## Download kustomize locally if necessary.
 $(KUSTOMIZE): $(LOCALBIN)
@@ -238,6 +247,11 @@ envtest: $(ENVTEST) ## Download setup-envtest locally if necessary.
 $(ENVTEST): $(LOCALBIN)
 	$(call go-install-tool,$(ENVTEST),sigs.k8s.io/controller-runtime/tools/setup-envtest,$(ENVTEST_VERSION))
 
+.PHONY: kind
+kind: $(KIND) ## Download kind locally if necessary.
+$(KIND): $(LOCALBIN)
+	$(call go-install-tool,$(KIND),sigs.k8s.io/kind,$(KIND_VERSION))
+
 .PHONY: golangci-lint
 golangci-lint: $(GOLANGCI_LINT) ## Download golangci-lint locally if necessary.
 $(GOLANGCI_LINT): $(LOCALBIN)

agent/README.md

@@ -17,11 +17,10 @@ All the commands below run from this directory.
 
 ## Prerequisites
 
-- Go 1.26+ for building and testing. controller-gen, kustomize, envtest and
-  golangci-lint are downloaded into `bin/` at pinned versions by the
+- Go 1.26+ for building and testing. controller-gen, kustomize, envtest,
+  golangci-lint and kind are downloaded into `bin/` at pinned versions by the
   `Makefile`.
-- `kubectl`, and `kind` for the e2e suite. Neither is downloaded: the targets
-  expect them on your `PATH`.
+- `kubectl`. It is not downloaded: the targets expect it on your `PATH`.
 - Docker, or another `CONTAINER_TOOL`, for the image.
 - A cluster on Kubernetes 1.25+, which is where CEL validation of custom
   resources landed. The CRD relies on it.