Repository navigation
ci: run the agent e2e suite on kind for each pull request (MK8S-450) - #33
Merged
Merged
Conversation
ezekiel-alexrod
force-pushed
the
ci/MK8S-450-e2e-on-kind
branch
from
October 14, 2026 10:21
150238a to
5c53316
Compare
ezekiel-alexrod
force-pushed
the
ci/MK8S-450-e2e-on-kind
branch
from
October 14, 2026 10:54
5c53316 to
de20d05
Compare
ezekiel-alexrod
force-pushed
the
ci/MK8S-450-e2e-on-kind
branch
from
October 14, 2026 11:59
de20d05 to
d5b5fd1
Compare
ezekiel-alexrod
requested review from
TeddyAndrieux,
anthony-treuillier-scality and
eg-ayoub
and removed request for
a team
October 14, 2026 12:13
eg-ayoub
approved these changes
Oct 15, 2026
anthony-treuillier-scality
approved these changes
Oct 16, 2026
The e2e suite only ran by hand, so a broken manifest, RBAC rule or DaemonSet showed up late. A new e2e job in agent-ci runs it on every pull request and on pushes to main that touch agent/. The Makefile now downloads kind at a pinned version, like the other tools, so local runs and CI use the same kind and the same node image. CI pins kubectl to the minor version of that image. No image is pushed: the suite loads it into kind. Relates-to: MK8S-450
test-e2e meant to tear the cluster down whatever the outcome, but the Makefile shell runs with -e, so a failing go test ended the recipe before cleanup-test-e2e. The next run then reused that cluster without a word, as the comment above the recipe warns. Relates-to: MK8S-450
The e2e suite never pulled an image: it checked the DaemonSet, the garbage collection at startup and the pending label with an unreachable source. A broken pull, extraction or synced label would only show in MetalK8s. The cluster now has a control plane and a worker, and two registries run in it. One serves plain HTTP: an ImageCache selecting the worker gets synced there, with the archive and the sentinel on disk, nothing lands on the control plane, and deletion collects it. The other is signed by a throwaway CA: the agent first refuses it, then pulls once the registry-ca ConfigMap and the shipped CA patch are applied. Relates-to: MK8S-450
kind, kubectl and the registry image of the e2e only moved by hand. They get the annotations our other kubebuilder repos use. The tools the other jobs share, golangci-lint, kustomize and controller-tools, are left to their own change. kubectl now comes from the registry.k8s.io/kubectl image, pinned by tag and digest. Renovate cannot follow the sha256 of a file on dl.k8s.io, but it follows an image digest, and the binary is the same. The tag and digest sit alone in their value, as the YAML and Go regexes of the shared preset do not read a full reference. Relates-to: MK8S-450
The cleanup listed the ImageCaches by name, so each new spec had to be added there or left one behind. They are cluster-scoped and the kind cluster is the suite's own, so --all removes only what it created. Relates-to: MK8S-450
The suite faked the seeded directory with a hand-written sentinel and never created a resource for it. So neither imagecachectl nor the adoption ran: a broken diff ID check would only show in MetalK8s. The spec builds imagecachectl and imports two docker archives on the worker. The agent adopts the one holding the image its resource names, with the archive untouched, and replaces the one holding another image. Relates-to: MK8S-450
ImageCache is cluster-scoped, so its SyncFailed and AdoptFailed events land in the default namespace, which the dump did not read. Those are the ones that say why a sync spec failed. Relates-to: MK8S-450
ezekiel-alexrod
force-pushed
the
ci/MK8S-450-e2e-on-kind
branch
from
October 19, 2026 14:42
d5b5fd1 to
e646d27
Compare
ezekiel-alexrod
requested review from
g-carre
and removed request for
g-carre
October 19, 2026 14:42
anthony-treuillier-scality
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Component
ci, agent, docs
Problem
The agent e2e suite only ran by hand, and it never pulled an image.
agent-ci.yamlruns the lint, the unit tests and adocker-build, but nothing deploys the agent on a cluster. A broken manifest, pull, extraction or synced label shows up late, at best in MetalK8s.Fix
Start with the third commit, the specs, then the last one, added after your approvals: the adoption spec. I'd mostly like eyes on what they cover and on the version pinning.
e2ejob inagent-ci.yamlrunsmake -C agent test-e2eon every pull request and on pushes tomainthat touchagent/, withtimeout-minutes: 30. No image is pushed: the suite loads it into kind.Makefiledownloads kind at a pinned version (v0.33.0, node image Kubernetes 1.37.0). CI takes kubectlv1.37.1from theregistry.k8s.io/kubectlimage, pinned by digest.test/e2e/kind-config.yaml).setup-test-e2erefuses a leftover cluster with another node count instead of reusing it.registry:3.1.2, pinned by digest), pushed throughkubectl port-forward:--plain-http: an ImageCache selecting the worker gets itsynced, with the archive and the sentinel on disk. Nothing lands on the control plane, and deleting the ImageCache removes the directory and the label.certificate signed by unknown authorityand stayspending, then pulls once theregistry-caConfigMap and the shippedconfig/default/manager_registry_ca_patch.yamlare applied.imagecachectlon the worker, from two docker archives, before any ImageCache exists. The agent then adopts the directory holding the image its resource names (Adoptedevent, same inode, owner switched) and replaces the one holding another image. Before this, the seeded directory was a hand-written sentinel, so neither the command nor the adoption ran.Makefileare annotated in ci(agent): bump golangci-lint to v2.14.0, let Renovate track the tools (MK8S-470) #35.test-e2enow deletes the cluster when the suite fails. It did not:.SHELLFLAGShas-e, so a failinggo testended the recipe before the cleanup, and the next run reused the cluster. It also gets-timeout 25m: the 10m default can kill a failing suite before Ginkgo prints which spec failed.Test
make -C agent test-e2elocally, two nodes, rebased onmain--plain-httpswapped for a no-op flagsynced, as it shouldadoptforced to replaceno Adopted event, as it shouldmake -C agent setup-test-e2ewith a 1-node cluster left overmake -C agent lint, andgolangci-lint --build-tags e2e ./test/...// +buildlines that were already thereactionlint .github/workflows/agent-ci.yamle2ejob on GitHub, first commit onlye2ejob on GitHub, kubectl from the imageOut of scope
imagejob stays: no check is required onmaintoday, and it fails faster than the e2e when only the build breaks.e2ea required check: to decide once we have its duration with the new specs.--ca-fileonce at startup, as documented.README.md,agent/README.md,DESIGN.md,agent/DESIGN.md,CONTRIBUTING.md, whichever owns it.Relates-to: MK8S-450