Skip to content

ci: run the agent e2e suite on kind for each pull request (MK8S-450) - #33

Merged
ezekiel-alexrod merged 7 commits into
mainfrom
ci/MK8S-450-e2e-on-kind
Oct 20, 2026
Merged

ezekiel-alexrod merged 7 commits into
mainfrom
ci/MK8S-450-e2e-on-kind

Conversation

@ezekiel-alexrod

@ezekiel-alexrod ezekiel-alexrod commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Component

ci, agent, docs

Problem

The agent e2e suite only ran by hand, and it never pulled an image. agent-ci.yaml runs the lint, the unit tests and a docker-build, but nothing deploys the agent on a cluster. A broken manifest, pull, extraction or synced label shows up late, at best in MetalK8s.

Fix

Start with the third commit, the specs, then the last one, added after your approvals: the adoption spec. I'd mostly like eyes on what they cover and on the version pinning.

  • A new e2e job in agent-ci.yaml runs make -C agent test-e2e on every pull request and on pushes to main that touch agent/, with timeout-minutes: 30. No image is pushed: the suite loads it into kind.
  • The Makefile downloads kind at a pinned version (v0.33.0, node image Kubernetes 1.37.0). CI takes kubectl v1.37.1 from the registry.k8s.io/kubectl image, pinned by digest.
  • The kind cluster now has a control plane and a worker (test/e2e/kind-config.yaml). setup-test-e2e refuses a leftover cluster with another node count instead of reusing it.
  • Two specs pull a real image from a registry running in the cluster (registry:3.1.2, pinned by digest), pushed through kubectl port-forward:
    • plain HTTP with --plain-http: an ImageCache selecting the worker gets it synced, with the archive and the sentinel on disk. Nothing lands on the control plane, and deleting the ImageCache removes the directory and the label.
    • TLS signed by a throwaway CA: the agent first logs certificate signed by unknown authority and stays pending, then pulls once the registry-ca ConfigMap and the shipped config/default/manager_registry_ca_patch.yaml are applied.
  • A third spec runs the real imagecachectl on the worker, from two docker archives, before any ImageCache exists. The agent then adopts the directory holding the image its resource names (Adopted event, same inode, owner switched) and replaces the one holding another image. Before this, the seeded directory was a hand-written sentinel, so neither the command nor the adoption ran.
  • Renovate now tracks kind, kubectl and the registry image. The other tools of the Makefile are annotated in ci(agent): bump golangci-lint to v2.14.0, let Renovate track the tools (MK8S-470) #35.
  • test-e2e now deletes the cluster when the suite fails. It did not: .SHELLFLAGS has -e, so a failing go test ended the recipe before the cleanup, and the next run reused the cluster. It also gets -timeout 25m: the 10m default can kill a failing suite before Ginkgo prints which spec failed.

Test

Check Result
make -C agent test-e2e locally, two nodes, rebased on main 6/6 specs passed, 177 s
same, with --plain-http swapped for a no-op flag the plain HTTP spec fails waiting for synced, as it should
same, with adopt forced to replace the adoption spec fails on no Adopted event, as it should
Renovate 41 dry-run on the branch, shared preset kind, kubectl and registry found
kubectl extracted from the image vs the one from dl.k8s.io same sha256
make -C agent setup-test-e2e with a 1-node cluster left over refused, exit 1
make -C agent lint, and golangci-lint --build-tags e2e ./test/... 0 issues, only the // +build lines that were already there
actionlint .github/workflows/agent-ci.yaml OK
the e2e job on GitHub, first commit only passed in 4 min 27 s
the e2e job on GitHub, kubectl from the image 6/6 specs passed, 7 min 44 s

Out of scope

  • The image job stays: no check is required on main today, and it fails faster than the e2e when only the build breaks.
  • Making e2e a required check: to decide once we have its duration with the new specs.
  • CA rotation: the agent reads --ca-file once at startup, as documented.

  • The docs describing this behaviour are updated in the same pull request: README.md, agent/README.md, DESIGN.md, agent/DESIGN.md, CONTRIBUTING.md, whichever owns it.

Relates-to: MK8S-450

@ezekiel-alexrod
ezekiel-alexrod requested a review from a team as a code owner October 14, 2026 09:19
Comment thread agent/AGENTS.md Outdated
@ezekiel-alexrod
ezekiel-alexrod force-pushed the ci/MK8S-450-e2e-on-kind branch from 150238a to 5c53316 Compare October 14, 2026 10:21
Comment thread agent/test/e2e/e2e_test.go
Comment thread agent/Makefile Outdated
@ezekiel-alexrod
ezekiel-alexrod force-pushed the ci/MK8S-450-e2e-on-kind branch from 5c53316 to de20d05 Compare October 14, 2026 10:54
Comment thread .github/workflows/agent-ci.yaml Outdated
@ezekiel-alexrod
ezekiel-alexrod force-pushed the ci/MK8S-450-e2e-on-kind branch from de20d05 to d5b5fd1 Compare October 14, 2026 11:59
@ezekiel-alexrod
ezekiel-alexrod requested review from TeddyAndrieux, anthony-treuillier-scality and eg-ayoub and removed request for a team October 14, 2026 12:13
@ezekiel-alexrod ezekiel-alexrod self-assigned this Oct 15, 2026
@ezekiel-alexrod ezekiel-alexrod added the P2 Medium priority label Oct 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small comments

Comment thread agent/Makefile
Comment thread agent/test/e2e/e2e_test.go Outdated
The e2e suite only ran by hand, so a broken manifest, RBAC rule or
DaemonSet showed up late. A new e2e job in agent-ci runs it on every
pull request and on pushes to main that touch agent/.

The Makefile now downloads kind at a pinned version, like the other
tools, so local runs and CI use the same kind and the same node image.
CI pins kubectl to the minor version of that image. No image is pushed:
the suite loads it into kind.

Relates-to: MK8S-450
test-e2e meant to tear the cluster down whatever the outcome, but the
Makefile shell runs with -e, so a failing go test ended the recipe
before cleanup-test-e2e. The next run then reused that cluster without
a word, as the comment above the recipe warns.

Relates-to: MK8S-450
The e2e suite never pulled an image: it checked the DaemonSet, the
garbage collection at startup and the pending label with an unreachable
source. A broken pull, extraction or synced label would only show in
MetalK8s.

The cluster now has a control plane and a worker, and two registries
run in it. One serves plain HTTP: an ImageCache selecting the worker
gets synced there, with the archive and the sentinel on disk, nothing
lands on the control plane, and deletion collects it. The other is
signed by a throwaway CA: the agent first refuses it, then pulls once
the registry-ca ConfigMap and the shipped CA patch are applied.

Relates-to: MK8S-450
kind, kubectl and the registry image of the e2e only moved by hand.
They get the annotations our other kubebuilder repos use. The tools the
other jobs share, golangci-lint, kustomize and controller-tools, are
left to their own change.

kubectl now comes from the registry.k8s.io/kubectl image, pinned by tag
and digest. Renovate cannot follow the sha256 of a file on dl.k8s.io,
but it follows an image digest, and the binary is the same. The tag and
digest sit alone in their value, as the YAML and Go regexes of the
shared preset do not read a full reference.

Relates-to: MK8S-450
The cleanup listed the ImageCaches by name, so each new spec had to be
added there or left one behind. They are cluster-scoped and the kind
cluster is the suite's own, so --all removes only what it created.

Relates-to: MK8S-450
The suite faked the seeded directory with a hand-written sentinel and
never created a resource for it. So neither imagecachectl nor the
adoption ran: a broken diff ID check would only show in MetalK8s.

The spec builds imagecachectl and imports two docker archives on the
worker. The agent adopts the one holding the image its resource names,
with the archive untouched, and replaces the one holding another image.

Relates-to: MK8S-450
ImageCache is cluster-scoped, so its SyncFailed and AdoptFailed events
land in the default namespace, which the dump did not read. Those are
the ones that say why a sync spec failed.

Relates-to: MK8S-450
@ezekiel-alexrod
ezekiel-alexrod force-pushed the ci/MK8S-450-e2e-on-kind branch from d5b5fd1 to e646d27 Compare October 19, 2026 14:42
@ezekiel-alexrod
ezekiel-alexrod requested a review from a team as a code owner October 19, 2026 14:42
@ezekiel-alexrod
ezekiel-alexrod requested review from g-carre and removed request for g-carre October 19, 2026 14:42
Comment thread agent/test/e2e/e2e_test.go
@ezekiel-alexrod
ezekiel-alexrod merged commit 9333595 into main Oct 20, 2026
10 of 11 checks passed
@ezekiel-alexrod
ezekiel-alexrod deleted the ci/MK8S-450-e2e-on-kind branch October 20, 2026 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Medium priority

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants