Skip to content

fix(kubernetes): allow extra Keycloak env vars, disable the gzip resource encoder in e2e/dev - #7429

Merged
anna-parker merged 3 commits into
mainfrom
fix/keycloak-extra-env-theme-cache
Oct 19, 2026
Merged

anna-parker merged 3 commits into
mainfrom
fix/keycloak-extra-env-theme-cache

Conversation

@corneliusroemer-agent

@corneliusroemer-agent corneliusroemer-agent commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

The integration-test job flaked multiple times on a console 404 for a Keycloak theme asset (:8083/resources/<version>/login/loculus/…js) during login, causing the login form to never render. Between 09-23 and 09-27 it hit 5 job-attempts on 4 unrelated branches.

This is a race (bug) in Keycloak itself: keycloak/keycloak#53238 which I reported today.

KC_SPI_RESOURCE_ENCODING_GZIP_ENABLED=false is a narrow workaround for the bug. It only disables gzip encoding of theme files which should not have any impact on test correctness.

What changes

A new keycloakExtraEnv value: a list of Kubernetes EnvVar objects appended to the Keycloak container's env. values_e2e_and_dev.yaml uses it to set KC_SPI_RESOURCE_ENCODING_GZIP_ENABLED=false. With the default values the rendered manifests are byte-identical to main, so production keeps gzip encoding of theme resources.

https://claude.ai/code/session_01KGPjaZYeyyUnPSHeDUrvtX

🚀 Preview: Add preview label to enable

…n e2e/dev

Add `keycloakExtraEnv`, a list of Kubernetes EnvVar objects appended to
the Keycloak container's env. Use it in values_e2e_and_dev.yaml to set
KC_SPI_THEME_CACHE_THEMES=false.

With theme caching on, Keycloak gzips each theme asset into
data/tmp/kc-gzip-cache on its first request. Concurrent first requests
for the same asset race (unlocked exists() check, then a non-atomic
Files.move with REPLACE_EXISTING), and the loser gets a
FileNotFoundException that Keycloak turns into a 404. In CI this breaks
the parallel logins at the start of the cli job. With theme caching off
Keycloak serves assets directly and never writes the gzip cache.
Production values are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGPjaZYeyyUnPSHeDUrvtX
@claude claude Bot added deployment Code changes targetting the deployment infrastructure keycloak Related to keycloak and keycloakify labels Sep 27, 2026
@claude

claude Bot commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

This PR may be related to: #5426 (flaky CLI integration test due to Keycloak theme asset serving issue, opened the same day this fix landed).

…ache in e2e/dev

Turning off the gzip provider removes only the racing component; theme and
template caching stay on, so CI stays closer to production.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@corneliusroemer-agent corneliusroemer-agent changed the title fix(kubernetes): allow extra Keycloak env vars, disable theme cache in e2e/dev fix(kubernetes): allow extra Keycloak env vars, disable the gzip resource encoder in e2e/dev Sep 27, 2026
@corneliusroemer
corneliusroemer marked this pull request as ready for review September 27, 2026 21:43
@claude

claude Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @corneliusroemer's task in 23s —— View job


Review

The change is small and correct, and I have no code issues to flag inline. I read the template by eye; I couldn't run helm template here because it needs approval.

Template (keycloak-deployment.yaml:111-113)

  • with plus toYaml | nindent 12 puts the entries at the right indent in env:. When the list is empty or unset, nothing renders, so default installs don't change.
  • This follows the same pattern as backendExtraArgs.

Schema (values.schema.json:2206)

  • Setting additionalProperties: false with required: ["name"] is a good choice. A misspelled key such as valeu will fail at install instead of being silently dropped.
  • The schema also accepts an entry with both value and valueFrom, or with neither. Kubernetes rejects the first case at apply time and accepts the second, so neither needs a fix here.
  • If an extra entry uses a name the chart already sets, Kubernetes keeps the last one. That makes this an override hook as well, which seems fine.

Workaround (values_e2e_and_dev.yaml:12-15)

  • The fix is narrowly scoped, and the linked upstream issue is in a comment next to it, so it's easy to remove once Keycloak fixes the bug.
  • One thing to consider: the race is in Keycloak itself, so production could in principle hit the same 404 on a cold theme asset. It shows up in e2e because many logins happen at once right after startup. Keeping production unchanged is a reasonable call, but it may be worth saying so explicitly.

PR description (only needs a wording fix)

  • "so production keeps theme caching" is left over from the first commit. The final change disables gzip encoding, not the theme cache, so this should say something like "production keeps gzip encoding of theme resources".
  • Typo: "causing the login form to never renders" should be "to never render".

Tests and docs: there's nothing to add beyond the schema description. The e2e CI run is the real test of whether the flake goes away.

LGTM.

Comment thread kubernetes/loculus/values.schema.json Outdated

@tombch tombch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise looks good to me!

@maverbiest maverbiest left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Would be nice to get merged, I ran into this failure mode couple of times today when running CI :)

Comment thread kubernetes/loculus/values.schema.json Outdated
@anna-parker

Copy link
Copy Markdown
Contributor

@maverbiest I updated with @tombch's docs suggestion - as Cornelius is on vacay feel free to merge this if all tests pass :-)

@anna-parker
anna-parker merged commit bc84319 into main Oct 19, 2026
48 checks passed
@anna-parker
anna-parker deleted the fix/keycloak-extra-env-theme-cache branch October 19, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment Code changes targetting the deployment infrastructure e2e E2E test related keycloak Related to keycloak and keycloakify

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants