Repository navigation
Support keyless (workload identity) authentication for the AWS, Azure and Vault secrets backends #3488
Description
The credentials backends currently require a long-lived static credential for everything but GCP:
- AWS Secrets Manager takes an access key and secret only.
- Azure Key Vault takes a service principal client secret only.
- Vault takes a static token only (config.proto noted that application role auth was wanted instead).
Operators on EKS, AKS, or with Vault/OpenBao in-cluster usually have a keyless option available: EKS Pod Identity or
IRSA, AKS Workload Identity, and Vault's Kubernetes auth method. Supporting those means no stored secret has to be
rotated for the control plane or CAS. GCP already works this way, since its credentials file can be a workload
identity federation configuration.
Each backend is proposed as a separate PR, so they can be reviewed and merged independently: #3486 (AWS), #3485
(Azure), #3487 (Vault). Every change is backwards compatible: existing static-credential configurations behave exactly
as before.
Activity
migmartri commentedon Sep 30, 2026
Thank you, we will review it.
Out of curiosity, have you run chainloop in that kind of environment, in other words, did you perform end to end tests with any specific env?
khrisrichardson commentedon Sep 30, 2026
Tested locally.
Vault:
Kubernetes auth against a kind cluster with Vault 1.12.3: login, lease renewal and re-login past max_ttl all worked, and unbound service accounts, unknown roles and unsigned tokens were refused.
Note that predating this PR, a bad token to Vault shows up as "context deadline exceeded" instead of the real reason. Vault 1.12.3 returns HTTP 500 for an unsigned token. Chainloop's Vault client retries 500s, and its existing 1-second timeout cuts the retries off before the error comes back. It still fails closed.
AWS:
The default credential chain against an AWS emulator (floci): web identity → AssumeRoleWithWebIdentity → Secrets Manager signed with the session credentials; with no identity it fails. That covers the wiring, not IAM trust, since the emulator doesn't validate tokens.
Azure:
WorkloadIdentityCredential against an Azure emulator (floci-az): federated token → Entra token exchange → Key Vault save/read/delete; with no token file it fails (Microsoft's global instance discovery was stubbed, since the emulator doesn't serve it). The emulators don't validate tokens, so this covers the wiring, not IAM/Entra trust.
khrisrichardson commentedon Sep 30, 2026
A follow-up on why only part of this local testing translates into the repo's own harness.
pkg/credentials already runs container-backed suites (vault:1.12.3 via testcontainers-go, skipped only when SKIP_INTEGRATION=true), so I looked at extending that pattern to the keyless paths. It fits some and not others.
AWS (default credential chain): fits. An AWS emulator in a GenericContainer (floci or LocalStack) serves STS AssumeRoleWithWebIdentity and Secrets Manager, so a suite can point the SDK at it with an IRSA-style environment (AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE), do a save/read/delete round trip, and check that no identity fails. One detail: AWS_ENDPOINT_URL alone isn't enough, because config.LoadDefaultConfig builds the web-identity STS client before it applies the base endpoint. Setting AWS_ENDPOINT_URL_STS (and AWS_ENDPOINT_URL_SECRETS_MANAGER) routes both calls to the emulator.
Vault Kubernetes auth: fits, at a cost. Vault verifies the service-account token through the Kubernetes TokenReview API, so the suite needs a real API server as well as Vault. The faithful option is testcontainers-go's k3s module, a new dependency that adds roughly 30s of startup. The lighter option is an httptest TokenReview stub that the Vault container calls back through the host, which exercises less of the real path. We'd go with whichever you prefer.
Azure (Workload Identity): doesn't fit well. I ran it end to end against an Azure emulator (floci-az: Entra token exchange, then Key Vault save/read/delete, plus a no-token-file negative case), but it took test-only scaffolding that wouldn't belong in the repo:
The emulator must run in TLS mode with its CA trusted, since azidentity rejects a plain-HTTP authority.
azsecrets verifies that the Key Vault challenge's resource (https://vault.azure.net) matches the vault URL's host, so a 127.0.0.1 vault URL is rejected. Keeping a real .vault.azure.net URL meant putting a CONNECT proxy in front of the emulator.
For a non-default authority host, azidentity first queries Microsoft's global instance-discovery endpoint (login.microsoftonline.com/common/discovery/instance), which no emulator serves, so that call had to be stubbed.
Both of the last two could be avoided with client options (DisableChallengeResourceVerification, DisableInstanceDiscovery), but NewManager doesn't expose them, and adding production knobs only to make a test pass seems like the wrong trade. So I'd keep Azure covered by the existing unit tests plus a documented manual procedure, unless you'd rather take a different approach.
In every case the emulators accept any token, so these tests prove the credential wiring (the SDK chose the federated path and used what came back), not that AWS, Entra or Vault would trust the identity in a real deployment.
khrisrichardson commentedon Oct 14, 2026
To help improve the developer experience for contributors, have the maintainers considered moving to a Helm chart dynamic version injection pattern in the CI pipeline?
Many popular cloud-native and CNCF projects (like Prometheus, Dapr, Mastodon, and Chaos Mesh) handle this by:
This decouples the codebase logic from the release metadata, meaning multiple contributors can work on different PRs simultaneously without ever triggering a version-bump conflict.
Just wanted to throw this out there as a potential quality-of-life upgrade for the repository's workflow whenever the team has the bandwidth to look into CI optimizations. Thanks for all your work managing these components!