Skip to content

feat(bigquery): accept external_account (workload identity federation) credentials - #2288

Open
Azhar-ud wants to merge 1 commit into
libredb:mainfrom
Azhar-ud:feat/bigquery-external-account
Open

Azhar-ud wants to merge 1 commit into
libredb:mainfrom
Azhar-ud:feat/bigquery-external-account

Conversation

@Azhar-ud

@Azhar-ud Azhar-ud commented Nov 12, 2026 •

Copy link
Copy Markdown

Description

Adds external_account (Google Workload Identity Federation) credentials to the BigQuery provider, as proposed in #2278. An operator-seeded connection can carry a standard external_account JSON. Studio then gets a federated access token itself (subject token → Google STS) and renews it before it expires. This lets a deployment give each person their own BigQuery identity, with grants per table, without keys per person or a sidecar that rewrites short-lived access_tokens.

Type of Change

  • New feature (non-breaking change which adds functionality)
  • Documentation update
  • Test addition or update

Related Issue

Closes #2278

Changes Made

  • auth.ts: parses external_account. It fetches the subject token from credential_source.url (text or json format, optional headers) and exchanges it at https://sts.googleapis.com/v1/token (grant_type=token-exchange, BigQuery scope). Caching and refresh are shared with the other credential types: the token is renewed 60 s before expiry, and the subject token is fetched again each time.
  • transport.ts: a second single-socket lane, pinned to the subject token's origin, with TLS checked against that host. It uses the same egress guard (DB_HTTP_BLOCK_PRIVATE_HOSTS), follows no redirects, and is closed together with the rest of the transport.
  • index.ts: external_account is accepted only for operator-seeded (managed) connections. On a user-created connection the server would fetch whatever URL the user typed, so it is refused before anything is sent.
  • Kept narrow on purpose. The following are refused with clear messages, and their contents are never echoed:
    • service_account_impersonation_url
    • file, executable and environment sources
    • non-STS token_urls
    • subject token types other than jwt and id_token
    • plain http to anything other than this machine
  • docs/providers/bigquery.md: a new "External account" section with an example and the rules above. It also removes external_account from the list of refused types.
  • The agent credential is unchanged: it must still be a service_account key.

Testing

  • I have tested this locally
  • I have added/updated tests
  • All existing tests pass

Automated tests:

  • Unit: tests/unit/db/bigquery/external-account.test.ts, which is new, plus additions to the transport, provider and token-source tests.
  • Integration: tests/integration/db/bigquery-provider.test.ts:
    • a full connect through subject → STS → SELECT 1 with the federated bearer
    • a refusal on an unmanaged connection, before any request is sent
  • Every BigQuery test passes, and the changed source files have 100% line coverage.

Checks: format, lint, typecheck, knip, security:check and build all pass.

Full suite: green, except one unrelated editor test, "WITH list of 1,000,000 CTEs", which hit its 5 s limit once under full parallel load and passes on its own.

Live against Google (a real project, a real workload identity pool, and an OIDC provider with an uploaded JWKS):

  • A table-level roles/bigquery.dataViewer grant to principal://…/subject/test-person was given on one table only.
  • SELECT * FROM <dataset>.orders returned rows, and the bigger <dataset>.salaries table → Access Denied.
  • I used a 150 s subject token, so the Google token was capped at the same lifetime. Studio fetched a new subject token and exchanged it again before expiry, and queries kept working.
  • The same connection was refused when it wasn't operator-seeded.

Not covered live:

  • The live run served the subject token over plain http from the same machine, which is the loopback exception. The https path to a remote host is covered by the unit and integration tests only.
  • Token renewal was observed through the provider directly. In the UI it was not, though it uses the same code path.

Test Environment

  • LibreDB Studio Version: 1.3.0 (this branch)
  • Browser: Firefox
  • OS: Arch Linux
  • Node.js/Bun Version: Bun 1.4.4
  • Database Type: BigQuery

Screenshots

The granted table returns rows, and the table without a grant is refused by Google:

1-orders-granted 2-salaries-denied

Checklist

  • My code follows the project's code style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • The required CI test job passes the 100% line-coverage gate
  • If I changed src/lib/db/providers/, I updated the matching docs/providers/ documentation and tests/integration/db/ tests in the same PR (provider triad)
  • Agent leg: not affected. The agent credential still requires a service_account key.
  • Any dependent changes have been merged and published

Additional Notes

The sidebar's "Nothing to show" in the screenshots is expected. The test identity was granted one table and nothing at the dataset level, so it can't list datasets. That is the per-table setup this feature is meant for.

🤖 Generated with Claude Code

@gitguardian

gitguardian Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

…) credentials

An operator-seeded BigQuery connection can now carry an external_account
config. Studio fetches the subject token from credential_source.url,
exchanges it at Google STS for an access token, and repeats both before
the token expires, so per-person access no longer needs a key per person
or an outside job rewriting short-lived access_tokens.

- url sources only, over https unless the host is this machine; the
  fetch runs through the egress guard and never follows redirects
- STS at https://sts.googleapis.com/v1/token only; jwt and id_token
  subject types; no service-account impersonation
- refused on user-created connections, since the server would fetch a
  URL the user named

Closes libredb#2278

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Azhar-ud
Azhar-ud force-pushed the feat/bigquery-external-account branch from 45303a1 to d593ccc Compare November 12, 2026 17:57
@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Azhar-ud

Copy link
Copy Markdown
Author

@cevheri thanks for approving the workflow runs. All checks are green now and the branch has no conflicts.

Whenever you have time, I'd appreciate a review. The change is limited to the BigQuery provider: it accepts external_account credentials for operator-seeded connections only, and closes #2278. Happy to adjust the scope or split it if that makes it easier to review.

@cevheri

cevheri commented Nov 15, 2026

Copy link
Copy Markdown
Member

We want this feature but are not merging it in this form: BigQuery and Firestore carry duplicate Google credential code (D294 in docs/BACKLOG.md), and adding external_account to one copy widens that split. We are first moving Google auth into src/lib/db/auth/google in PRs we will link here; once they land, could you rebuild this change on top of it? The main points: external_account lives in the shared module and BigQuery opts in through its policy; it is accepted on a managed connection the operator's seed file or the governance control plane writes, never on one pasted in the browser or from a literal-mode seed file, with no server opt-in, so it works in governance mode as #2278 asks; subject and STS requests use module-built lanes that refuse link-local addresses and protocol upgrades; repeated or transport-owned header names and workforce pool audiences are refused; fields that change what is sent are refused, not ignored; an STS answer without expires_in is refused. We will post the full checklist once those PRs are merged. When the rebuild is ready, please test it against Google again, as you did here, and share the result. Thank you for the PR and the live test.

@cevheri

cevheri commented Nov 15, 2026

Copy link
Copy Markdown
Member

The shared Google auth is now on main (#2299, #2302, #2305, #2306), so this can be rebuilt on top of it. The checklist:

  1. Rebase on main and drop the unrelated re-wrapping in tests/integration/db/bigquery-provider.test.ts.
  2. The mechanism goes in src/lib/db/auth/google/external-account.ts plus one strategy in token-source.ts; BigQuery's only code change is adding external_account to BIGQUERY_KINDS.
  3. Build the sts and subject lanes through deps.createTransport with authLaneOptions.
  4. parseGoogleCredentialText gains a serverWritten boolean, fed by isServerWritten(connection) and checked right after the type is read; no server opt-in and no admitServerIdentity.
  5. Closed-world parse: refuse repeated or transport-owned header names (export isOwnedHeader, isHeaderValue and HEADER_NAME from node-transport.ts), workforce audiences, impersonation and other fields that change what is sent, and every source but url.
  6. Refuse an STS answer without expires_in, never cache the subject token, and pass STS reasons through serverText.
  7. Tests: zero requests for a browser record and for a literal-mode seed row, a governed-shape row signing in end to end, a 302 from the subject URL refused, Firestore untouched, coverage 100%.
  8. Docs: DB_AUTH.md, providers/bigquery.md, SEED_CONNECTIONS.md, GOVERNANCE.md and SECURITY.md row 3.40.

Then the live test against Google, please.

@cevheri cevheri added bigquery database-provider security Supply-chain, auth, or hardening work labels Nov 15, 2026
@Azhar-ud

Copy link
Copy Markdown
Author

Thanks for landing the shared Google auth and for the checklist. It is clear, and I agree with the direction: the mechanism in the shared module, BigQuery opting in through its policy, and external_account accepted only on a server-written connection.

I will rebuild this on top of main following the eight points, and run the live test against Google again when it is ready.

@cevheri cevheri added the loop:needs-info Maintainer-loop task blocked on human-reviewed clarification label Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bigquery database-provider loop:needs-info Maintainer-loop task blocked on human-reviewed clarification security Supply-chain, auth, or hardening work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] BigQuery: support external_account credentials (Workload Identity Federation)

2 participants