From 9d366ba3f5ecf19936c555b86de8917ca9a16be9 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Wed, 4 Nov 2026 22:52:44 +0100 Subject: [PATCH] docs(security): the Iceberg sink catalogAuth checks and the Debezium Server's dotenv names Three forge-cli fixes on main, not yet in a release: - forge-cli #828: the sink's catalogAuth host check refuses the cloud metadata addresses, as the other internal-endpoint checks do; - forge-cli #830: with catalogAuth, text the sink expands later (${, a leading env:, a control character) is refused in a catalog URL and in the override values Iceberg reads; - forge-cli #833: a project dotenv file cannot set the embedded Debezium Server's configuration, its JVM options or its launcher's settings. --- docs/advanced/network-safety.md | 2 ++ docs/advanced/project-dotenv-trust.md | 2 ++ docs/advanced/source-aligned-acquisition.md | 12 +++++++----- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/advanced/network-safety.md b/docs/advanced/network-safety.md index ca55e8a..faefe9e 100644 --- a/docs/advanced/network-safety.md +++ b/docs/advanced/network-safety.md @@ -60,6 +60,8 @@ The webhook, federation and Command Center variables take comma-separated host s An IPv4-mapped address (`::ffff:...`) or a NAT64 address (`64:ff9b::/96`) that carries a link-local or one of these metadata addresses is refused too. The surrounding ranges stay allowed: a Tailscale tailnet (CGNAT, `100.100.100.100` included) and an IPv6 cluster on ULA addresses need no change. +*On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* A fourth check, the `catalogAuth` host check of the Kafka Connect and embedded Debezium Iceberg sink, refuses the same addresses, even when the host is listed. See [Operator allowlists](source-aligned-acquisition.md#operator-allowlists). + ## Cloud metadata and the credential resolver `FLUID_ALLOW_METADATA_SERVICE=1` lets the metadata-source [credential resolver](./credential-resolver.md) fall back to cloud workload identity. It is off by default, and it is not an SSRF switch: it does not relax the checks above. diff --git a/docs/advanced/project-dotenv-trust.md b/docs/advanced/project-dotenv-trust.md index 3619f86..de1d4ab 100644 --- a/docs/advanced/project-dotenv-trust.md +++ b/docs/advanced/project-dotenv-trust.md @@ -75,6 +75,8 @@ What it may not set is any variable that project file has nothing to supply there. The full list is `fluid_build/credentials/protected_env.py` in forge-cli. +*On forge-cli `main`, not yet in a release ([forge-cli #833](https://github.com/Agenticstiger/forge-cli/pull/833)).* The embedded Debezium Server inherits the environment `fluid apply` loads, and its SmallRye Config reads a variable over `application.properties`. Its JVM reads `-D` system properties, which outrank both, from `JDK_JAVA_OPTIONS` or the launcher's `JAVA_OPTS`. So a project file could send a `sink.catalogAuth` credential to a catalog the operator never paired with it, change the source database, load another config file, or start another `java`. A project dotenv file can no longer set `DEBEZIUM_*`, `QUARKUS_*`, `SMALLRYE_*` or `MP_CONFIG_*` variables in the forms SmallRye Config matches, profile forms included (`debezium.sink.iceberg.uri`, `_PROD_DEBEZIUM_...`, `%prod....`), nor `JDK_JAVA_OPTIONS`, `JAVA_OPTS`, `JAVA_HOME`, or the launcher's `EXTRA_CONNECTOR`, `JMX_HOST`, `JMX_PORT`, `JMX_EXPORTER_PORT`, `JMX_EXPORTER_CONFIG` and `OTEL_JAVAAGENT_*`. Each is ignored with the usual warning. Set it in the shell or CI instead. + ::: tip Two more since `0.23.0` `FLUID_ACQUISITION_SECRET_REF_ALLOWLIST`, which allows an acquisition source's `env://` reference against a remote host, and `TESTCONTAINERS_HOST_OVERRIDE`, which rewrites a diff --git a/docs/advanced/source-aligned-acquisition.md b/docs/advanced/source-aligned-acquisition.md index 8ae0e9c..733c5e7 100644 --- a/docs/advanced/source-aligned-acquisition.md +++ b/docs/advanced/source-aligned-acquisition.md @@ -524,6 +524,8 @@ The same contract declaring `fluidVersion: "0.7.5"` fails, because the 0.7.5 sch - **`oauth2ServerUri`** starts with a lowercase `http://` or `https://` and has no `user:password@` part. *(since 0.27.0, [forge-cli #818](https://github.com/Agenticstiger/forge-cli/pull/818))* `fluid validate` and the run then refuse `http://`; see [Operator allowlists](#operator-allowlists). Iceberg treats any other spelling as a path on the catalog host and sends the credentials there. The query string is not checked, so keep secrets out of it too. - **`oauth2ServerUri`, `scope`, `audience` and `resource`** are written as they are. None may hold a `${...}` expression, which Debezium Server would expand in `application.properties` while Kafka Connect would pass it to Iceberg as text, a control character such as a line break or a tab, or a value starting with `env:`, which Iceberg would resolve from the environment. Only the secret comes from the environment. +- **Override text.** *On forge-cli `main`, not yet in a release ([forge-cli #830](https://github.com/Agenticstiger/forge-cli/pull/830)).* With `catalogAuth`, a catalog URL that holds `${`, starts with `env:` or holds a control character is refused wherever it is set, in `binding.location.uri` or in an override map, at `fluid validate` and in the run's preflight. The check reads the text before the sink expands it: Debezium Server expands a `${...}` expression when it starts, Kafka Connect resolves a `${provider:path:key}` reference with the worker's config providers, and Iceberg replaces a value that starts with `env:` by that variable. The same text is refused in every other override value Iceberg reads: Kafka Connect `iceberg_catalog_overrides`, the `iceberg.*` keys of `sink_connector_config` except `iceberg.kafka.*`, and every key of an embedded Debezium `server.sink.config`. A storage credential comes from the worker's or the server's own credential chain, not from the contract. Kafka-side keys such as `consumer.override.*` and `iceberg.kafka.*` keep their config providers. A build without `catalogAuth` is unchanged. + A message about a `catalogAuth` field that fails a check never shows the field's value, because it may be a secret pasted in place of a name. A variable name appears in a message only after it has passed the name checks. #### What the derived config holds @@ -579,7 +581,7 @@ For each host the credential reaches, the run needs: - **The pairing**, `NAME@host` in `FLUID_ACQUISITION_SECRET_REF_ALLOWLIST`, unless the host is loopback. - **A host entry for a hostname**, in `FLUID_ACQUISITION_DEPLOYMENT_HOST_ALLOWLIST`, because the worker (or Debezium Server) resolves the name after forge-cli's check, so no lookup made by forge-cli shows where it will connect. `localhost` needs none. -An IP address keeps the deployment host list's rules instead of needing an entry as a name does: a link-local address is never allowed, a private one needs no host entry (only the pairing), and a public one only when listed. A loopback host needs neither entry. +An IP address keeps the deployment host list's rules instead of needing an entry as a name does: a link-local or cloud metadata address is never allowed, a private one needs no host entry (only the pairing), and a public one only when listed. A loopback host needs neither entry. *On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* An OAuth2 credential reaches two hosts, so it needs a pairing for the token endpoint and one for the catalog (one entry covers both when they are the same host, as with the Polaris recipe below). The example contract needs: @@ -597,19 +599,19 @@ Two cautions on how entries match: So prefer `NAME@host`, and list the exact host. -The run refuses before it makes any Connect REST call, writes `application.properties` or reads the variable. `fluid validate` may run where the operator's lists are not set, so for a host that only lacks an entry it warns, with the `export` line the run needs (the warning above), and does not fail. It refuses outright, with no `export` line, an `http://` URL and a link-local or unspecified address, which no entry allows. With `oauth2ServerUri: http://keycloak:8080/...`: +The run refuses before it makes any Connect REST call, writes `application.properties` or reads the variable. `fluid validate` may run where the operator's lists are not set, so for a host that only lacks an entry it warns, with the `export` line the run needs (the warning above), and does not fail. It refuses outright, with no `export` line, an `http://` URL and a link-local, unspecified or cloud metadata address, which no entry allows. *On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* With `oauth2ServerUri: http://keycloak:8080/...`: ```text 1. iceberg sink (build 'stream_orders'): sink.catalogAuth.oauth2ServerUri must be an https:// URL: the OAuth2 token endpoint receives the credential in LAKEKEEPER_CREDENTIAL (the client id and secret) on every request, and http would carry it in clear text, so a loopback host needs https too ``` -With `binding.location.uri: https://169.254.169.254`, a cloud metadata address: +With `binding.location.uri: https://169.254.169.254`, a cloud metadata address, the message reads as follows. *On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* ```text - 1. iceberg sink (build 'stream_orders'): the Iceberg catalog host '169.254.169.254' (binding.location.uri) is link-local or unspecified, where cloud instance metadata lives; this is refused even with an allowlist entry + 1. iceberg sink (build 'stream_orders'): the Iceberg catalog host '169.254.169.254' (binding.location.uri) is link-local, unspecified or a known cloud metadata endpoint; no Iceberg catalog is ever there, and this is refused even with an allowlist entry ``` -Both lists are operator-only: a value set in a project `.env`, `.env.` or `FLUID_SECRETS_FILE` is ignored (`credentials/protected_env.py` lists both). Set them in the shell or the CI job that runs `fluid`. [Network safety](network-safety.md) lists the further metadata addresses that the orchestrator guard, the OpenLineage emitter and the managed Flink host check refuse; this sink check refuses link-local and unspecified addresses only. +Both lists are operator-only: a value set in a project `.env`, `.env.` or `FLUID_SECRETS_FILE` is ignored (`credentials/protected_env.py` lists both). Set them in the shell or the CI job that runs `fluid`. [Network safety](network-safety.md) lists the further metadata addresses that the orchestrator guard, the OpenLineage emitter and the managed Flink host check refuse; this sink check refuses those too. *On forge-cli `main`, not yet in a release ([forge-cli #828](https://github.com/Agenticstiger/forge-cli/pull/828)).* It refuses the same addresses as the orchestrator guard, the OpenLineage emitter and the managed Flink host check, even when listed. Before that change it refused link-local and unspecified addresses only. Related text checks, which apply whether or not the sink has a `catalogAuth`: `binding.location` text that reaches the derived sink config may not start with `env:`, hold `${`, or hold a control character, and `binding.location.region` must be one host label (letters, digits and inner hyphens, at most 63). Only `sink.catalogAuth` may name a variable. A value is never quoted in the message. With `warehouse: "env:AWS_SECRET_ACCESS_KEY"`: