chore(streaming): re-sync #784 with main after #810, #816, #818 and #822

CHANGELOG.md

@@ -262,6 +262,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **The `fluid forge` welcome scan kept walking the whole directory tree after it had returned.**
+  Its type-suggestion probe listed contracts with `sorted(cwd.rglob("contract.fluid.yaml"))[:8]`,
+  which reads every file under the current directory (279 s under a folder holding many repos) and
+  goes into `.venv`, `.git` and `node_modules`. The scan returns at its 1.0 s budget, but Python
+  cannot kill a thread, so the walker went on at about half a core, and the mode picker, the
+  interview and the welcome panel each start a scan: up to three overlapped. The walk is now
+  bounded and ends with its scan. It skips the directories `util/upstream_discovery` already skips
+  (`.git`, `.venv`, `site-packages`, `node_modules`, `dist`, `build`, ...), lists at most 6 levels
+  below the current directory, reads at most 10,000 directory entries, and stops at its next
+  directory once the scan has returned; one walk runs at a time per process. It tallies the same
+  first 8 contracts as before wherever those limits do not cut the walk. A symlink named
+  `contract.fluid.yaml`, or one that is not a regular file (a FIFO, a device), is no longer
+  opened.
+
 - **`fluid apply --mode amend-and-build` provisioned a product, then refused its id.** The build
   phase keeps run records, cursors and watermarks under `.fluid/runs/<contract.id>/<build.id>/`,
   so its ids must start with a letter or underscore, use only letters, digits, `_`, `.` and `-`,
@@ -591,6 +605,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Security
 
+- **A contract chose both the variable and the hosts an Iceberg sink's catalog credential went
+  to.** On the Kafka Connect and embedded Debezium Iceberg sinks, `sink.catalogAuth` names an
+  environment variable (`tokenEnv` / `credentialEnv`), and the same contract names the catalog
+  (`binding.location.uri`) and the OAuth2 token endpoint (`oauth2ServerUri`). The config forge-cli
+  pushes or writes carries `env:<NAME>`, which Iceberg resolves on the Connect worker, or on
+  embedded Debezium from the runner's whole environment, so any variable there could be sent to a
+  host the contract picked. Every host the credential reaches must now be https (a loopback one
+  included) and allowed by the operator: the variable paired with it as `NAME@host` in
+  `FLUID_ACQUISITION_SECRET_REF_ALLOWLIST`, and a hostname listed in
+  `FLUID_ACQUISITION_DEPLOYMENT_HOST_ALLOWLIST`, because the worker resolves the name itself,
+  later. An IP address keeps that list's rules (link-local never, a private one with no entry, a
+  public one only when listed), and a loopback host needs neither entry. A bare `NAME` entry
+  allows every host and a listed hostname also covers its subdomains, so prefer `NAME@host` and
+  the exact host. The run refuses before any Connect REST call, `application.properties` write or
+  read of the variable; `fluid validate` warns with the `export` line the run needs, and refuses
+  an `http://` or link-local host outright. `binding.location` text that reaches the derived sink
+  config may no longer start with `env:` or hold `${` or a control character, and its `region`
+  must be one host label. **Upgrade:** a sink whose `catalogAuth` reaches a non-loopback host
+  fails its run until the operator adds both entries, and one on an `http://` catalog or token
+  endpoint fails `fluid validate`. A derived sink whose `binding.location` holds such text, or a
+  region that is not one label, now fails `fluid validate` and the run, `catalogAuth` or not;
+  write the value itself.
+
 - **`validate_ident` let a name with a trailing newline through.** Python's `$` also matches just
   before a trailing newline, so the identifier allowlist in `providers/_sql_safety.py`, which the
   DDL and OpenTofu emitters route names through, accepted `orders\n` and passed it on with the
@@ -621,6 +658,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   project dotenv file cannot point them elsewhere while `FLUID_LLM_API_KEY` or a stored key is in
   play, as for every other provider endpoint.
 
+- **The litellm endpoint check could not see a provider's per-kind route.** It looked for names
+  ending in `_API_BASE`, so the nine `DASHSCOPE_`, `QWENCLOUD_` and `QWEN_AI_PLATFORM_API_BASE_IMAGE`,
+  `_RERANK` and `_VIDEO` names litellm reads (three of them new in 1.108.0) were never listed, and
+  a project dotenv file could set them unpaired. They are listed now and the check reads a kind
+  after `_API_BASE`. FLUID calls only `litellm.completion`, which none of them routes, so nothing
+  reached them. The Databricks model-serving group also names `DATABRICKS_CLIENT_SECRET`, which
+  litellm's OAuth M2M path posts to the workspace `DATABRICKS_API_BASE` names; the name-shape rule
+  already refused that pairing, and the group now says why.
+
 ### Upgrade notes
 
 - New validate warning, so `--strict` fails: a contract with builds whose `contract.id` or a

CLAUDE.md

@@ -420,10 +420,11 @@ hand-rolling the protocol.
 
 `cli/_welcome_scan.py::run_welcome_scan` now has a budget that means what it says:
 
-- **One wall-clock cap.** `_TOTAL_TIMEOUT_S` (1.0 s) is the only scan budget; the old per-probe `_PROBE_TIMEOUT_S` is gone. Probes run on daemon threads (`_spawn_probe`: one `Future` each, deliberately not a `ThreadPoolExecutor`, whose non-daemon workers are joined at interpreter exit and would hold the CLI open for an abandoned probe). The scan returns at the deadline and drops stragglers. Python cannot kill a thread, so an abandoned probe keeps running in the background until it finishes; the one that can run long is `_probe_specialization_suggestion` (unbounded `rglob`), and the picker, the interview signals and the welcome panel each start a scan, so up to three walkers can overlap on a huge cwd. Probes must not write to stdout/stderr (a daemon thread caught mid-write at interpreter exit can abort the process).
+- **One wall-clock cap.** `_TOTAL_TIMEOUT_S` (1.0 s) is the only scan budget; the old per-probe `_PROBE_TIMEOUT_S` is gone. Probes run on daemon threads (`_spawn_probe`: one `Future` each, deliberately not a `ThreadPoolExecutor`, whose non-daemon workers are joined at interpreter exit and would hold the CLI open for an abandoned probe). The scan returns at the deadline and drops stragglers. Python cannot kill a thread, so an abandoned probe keeps running until it finishes (the one that used to run for minutes is the tree walk, below). Probes must not write to stdout/stderr (a daemon thread caught mid-write at interpreter exit can abort the process).
 - **Cold imports are off the clock.** `_warm_probe_imports` imports `_WARM_MODULES` (the AI-config pydantic chain, the dbt runner) from the main thread while the probes run, then the budget starts. Probe imports stay function-local: if a probe starts importing something new, list it in `_WARM_MODULES` or `tests/test_welcome_scan_budget.py::test_probes_import_nothing_new_after_the_warm_up` fails. NB `build_parser()` already imports `ai_setup`, the dbt runner and pydantic, so in a real CLI process only `copilot.unified_config` is still cold; the warm-up matters for tests and library callers.
 - **dbt has its own cap.** `_DBT_VERSION_TIMEOUT_S` (0.32 s) sizes the `dbt --version` subprocess and is not derived from the scan budget and must stay below it, so the child is killed by its own timeout instead of being orphaned. With Python dbt-core on PATH it usually waits out that cap, which is the scan's floor (~0.35 s).
-- `scan_duration_ms` is the real wall-clock of the call, imports included. Pins: `tests/test_welcome_scan_budget.py`.
+- **The specialization walk is bounded and ends with its scan.** `_probe_specialization_suggestion` used to `rglob` the whole cwd tree (279 s under a directory holding many repos), and the picker, the interview signals and the welcome panel each start a scan, so up to three abandoned walkers overlapped (~0.5 core). Its contracts now come from `_find_contracts`: depth-first in name order, so the first 8 are exactly `sorted(rglob(...))[:8]` wherever the bounds do not cut. It skips `util/upstream_discovery`'s `IGNORED_DIRS | BUILD_OUTPUT_DIRS`, lists at most `_SPECIALIZATION_MAX_DEPTH` (6) levels, reads at most `_SPECIALIZATION_MAX_ENTRIES` (10,000) entries, and does not follow symlinks or take anything but a regular file. `run_welcome_scan` sets a `threading.Event` when it returns and the walk ends at its next directory; `_WALK_GATE` runs one walk at a time, so a walker stuck in a filesystem call (a dead mount) is the only one a process can hold. `tests/fragments/test_no_raw_readers.py` registers `_probe_specialization_suggestion` as an accepted raw reader. The scanner finds it through the word `contract` in that function (the `contract.fluid.yaml` in its docstring, the `_find_contracts` call); the registry only checks that the function exists, so losing both silently orphans the entry.
+- `scan_duration_ms` is the real wall-clock of the call, imports included. Pins: `tests/test_welcome_scan_budget.py`, `tests/test_welcome_scan_walk.py`.
 
 ## Working style
 

fluid_build/build_runners/_secret_policy.py

@@ -197,8 +197,9 @@ def variable_allowed(
 ) -> bool:
     """Whether ``variable`` may be sent to ``host``: the decision, without wording.
 
-    For a caller that phrases its own refusal (the managed Flink lane's catalog
-    token, ``streaming/policy.py``). A loopback host needs no entry.
+    For a caller that phrases its own refusal: the managed Flink lane's catalog
+    token (``streaming/policy.py``) and the Iceberg sinks' catalog token
+    (``kafka_connect/iceberg_sink_validation.py``). A loopback host needs no entry.
     """
     return is_loopback(host) or _allows(variable, host, allowlist(environ))
 

fluid_build/build_runners/debezium/runner.py

@@ -706,7 +706,8 @@ def _execute_debezium_server(
 
     # The server inherits this process's environment, which is where Iceberg
     # resolves the ``env:<NAME>`` catalog properties ``sink.catalogAuth``
-    # emits, and where the preflight checked the variable is set. The
+    # emits, and where the preflight checked the variable is set, once the
+    # operator had paired it with every host it reaches. The
     # ``FLUID_DBZ_*`` credentials the file refers to are added to it. No
     # secret is written to application.properties.
     try: