Skip to content

PROTM-2269: Add GCS HMAC connector distinction + GCS sink destination - #184

Open
sandshoes wants to merge 1 commit into
mainfrom
tommy/protm-2269-ts-sdk-gcs-hmac-connector-sink
Open

sandshoes wants to merge 1 commit into
mainfrom
tommy/protm-2269-ts-sdk-gcs-hmac-connector-sink

Conversation

@sandshoes

Copy link
Copy Markdown
Contributor

Summary

Closes two gaps vs. the Forward CLI identified in PROTM-2269:

  1. defineGCSConnection previously only covered the service-account JSON auth mode (gcs/gcs_service_account in Forward's DataConnectorType). Added defineGCSHmacConnection for the S3-compatible, access-key-id + secret auth mode (gcs_hmac), following the same shape as defineS3Connection.
  2. defineSinkPipe previously only supported Kafka and S3 export destinations. Added a GCSHmacSinkConfig bucket-export destination, identical in shape to the existing S3SinkConfig (bucket URI, file template, format, schedule, strategy, compression), just backed by a GCS HMAC connection.

Changes

  • src/schema/connection.ts — GCSHmacConnectionOptions/GCSHmacConnectionDefinition, defineGCSHmacConnection, isGCSHmacConnectionDefinition, added to the ConnectionDefinition union.
  • src/generator/connection.ts — emits TYPE gcs_hmac / GCS_HMAC_ACCESS_ID / GCS_HMAC_SECRET.
  • src/schema/pipe.ts — GCSHmacSinkConfig added to SinkConfig; validateSinkConfig now accepts an S3 or GCS HMAC connection for bucket-based sinks.
  • src/generator/pipe.ts — no change needed; the existing bucket-sink branch of generateSink is generic over the shared field shape.
  • src/migrate/* (parse-connection.ts, parse-pipe.ts, emit-ts.ts, types.ts) — extended the datafile → TS migration path (used by tinybird migrate) so hand-written TYPE gcs_hmac connections and EXPORT_SERVICE gcs_hmac sinks round-trip into defineGCSHmacConnection/the new sink config instead of being rejected or silently misclassified. (src/codegen/ has no connection or sink-pipe reverse-codegen at all today — only datasources and endpoint/materialized/copy pipes — so there was nothing to extend there.)
  • src/index.ts — exports the new public types/functions.

Where the field names came from

Traced in /Users/tommy/Documents/tinybird/analytics (the Forward CLI/platform monorepo; the tinybird-sdk-python and tinybird/platform-api repos had no matches for gcloud_storage_hmac/gcs_hmac):

  • tinybird/connector_settings.py: DataConnectorType.GCLOUD_STORAGE_HMAC = "gcs_hmac", with settings gcs_hmac_access_id / gcs_hmac_secret.
  • tinybird/tb/modules/datafile/format_connection.py: GCS_SETTINGS_ORDER confirms the uppercase datafile keys GCS_HMAC_ACCESS_ID / GCS_HMAC_SECRET under TYPE gcs_hmac.
  • tests_tb/test_fmt.py (test_fmt_gcs_hmac_connection): confirms a hand-authored .connection file with TYPE gcs_hmac / GCS_HMAC_ACCESS_ID / GCS_HMAC_SECRET round-trips through tb fmt unchanged.
  • tests/data_projects/test_sinks.py and tinybird/data_connector.py's ExportReplacements.SERVICES = ("gcs_hmac", "s3", "s3_iamrole", "kafka"): confirms EXPORT_SERVICE gcs_hmac plus the same EXPORT_BUCKET_URI/EXPORT_FILE_TEMPLATE/EXPORT_FORMAT/EXPORT_SCHEDULE/EXPORT_STRATEGY/EXPORT_COMPRESSION fields used by S3 sinks.
  • tinybird/datafile/common.py (the parser actually used by the modern Forward CLI's build/deploy): EXPORT_SERVICE is deprecated and silently ignored for sink pipes — the export service is inferred from the referenced connection instead. This matches the existing TS generator's S3 sink behavior (it never emits EXPORT_SERVICE), so the new GCS HMAC sink also doesn't emit it.
  • Real production .pipe/.connection fixtures under tinybird/data_projects/Internal/ confirm S3 sinks in the wild never include EXPORT_SERVICE either, reinforcing the above.

Risk flag for reviewers: I found no example of a hand-authored .pipe file combining EXPORT_SERVICE gcs_hmac with real production bucket paths (only test fixtures), and the migrate reverse-parser cannot disambiguate an S3 vs. GCS HMAC bucket sink from directives alone when EXPORT_SERVICE is omitted (both share identical field names) — it still defaults to s3 in that ambiguous case, the same way it already did before this change. The cross-check against the referenced connection's actual type (already present in src/cli/commands/migrate.ts) will raise a clear error in that scenario rather than silently mis-migrating, but it's worth double-checking against current API docs/behavior if this matters for your use case.

Test plan

  • pnpm typecheck passes
  • pnpm test:run passes (1082 tests, including new coverage for: defining a GCS HMAC connection, its validation errors, its generated .connection content; defining a GCS HMAC sink pipe, its validation errors, its generated .pipe content; migrate round-trip of a TYPE gcs_hmac connection and EXPORT_SERVICE gcs_hmac sink into TS, plus a connection/sink type-mismatch error case)
  • No changes to CHANGELOG.md or package.json version
  • New public types/functions exported from src/index.ts

🤖 Generated with Claude Code

Add defineGCSHmacConnection for GCS's S3-compatible HMAC (access-key-id +
secret) auth mode, distinct from defineGCSConnection's service account
JSON mode, and extend defineSinkPipe's sink destinations with a GCS HMAC
bucket export variant alongside the existing Kafka and S3 ones.

- schema/connection.ts: GCSHmacConnectionOptions/Definition,
  defineGCSHmacConnection, isGCSHmacConnectionDefinition
- generator/connection.ts: emits `TYPE gcs_hmac` / GCS_HMAC_ACCESS_ID /
  GCS_HMAC_SECRET
- schema/pipe.ts: GCSHmacSinkConfig (same shape as S3SinkConfig),
  validateSinkConfig accepts S3 or GCS HMAC connections for bucket sinks
- migrate/*: parse-connection.ts, parse-pipe.ts, emit-ts.ts, and
  types.ts extended so the datafile->TS migration path round-trips
  gcs_hmac connections and sinks instead of rejecting/misclassifying
  them (src/codegen/ has no connection/sink reverse-codegen at all
  today, so nothing to extend there)
- index.ts: exports the new public types/functions

Field names (TYPE gcs_hmac, GCS_HMAC_ACCESS_ID, GCS_HMAC_SECRET,
EXPORT_SERVICE gcs_hmac) were traced to tinybird/connector_settings.py,
tinybird/tb/modules/connection.py, and
tinybird/tb/modules/datafile/format_connection.py in the analytics
monorepo (DataConnectorType.GCLOUD_STORAGE_HMAC = "gcs_hmac"). Per
tinybird/datafile/common.py, the modern Forward backend treats
EXPORT_SERVICE as deprecated/inferred-from-connection for sink pipes,
so the TS generator (like the existing S3 sink) does not emit it.

Refs PROTM-2269

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant