Repository navigation
Conversation
Add defineGCSHmacConnection for GCS's S3-compatible HMAC (access-key-id + secret) auth mode, distinct from defineGCSConnection's service account JSON mode, and extend defineSinkPipe's sink destinations with a GCS HMAC bucket export variant alongside the existing Kafka and S3 ones. - schema/connection.ts: GCSHmacConnectionOptions/Definition, defineGCSHmacConnection, isGCSHmacConnectionDefinition - generator/connection.ts: emits `TYPE gcs_hmac` / GCS_HMAC_ACCESS_ID / GCS_HMAC_SECRET - schema/pipe.ts: GCSHmacSinkConfig (same shape as S3SinkConfig), validateSinkConfig accepts S3 or GCS HMAC connections for bucket sinks - migrate/*: parse-connection.ts, parse-pipe.ts, emit-ts.ts, and types.ts extended so the datafile->TS migration path round-trips gcs_hmac connections and sinks instead of rejecting/misclassifying them (src/codegen/ has no connection/sink reverse-codegen at all today, so nothing to extend there) - index.ts: exports the new public types/functions Field names (TYPE gcs_hmac, GCS_HMAC_ACCESS_ID, GCS_HMAC_SECRET, EXPORT_SERVICE gcs_hmac) were traced to tinybird/connector_settings.py, tinybird/tb/modules/connection.py, and tinybird/tb/modules/datafile/format_connection.py in the analytics monorepo (DataConnectorType.GCLOUD_STORAGE_HMAC = "gcs_hmac"). Per tinybird/datafile/common.py, the modern Forward backend treats EXPORT_SERVICE as deprecated/inferred-from-connection for sink pipes, so the TS generator (like the existing S3 sink) does not emit it. Refs PROTM-2269 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes two gaps vs. the Forward CLI identified in PROTM-2269:
defineGCSConnectionpreviously only covered the service-account JSON auth mode (gcs/gcs_service_accountin Forward'sDataConnectorType). AddeddefineGCSHmacConnectionfor the S3-compatible, access-key-id + secret auth mode (gcs_hmac), following the same shape asdefineS3Connection.defineSinkPipepreviously only supported Kafka and S3 export destinations. Added aGCSHmacSinkConfigbucket-export destination, identical in shape to the existingS3SinkConfig(bucket URI, file template, format, schedule, strategy, compression), just backed by a GCS HMAC connection.Changes
src/schema/connection.ts—GCSHmacConnectionOptions/GCSHmacConnectionDefinition,defineGCSHmacConnection,isGCSHmacConnectionDefinition, added to theConnectionDefinitionunion.src/generator/connection.ts— emitsTYPE gcs_hmac/GCS_HMAC_ACCESS_ID/GCS_HMAC_SECRET.src/schema/pipe.ts—GCSHmacSinkConfigadded toSinkConfig;validateSinkConfignow accepts an S3 or GCS HMAC connection for bucket-based sinks.src/generator/pipe.ts— no change needed; the existing bucket-sink branch ofgenerateSinkis generic over the shared field shape.src/migrate/*(parse-connection.ts,parse-pipe.ts,emit-ts.ts,types.ts) — extended the datafile → TS migration path (used bytinybird migrate) so hand-writtenTYPE gcs_hmacconnections andEXPORT_SERVICE gcs_hmacsinks round-trip intodefineGCSHmacConnection/the new sink config instead of being rejected or silently misclassified. (src/codegen/has no connection or sink-pipe reverse-codegen at all today — only datasources and endpoint/materialized/copy pipes — so there was nothing to extend there.)src/index.ts— exports the new public types/functions.Where the field names came from
Traced in
/Users/tommy/Documents/tinybird/analytics(the Forward CLI/platform monorepo; thetinybird-sdk-pythonandtinybird/platform-apirepos had no matches forgcloud_storage_hmac/gcs_hmac):tinybird/connector_settings.py:DataConnectorType.GCLOUD_STORAGE_HMAC = "gcs_hmac", with settingsgcs_hmac_access_id/gcs_hmac_secret.tinybird/tb/modules/datafile/format_connection.py:GCS_SETTINGS_ORDERconfirms the uppercase datafile keysGCS_HMAC_ACCESS_ID/GCS_HMAC_SECRETunderTYPE gcs_hmac.tests_tb/test_fmt.py(test_fmt_gcs_hmac_connection): confirms a hand-authored.connectionfile withTYPE gcs_hmac/GCS_HMAC_ACCESS_ID/GCS_HMAC_SECRETround-trips throughtb fmtunchanged.tests/data_projects/test_sinks.pyandtinybird/data_connector.py'sExportReplacements.SERVICES = ("gcs_hmac", "s3", "s3_iamrole", "kafka"): confirmsEXPORT_SERVICE gcs_hmacplus the sameEXPORT_BUCKET_URI/EXPORT_FILE_TEMPLATE/EXPORT_FORMAT/EXPORT_SCHEDULE/EXPORT_STRATEGY/EXPORT_COMPRESSIONfields used by S3 sinks.tinybird/datafile/common.py(the parser actually used by the modern Forward CLI'sbuild/deploy):EXPORT_SERVICEis deprecated and silently ignored for sink pipes — the export service is inferred from the referenced connection instead. This matches the existing TS generator's S3 sink behavior (it never emitsEXPORT_SERVICE), so the new GCS HMAC sink also doesn't emit it..pipe/.connectionfixtures undertinybird/data_projects/Internal/confirm S3 sinks in the wild never includeEXPORT_SERVICEeither, reinforcing the above.Risk flag for reviewers: I found no example of a hand-authored
.pipefile combiningEXPORT_SERVICE gcs_hmacwith real production bucket paths (only test fixtures), and themigratereverse-parser cannot disambiguate an S3 vs. GCS HMAC bucket sink from directives alone whenEXPORT_SERVICEis omitted (both share identical field names) — it still defaults tos3in that ambiguous case, the same way it already did before this change. The cross-check against the referenced connection's actual type (already present insrc/cli/commands/migrate.ts) will raise a clear error in that scenario rather than silently mis-migrating, but it's worth double-checking against current API docs/behavior if this matters for your use case.Test plan
pnpm typecheckpassespnpm test:runpasses (1082 tests, including new coverage for: defining a GCS HMAC connection, its validation errors, its generated.connectioncontent; defining a GCS HMAC sink pipe, its validation errors, its generated.pipecontent; migrate round-trip of aTYPE gcs_hmacconnection andEXPORT_SERVICE gcs_hmacsink into TS, plus a connection/sink type-mismatch error case)CHANGELOG.mdorpackage.jsonversionsrc/index.ts🤖 Generated with Claude Code