[Go] Support non-default universes for GCS and Dataflow clients by shunping · Pull Request #40686 · apache/beam

shunping · 2026-10-20T14:53:07Z

Summary

The Go SDK's GCS and Dataflow clients default to *.googleapis.com endpoints, causing failures in non-default universes like Trusted Partner Cloud (TPC) due to universe-mismatch errors or hardcoded URLs.

Following the Python fix (#40674), this PR adds non-default universe support and introduces a --gcs_endpoint override (aligning with Java's --gcsEndpoint and the existing --dataflow_endpoint).

Changes

gcpopts:
- Added --gcs_endpoint flag and GetGcsEndpoint() (falls back to pipeline options on workers).
- Added cached GetUniverseDomain(ctx) reading GOOGLE_CLOUD_UNIVERSE_DOMAIN or default credentials.
gcsx: Configured NewClient and NewUnauthenticatedClient with --gcs_endpoint and universe domain options.
dataflowlib: Migrated deprecated df.New to df.NewService, passing endpoint and universe domain options.
dataflow runner: Updated bucket checks to use gcsx.NewClient for consistent endpoint and universe handling.

Behavior changes

Dataflow client credentials now come from the newer Google auth library (through df.NewService) instead of oauth2/google. A mismatch between the client's and the credentials' universe now fails with a clear error instead of a confusing server-side failure.
A --dataflow_endpoint value containing :// is used as-is, as before. A bare host[:port] is now combined with the default endpoint. Previously it was set as BasePath directly and produced an invalid URL.
Nothing changes in the default universe (googleapis.com) when no flags or env vars are set.

Usage in TPC

Either rely on the universe domain from the credentials or the env var:

export GOOGLE_CLOUD_UNIVERSE_DOMAIN=<universe-domain>

or set the endpoints explicitly:

--dataflow_endpoint=https://dataflow.<universe-domain>/
--gcs_endpoint=https://storage.<universe-domain>/storage/v1/

Add a --gcs_endpoint flag. gcsx clients now pass the endpoint and the universe domain, taken from GOOGLE_CLOUD_UNIVERSE_DOMAIN or the default credentials, to storage.NewClient. This allows GCS access in Trusted Partner Cloud (TPC).

Replace the deprecated df.New with df.NewService, passing --dataflow_endpoint via option.WithEndpoint and the universe domain from GOOGLE_CLOUD_UNIVERSE_DOMAIN or the default credentials. Share the universe domain resolution with the GCS client via gcpopts.

codecov · 2026-10-20T15:07:17Z

Codecov Report

❌ Patch coverage is 78.94737% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.90%. Comparing base (a8d6afd) to head (a724989).
⚠️ Report is 21 commits behind head on master.

Files with missing lines	Patch %	Lines
sdks/go/pkg/beam/util/gcsx/gcs.go	50.00%	6 Missing ⚠️
sdks/go/pkg/beam/options/gcpopts/options.go	90.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #40686       +/-   ##
===========================================
- Coverage   80.04%   28.90%   -51.15%     
===========================================
  Files         490      381      -109     
  Lines       82916    71208    -11708     
===========================================
- Hits        66373    20583    -45790     
- Misses      16543    48767    +32224     
- Partials        0     1858     +1858

Flag	Coverage Δ
go	`28.90% <78.94%> (?)`
python	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions

📋 Review Summary

This PR adds support for non-default Google Cloud universes (such as Trusted Partner Cloud) in the Go SDK's GCS and Dataflow clients by propagating the universe domain from GOOGLE_CLOUD_UNIVERSE_DOMAIN or default credentials via option.WithUniverseDomain, and introduces a --gcs_endpoint flag/pipeline option override. The implementation is clean, well-structured, and includes appropriate caching and unit tests.

🔍 General Feedback

The caching strategy in gcpopts.GetUniverseDomain properly avoids caching credential lookup failures while preventing repeated metadata server calls.
Migrating dataflowlib.NewClient from deprecated df.New to df.NewService with option.ClientOptions and switching the Dataflow runner's soft-delete check to gcsx.NewClient ensures consistent endpoint and universe handling across the SDK.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

github-actions · 2026-10-20T16:17:35Z

Assigning reviewers:

R: @lostluck for label go.
Backup reviewer: jrmccluskey (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

lostluck

Just a few nits & comments.

lostluck · 2026-10-20T23:42:17Z

+var cachedUniverseDomain struct {
+	mu       sync.Mutex
+	resolved bool
+	value    string
+}


Beautiful use of an anonymous struct type.

lostluck · 2026-10-20T23:44:03Z

+			t.Cleanup(func() { findDefaultCredentials = orig })
+
+			for i := 0; i < 2; i++ {
+				if got := GetUniverseDomain(context.Background()); got != test.want {


nit, use t.Context() here instead.

lostluck · 2026-10-20T23:44:29Z

+			}
+			t.Cleanup(func() { findDefaultCredentials = orig })
+
+			for i := 0; i < 2; i++ {


Add a comment why we're doing repetitions here please.

lostluck · 2026-10-20T23:46:19Z

+		t.Run(test.name, func(t *testing.T) {
+			t.Setenv("GOOGLE_CLOUD_UNIVERSE_DOMAIN", test.env)
+
+			client, err := NewClient(context.Background(), test.endpoint)


Same nit, prefer t.Context() in tests.

shunping added 2 commits October 20, 2026 10:42

shunping changed the title ~~Tpc 5~~ [Go] Support non-default universes for GCS and Dataflow clients Oct 20, 2026

github-actions Bot added go runners dataflow labels Oct 20, 2026

Add CHANGES.md

9072f14

[Go] Simplify universe domain handling and dedup tests

acbfd09

shunping marked this pull request as ready for review October 20, 2026 16:08

github-actions Bot reviewed Oct 20, 2026

View reviewed changes

github-actions Bot requested a review from lostluck October 20, 2026 16:17

github-actions Bot added the Next Action: Reviewers label Oct 20, 2026

lostluck approved these changes Oct 20, 2026

View reviewed changes

shunping added 3 commits October 20, 2026 21:26

Merge branch 'master' into tpc-5

7e75a53

Use t.Context() in new tests

f5afce3

Add a comment to explain the loop in test.

a724989

shunping merged commit f1ec3ac into apache:master Oct 21, 2026
14 of 15 checks passed

derrickaw mentioned this pull request Oct 22, 2026

[Go] Avoid importing gcpopts in gcsx and dataflowlib to prevent flag collisions #40729

Merged

Abacn mentioned this pull request Oct 22, 2026

[Bug]: Go SDK pipeline options define global flags with generic names, causing flag redefinition panics #40730

Open

1 of 17 tasks

Navigation Menu

Repository navigation

[Go] Support non-default universes for GCS and Dataflow clients - #40686

[Go] Support non-default universes for GCS and Dataflow clients#40686
shunping merged 7 commits into
apache:masterfrom
shunping:tpc-5

shunping commented Oct 20, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

lostluck left a comment

Uh oh!

lostluck Oct 20, 2026

Uh oh!

lostluck Oct 20, 2026

Uh oh!

shunping Oct 21, 2026

Uh oh!

lostluck Oct 20, 2026

Uh oh!

shunping Oct 21, 2026

Uh oh!

lostluck Oct 20, 2026

Uh oh!

shunping Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

shunping commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Behavior changes

Usage in TPC

Uh oh!

codecov Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

📋 Review Summary

🔍 General Feedback

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

lostluck left a comment

Choose a reason for hiding this comment

Uh oh!

lostluck Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

lostluck Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

shunping Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

lostluck Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

shunping Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

lostluck Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

shunping Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shunping commented Oct 20, 2026 •

edited

Loading

codecov Bot commented Oct 20, 2026 •

edited

Loading