[shunping]

Part of #40569. Note that the BigQuery apitools clients are currently being deprecated (#39889)

Problem

BigQueryWrapper._bigquery_client() builds the apitools BigqueryV2 client without a url, so it falls back to BASE_URL = 'https://bigquery.googleapis.com/bigquery/v2/'. apitools does not read GOOGLE_CLOUD_UNIVERSE_DOMAIN, and Beam has no option to override the endpoint. As a result, BigQuery REST calls (tables, jobs, datasets) always go to the public endpoint. This makes BigQuery IO unusable in Trusted Partner Cloud (TPC) and other non-default universes.

Changes

• New --bigquery_endpoint option in GoogleCloudOptions, the Python equivalent of Java's BigQueryOptions.bigQueryEndpoint (Allow setting BigQuery endpoint #32153). It takes the root URL, e.g. https://bigquery.example.com. The input is normalized: a missing scheme defaults to https, and a trailing / or /bigquery/v2 is stripped.
• Endpoint resolution for BigqueryV2, in this order:
  1. --bigquery_endpoint
  2. https://bigquery.<domain> when GOOGLE_CLOUD_UNIVERSE_DOMAIN is set to something other than googleapis.com
  3. The default public URL
     Step 2 matches how google-cloud-bigquery resolves its endpoint, so both clients target the same universe.
• google.cloud.bigquery.Client (streaming inserts) gets client_options={'api_endpoint': ...} when --bigquery_endpoint is set, merged with quota_project_id. When the option is unset, the library resolves the endpoint itself, including GOOGLE_CLOUD_UNIVERSE_DOMAIN.
• Worker-side fallback: when a wrapper is created without pipeline options, e.g. BigQueryWrapper() inside DoFns, the endpoint is read from the running pipeline's options via RuntimeValueProvider.

Not changed

• The Storage Read API (BigQueryReadClient) does not use --bigquery_endpoint. It is served from a different host (bigquerystorage.<domain>), and the client library already reads GOOGLE_CLOUD_UNIVERSE_DOMAIN. Java does apply its option to the storage clients; this can be revisited if a separate storage endpoint option is needed.
• Default behavior is unchanged when neither the option nor the env var is set.

[github-actions]

📋 Review Summary

This PR adds support for custom BigQuery API endpoints and sovereign cloud universe domains via the --bigquery_endpoint pipeline option and GOOGLE_CLOUD_UNIVERSE_DOMAIN environment variable, wiring them into both the apitools and google-cloud-bigquery clients. The implementation is clean, well-structured, and thoroughly covered by unit tests.

🔍 General Feedback

• The endpoint normalization logic in _normalize_bigquery_root handles trailing slashes, /bigquery/v2 suffixes, and bare hostnames cleanly.
• Passing the derived host from api_url to _HttpWithHeaders ensures x-goog-user-project quota headers continue to work seamlessly against custom endpoints and TPC universe domains without leaking to OAuth token endpoints.

---

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.08%. Comparing base (a8d6afd) to head (6414324).
⚠️ Report is 35 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #40675      +/-   ##
==========================================
+ Coverage   80.04%   80.08%   +0.03%     
==========================================
  Files         490      490              
  Lines       82916    83048     +132     
==========================================
+ Hits        66373    66508     +135     
+ Misses      16543    16540       -3     

Flag	Coverage Δ
python	80.08% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Assigning reviewers:

R: @claudevdm for label python.
Backup reviewer: jrmccluskey (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).