Skip to content

[Java] Use GcsUtil V2 by default - #40713

Merged
shunping merged 7 commits into
apache:masterfrom
shunping:gcs-migration-11
Oct 22, 2026
Merged

shunping merged 7 commits into
apache:masterfrom
shunping:gcs-migration-11

Conversation

@shunping

@shunping shunping commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

GcsUtil now uses GcsUtilV2 (java-storage) by default instead of GcsUtilV1 (gcsio). Before this, V2 was opt-in through the use_gcsutil_v2 experiment.

Experiments

Experiment Behavior
(none) GcsUtil V2 (new default)
disable_gcsutil_v2 New. Falls back to GcsUtil V1
use_grpc_for_gcs Falls back to GcsUtil V1, since only gcsio supports gRPC
use_gcsutil_v2 Accepted but does nothing now. Kept so existing pipelines don't break

Either fallback experiment wins over use_gcsutil_v2. The names follow the existing use_runner_v2 / disable_runner_v2 pair.

GcsUtil now uses GcsUtilV2 (java-storage) by default. The
disable_gcsutil_v2 experiment falls back to GcsUtilV1 (gcsio).
use_gcsutil_v2 is still accepted but does nothing.
@shunping shunping changed the title Use GcsUtil V2 by default [Java] Use GcsUtil V2 by default Oct 21, 2026
@shunping

Copy link
Copy Markdown
Collaborator Author

r: @damccorm

@shunping
shunping marked this pull request as ready for review October 21, 2026 22:25
@shunping
shunping requested a review from damccorm October 21, 2026 22:25
@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@codecov

codecov Bot commented Oct 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.04%. Comparing base (a8d6afd) to head (b4eb547).
⚠️ Report is 35 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #40713      +/-   ##
==========================================
- Coverage   80.04%   80.04%   -0.01%     
==========================================
  Files         490      490              
  Lines       82916    82888      -28     
==========================================
- Hits        66373    66350      -23     
+ Misses      16543    16538       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR switches GcsUtil to use GcsUtilV2 (java-storage) by default while providing --experiments=disable_gcsutil_v2 (and preserving use_grpc_for_gcs) to fall back to GcsUtilV1 (gcsio). It also fixes toStorageObject(Blob) to preserve kmsKeyName when delegating getObject to V2 and updates the corresponding unit, integration, and load tests.

🔍 General Feedback

  • The implementation and fallback precedence in GcsUtil are clean, well-documented, and thoroughly covered by unit and integration tests.
  • Good catch preserving kmsKeyName in GcsUtil.toStorageObject(Blob) and propagating the experiment in GcsUtilIT.testRewriteMultiPart.
  • One minor note: the two newly added .github/trigger_files/beam_PostCommit_Java_Examples_*.json files don't appear to have corresponding workflows listening to them in .github/workflows/ (unlike beam_PostCommit_Java_Examples_Dataflow_Java.json).

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

if (ExperimentalOptions.hasExperiment(options, DISABLE_GCSUTIL_V2_EXPERIMENT)) {
v1Experiment = DISABLE_GCSUTIL_V2_EXPERIMENT;
} else if (ExperimentalOptions.hasExperiment(options, USE_GRPC_FOR_GCS_EXPERIMENT)) {
v1Experiment = USE_GRPC_FOR_GCS_EXPERIMENT;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we plan to enable grpc with v2 ever? Or why is this scoped out?

@shunping shunping Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is in our plan to support gRPC in V2. However, since gRPC support in V1 is still controlled by an experiment, we didn't want it to block the default mode (HTTP support) switch to V2.

In addition, we'll need to run another round of benchmarks comparing HTTP/gRPC modes on V2 and gRPC mode on V1/V2 to make sure there are no discrepancies before we route gRPC support to V2.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM

@damccorm damccorm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@shunping
shunping merged commit 94b84b6 into apache:master Oct 22, 2026
39 of 45 checks passed
@shunping
shunping deleted the gcs-migration-11 branch October 22, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants