Repository navigation
[Java] Use GcsUtil V2 by default - #40713
Conversation
GcsUtil now uses GcsUtilV2 (java-storage) by default. The disable_gcsutil_v2 experiment falls back to GcsUtilV1 (gcsio). use_gcsutil_v2 is still accepted but does nothing.
|
r: @damccorm |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40713 +/- ##
==========================================
- Coverage 80.04% 80.04% -0.01%
==========================================
Files 490 490
Lines 82916 82888 -28
==========================================
- Hits 66373 66350 -23
+ Misses 16543 16538 -5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
📋 Review Summary
This PR switches GcsUtil to use GcsUtilV2 (java-storage) by default while providing --experiments=disable_gcsutil_v2 (and preserving use_grpc_for_gcs) to fall back to GcsUtilV1 (gcsio). It also fixes toStorageObject(Blob) to preserve kmsKeyName when delegating getObject to V2 and updates the corresponding unit, integration, and load tests.
🔍 General Feedback
- The implementation and fallback precedence in
GcsUtilare clean, well-documented, and thoroughly covered by unit and integration tests. - Good catch preserving
kmsKeyNameinGcsUtil.toStorageObject(Blob)and propagating the experiment inGcsUtilIT.testRewriteMultiPart. - One minor note: the two newly added
.github/trigger_files/beam_PostCommit_Java_Examples_*.jsonfiles don't appear to have corresponding workflows listening to them in.github/workflows/(unlikebeam_PostCommit_Java_Examples_Dataflow_Java.json).
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
| if (ExperimentalOptions.hasExperiment(options, DISABLE_GCSUTIL_V2_EXPERIMENT)) { | ||
| v1Experiment = DISABLE_GCSUTIL_V2_EXPERIMENT; | ||
| } else if (ExperimentalOptions.hasExperiment(options, USE_GRPC_FOR_GCS_EXPERIMENT)) { | ||
| v1Experiment = USE_GRPC_FOR_GCS_EXPERIMENT; |
There was a problem hiding this comment.
Do we plan to enable grpc with v2 ever? Or why is this scoped out?
There was a problem hiding this comment.
It is in our plan to support gRPC in V2. However, since gRPC support in V1 is still controlled by an experiment, we didn't want it to block the default mode (HTTP support) switch to V2.
In addition, we'll need to run another round of benchmarks comparing HTTP/gRPC modes on V2 and gRPC mode on V1/V2 to make sure there are no discrepancies before we route gRPC support to V2.
GcsUtilnow usesGcsUtilV2(java-storage) by default instead ofGcsUtilV1(gcsio). Before this, V2 was opt-in through theuse_gcsutil_v2experiment.Experiments
disable_gcsutil_v2use_grpc_for_gcsuse_gcsutil_v2Either fallback experiment wins over
use_gcsutil_v2. The names follow the existinguse_runner_v2/disable_runner_v2pair.