[lhotari]

Motivation

This is the follow-up to this review comment on #27343.

ZooKeeper 3.9.7 added the multiRead.maxOps (default 1000) and multiRead.maxResponseBytes (default 64 MiB) server limits. A multiRead request over either limit is rejected as a whole with BADARGUMENTS, and ZKMetadataStore failed every read in the batch.

ZKMetadataStoreBatchStrategy forms batches from estimated response sizes, so a batch can go over the limit even though each read in it is valid. With jute.maxbuffer=41943040 (40 MiB) and the default limits, two 35 MiB reads in one batch fail with BadArguments. On ZooKeeper 3.9.6 the same batch recovered, because it failed with CONNECTIONLOSS and the reads were retried individually.

The guidance added in #27343 was also wrong. It said that the response limit only has to be larger than jute.maxbuffer, but the case above fails with a jute.maxbuffer well below the limit, and raising the limit above jute.maxbuffer doesn't prevent it for larger batches.

Modifications

• ZKMetadataStore: when a read batch of two or more operations is rejected with BADARGUMENTS, halve it and retry both parts.
  • Only batches that contain nothing but GET and GET_CHILDREN operations are split. Write batches behave as before.
  • A rejected single read fails its own future and isn't retried, so a batch of n reads makes at most 2n - 1 requests.
  • The batch is halved rather than split into single reads, so setting multiRead.maxOps too low doesn't turn a large batch into one request per read.
• Guidance: corrected in conf/zookeeper.conf, conf/global_zookeeper.conf, conf/broker.conf, conf/standalone.conf and ServiceConfiguration.
  • Keeping multiRead.maxOps at least as large as metadataStoreBatchingMaxOperations now only avoids extra requests.
  • multiRead.maxResponseBytes must be larger than the response to any single read, such as the largest z-node's data or children list.
• TestZKServer: added setMultiReadLimits so that tests can change the limits on the running server.

The server limits stay enabled with their defaults. A single read over multiRead.maxResponseBytes still fails, even if jute.maxbuffer would allow it.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Added ZKMetadataStoreMultiReadLimitsTest, which runs against a real ZooKeeper server and covers a batch over the response limit, a batch over the operation count limit, and a single oversized read failing without affecting the other reads in its batch. All 3 tests fail with the fix disabled and pass with it.
• Ran the scenario from the review comment with a temporary test that isn't part of this PR: 35 MiB z-nodes, jute.maxbuffer=41943040, default limits and metadataStoreBatchingMaxOperations=2, with 3 reads. With the fix disabled, the 2 reads in the first batch fail with BadArguments. With the fix, all 3 succeed.
• MetadataStoreBatchingTest, ZKSessionTest and ZKMetadataStoreBatchIOperationTest pass locally.
• MetadataStoreTest passes locally except for testGetChildren[ZooKeeper] when run with retries off. That test also failed the same way on ZooKeeper 3.9.6 during the work on [fix][sec] Upgrade ZooKeeper to 3.9.7 to address several CVEs #27343, so this change doesn't cause it.

Not covered: there's no test that write batches aren't split, because I couldn't readily get the server to reject a write batch with BADARGUMENTS. The full broker suite and the integration tests were not run locally.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

[codelipenghui]

Code Review approval for commit 16b3920.

Summary:

No actionable findings identified within the reviewed scope.

The read-only splitting approach is bounded, preserves write behavior, and retains ZooKeeper’s limits. Disabling the fallback reproduced both rejection modes; restoring it resolved them. The final focused run passed 36 tests. An earlier session-event ordering failure remains an inconclusive validation limitation. All temporary source edits were restored; HEAD remains 16b392070ebd175b99faf2dc06bfe5cf9d08351d.