[merlimat]

Motivation

CI - Unit - Brokers - Broker Group 1 is the longest unit test job. In its last three scheduled master runs
(41443581285, 41587218802, 41634315240), the methods below spent about 127 s of each run waiting out fixed waits:
receives that must time out to show that nothing more arrives, client and coordinator retries, and delays the tests
hit by doing one thing at a time.

class	method	CI per run	waited for
LedgerLostAndSkipNonRecoverableTest	testMarkDeletedPositionCanForwardAfterTopicLedgerLost [0], [2]	18.0-18.7, 12.2-12.5 s	two 5 s receive timeouts each, and 100 ms of ack grouping before every ack receipt
TransactionTest	testReadCommittedWithReadCompacted	10.2 s	two 5 s receive timeouts
	testRetryExceptionOfEndTxn	5.4-5.5 s	a 5 s window showing the commit doesn't complete
	testTransactionAckMessages, testTransactionAckMessageList	5.1-5.2 s each	a 5 s receive timeout
	testTBSnapshotWriter	4.1-4.5 s	a 3 s sleep
	testReadCommittedWithCompaction	2.4-2.5 s	a 2 s receive timeout
TransactionMetadataStoreServiceTest	testEndTransactionOpRetry [0], [4]	5.7-5.9 s each	a 5 s window showing the end op doesn't complete
	testTimeoutTracker	4.6-7.9 s	the mock BookKeeper's 1 ms per add, 3000 adds
PendingAckPersistentTest	individualPendingAckReplayTest	14.2-15.6 s	the broker's 10 ms new entries check delay, 1000 times
	cumulativePendingAckReplayTest	8.9-13.0 s	the mock BookKeeper's 1 ms per add
	testBuildConsumerEncounterPendingAckInitFailure [0], [2], [4]	3.6-3.9 s each	the client's 3 s operation timeout
BrokerServiceTest	testCheckInactiveSubscriptionWhenNoMessageToAck	7.9-9.3 s	two 3 s windows showing the subscription isn't deleted

Modifications

Test-only. No assertion is removed, and no timer of the broker or the client that a check depends on is shortened.
One commit per class:

• LedgerLostAndSkipNonRecoverableTest: both receive loops receive the expected number of messages (all of
  them, then all but the acked first entry and the lost ledger) instead of receiving until a 5 s receive times out,
  and the consumer sends its acks at once (acknowledgmentGroupTime(0)): with ack receipts on, every
  acknowledge() waited for the 100 ms grouping. The two acks before the ledger loss go in one
  acknowledge(List), so the cursor persists them with one mark-delete: the cursor's mark-delete rate limit (1 per
  second) keeps a second mark-delete right after the first one unpersisted until the next cursor flush (60 s), and
  the test waits for the persisted position. The old receive timeout left the limiter time to refill.
• TransactionTest: the read-committed and transactional-ack tests receive the expected messages, then use
  assertNoMoreMessages ([improve][test] Speed up ten Broker Group 1 test classes #27322 moved it to TransactionTestBase): every subscription's cursor past the
  transaction buffer's max read position, nothing to redeliver, then a 200 ms receive. testTBSnapshotWriter
  waits until all 20 partitions are loaded, so every partition's transaction buffer holds the injected writer
  future before it fails, instead of sleeping 3 s. testRetryExceptionOfEndTxn checks that the commit doesn't
  complete for 2 s instead of 5 s while the coordinator's store isn't ready: the coordinator retries the end op
  every second (endTransactionRetryIntervalTime), so the window still covers a retry.
• TransactionMetadataStoreServiceTest: testEndTransactionOpRetry calls the service through a Mockito spy,
  which also sees the retries, waits until the end op has been retried (endTransaction ran a second time, 1 s
  later) and checks that it hasn't completed, instead of waiting 5 s. The mock BookKeeper's 1 ms delay per add is
  off: every new transaction and status change is an add to the coordinator's log.
• PendingAckPersistentTest: individualPendingAckReplayTest sent and received its 1000 messages one at a
  time, so every receive waited for the broker's new entries check delay (managedLedgerNewEntriesCheckDelayInMillis,
  10 ms). It now sends them all, still one entry each, then receives them. Both replay tests turn off the mock
  BookKeeper's 1 ms delay per add: every transactional ack is an add to the pending ack log. In
  testBuildConsumerEncounterPendingAckInitFailure, the broker fails every subscribe attempt of the
  non-retryable case the same way, and the client retries until its operation timeout, now 1 s instead of 3 s.
  That is still well past the pending ack handle's 100 ms retry backoff, so a broker retrying the failure would
  still let the subscribe succeed (the mock's next answer) and fail the test.
• BrokerServiceTest.testCheckInactiveSubscriptionWhenNoMessageToAck: the subscription has to stay for 500 ms
  instead of 3 s after each inactive subscription check. The check decides at once, and a subscription it deletes
  is gone within milliseconds. The 1 s sleep stays: both checks need the last activity to be older than their 1 s
  expiration time.

Kept: the transaction timeouts that TransactionMetadataStoreServiceTest's timeout tests wait for, which are what
they check, and the broker restart before each method of TransactionMetadataStoreServiceTest and
PendingAckPersistentTest. The former's methods rely on a fresh coordinator log (open transactions left by one
method would be recovered by the next), the latter's replace broker components and change its configuration
without restoring them, and #27325 adds methods to it. LedgerLostAndSkipNonRecoverableTest's 15-17 s outside its
methods in CI is the cold start of its shard's JVM: it runs first in its shard, and the first class of every shard
pays about the same.

Verifying this change

Back to back on the same machine, ./gradlew :pulsar-broker:test --tests <class> --rerun -PtestFailFast=false,
master and this change alternating, two rounds, times in seconds (class time includes the methods not changed):

class	master	this PR	changed methods, master → this PR
LedgerLostAndSkipNonRecoverableTest	30.0 / 29.9	3.6 / 3.6	15.5 → 0.7-0.8, 11.7 → 0.3
TransactionTest	49.0 / 48.9	22.1 / 23.0	testReadCommittedWithReadCompacted 10.2 → 0.8, testRetryExceptionOfEndTxn 5.4-5.5 → 2.2, testTransactionAckMessages 5.2 → 0.5, testTransactionAckMessageList 5.1 → 0.4, testTBSnapshotWriter 4.1-4.2 → 1.2-1.4, testReadCommittedWithCompaction 2.5 → 0.8
TransactionMetadataStoreServiceTest	38.9 / 40.6	30.0 / 31.9	testEndTransactionOpRetry 5.7 → 2.4-2.5 (each), testTimeoutTracker 5.0 → 2.5-2.8, testTimeoutTrackerMultiThreading 4.9 → 4.3-4.4
PendingAckPersistentTest	50.8 / 50.6	24.9 / 25.2	individualPendingAckReplayTest 15.7-15.9 → 1.6, cumulativePendingAckReplayTest 9.1-9.7 → 2.5-2.9, testBuildConsumerEncounterPendingAckInitFailure 3.5-3.8 → 1.9-2.1 (each)
BrokerServiceTest	51.4 / 60.1	45.9 / 46.1	7.5-7.6 → 2.8

The changed methods take ~32 s locally instead of ~125 s, about 1.5 minutes less test time per run of Broker Group 1
(~25 s less job time with its four shards).

All the changed methods together (25 tests, with the whole of LedgerLostAndSkipNonRecoverableTest and
TransactionMetadataStoreServiceTest), four forks, retries off (-PtestRetryCount=0): 10 rounds without a failure. In testTBSnapshotWriter, all 20 partitions still get the
injected writer failure (20 "Failed to create snapshot writer" logs per run, as on master).

Mutation checks, one broker change at a time, with master's test and this one (retries off):

Mutation	Test	master	this PR
The cursor doesn't auto-acknowledge a lost ledger (skipNonRecoverableLedger does nothing)	testMarkDeletedPositionCanForwardAfterTopicLedgerLost [0], [2]	fails, 30 s (method timeout)	fails, 30 s (method timeout)
The transaction buffer lets consumers read past open transactions	testReadCommittedWithReadCompacted	fails, 5.3 s	fails, 0.4 s
A committed transaction's individual acks are dropped (no cursor ack, no longer pending)	testTransactionAckMessages	fails, 0.3 s	fails, 0.4 s
	testTransactionAckMessageList	fails, 0.1 s	fails, 0.2 s
The coordinator doesn't retry an end op while its store isn't ready	testRetryExceptionOfEndTxn	fails, 0.0 s	fails, 0.0 s
	testEndTransactionOpRetry [0], [4]	fails, 0.2-0.4 s	fails, 10.2-10.6 s
A failed transaction buffer snapshot writer stays in the writer map	testTBSnapshotWriter	fails, 10 s (method timeout)	fails, 10 s (method timeout)
The pending ack replay drops individual acks	individualPendingAckReplayTest	fails, 12.3 s	fails, 1.0 s
The pending ack replay drops cumulative acks	cumulativePendingAckReplayTest	fails, 3.7 s	fails, 1.1 s
The pending ack handle retries a non-recoverable init failure	testBuildConsumerEncounterPendingAckInitFailure [0], [2], [4]	fails, 0.6-0.7 s	fails, 0.6-0.7 s
The inactive subscription check ignores the last active time	testCheckInactiveSubscriptionWhenNoMessageToAck	fails, 15.1 s	fails, 12.7 s
A recovered cursor is idle since its persisted last active time (what #22794 fixed)	testCheckInactiveSubscriptionWhenNoMessageToAck	fails, 15.1 s	fails, 12.8 s

testTransactionAckMessageList catches its mutation with the check before the changed one. A committed
transaction whose acks never reach the cursor but stay pending passes both versions of the ack tests: the dispatcher
doesn't deliver pending-acked positions. An expiration check without its connected-consumer guard also passes both versions of
testCheckInactiveSubscriptionWhenNoMessageToAck (run: 7.9 s and 3.1 s): the subscription's delete refuses while
a consumer is connected.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

[codelipenghui]

Reviewed 878f8b3. No actionable findings identified within the reviewed scope.

Local validation passed with retries disabled: 25 focused test invocations across the five changed classes, followed by all 37 tests in TransactionTest.

The existing Broker Group 1 CI failure remains unexplained; the local passing runs do not establish that it is unrelated to this change.