Repository navigation
[improve][ci] Enforce nightly chaos fault regressions - #27044
codelipenghui wants to merge 1 commit into
Conversation
Run bounded nightly fault scenarios through the required CI gate, validate real injection on framework changes, retain failure artifacts, and restore a bounded deduplication recovery regression outside quarantine. Assisted-by: DeepSeek Harness (gpt-6.1-sol)
lhotari
left a comment
lhotari
left a comment
There was a problem hiding this comment.
Great idea! I added some comments, mainly about adding a new workflow file such as pulsar-performance.yaml for the workflow changes.
| performance_network: >- | ||
| ${{ steps.changes.outputs.performance_network == 'true' | ||
| || github.event_name == 'workflow_dispatch' | ||
| || (github.event_name == 'schedule' && github.event.schedule == '0 6 * * *') }} | ||
| performance_faults: >- | ||
| ${{ github.event_name == 'workflow_dispatch' | ||
| || (github.event_name == 'schedule' && github.event.schedule == '0 6 * * *') }} |
There was a problem hiding this comment.
I think that it would be better to have a separate workflow file for these. It would allow adding more scenarios to be triggered manually (workflow_dispatch) with a selection or with scheduling.
|
|
||
| // TODO: Test disabled since it results in a OOME | ||
| @Test(timeOut = 300000, groups = "quarantine", enabled = false) | ||
| public void testClientDeduplicationCorrectnessWithFailure() throws Exception { |
There was a problem hiding this comment.
would it make sense to split this change to a separate PR?
| * Uses a private real BookKeeper ensemble: stopping every bookie must not affect the shared test cluster. | ||
| */ | ||
| @Test(groups = "broker-api") | ||
| public class ClientDeduplicationRecoveryTest extends BkEnsemblesTestBase { |
| with: | ||
| action: wait | ||
|
|
||
| performance-network-check: |
There was a problem hiding this comment.
please make changes to a separate workflow file such as pulsar-performance.yaml etc.
The workflow can be simplified since with Gradle, there's no need for separate build steps for the docker container and so on. A single job with multiple sequential steps could handle the tests to begin with. Later on it could be parallelized if the build time grows too much.
Since Java 25 is the default for Pulsar deployments, it's worth testing with Java 25 in the separate workflow. There could be a selection to choose Java 21 in the manual trigger (workflow_dispatch). |
Motivation
Existing performance-framework fault scenarios were not run in CI, and the real network-injection checks were excluded from ordinary launcher tests. The deduplication correctness test covering BookKeeper failures was also disabled because its unbounded workload could exhaust memory, leaving that regression in quarantine.
Make a small set of existing fault scenarios fail-on-error CI checks, and restore bounded deduplication failure coverage in the normal broker client API test group.
Modifications
networkCheckon relevant launcher, loader, scenario, test-image, and CI wiring changes, as well as nightly/manual runs.ClientDeduplicationRecoveryTestin the normalbroker-apigroup. It uses a private real BookKeeper ensemble, at most eight outstanding sends, 33 total send attempts, finite timeouts, an observed storage write failure, duplicate replay after recovery, and exact persisted sequence/completion assertions. Leave the unrelated legacy failure test quarantined.The performance fault suite retains the framework's at-least-once delivery contract: consumer duplicates alone do not fail a run. It does not establish performance baselines or a recovery-time SLA.
Verifying this change
This change added tests and can be verified as follows:
./gradlew quickCheck --max-workers=2(license headers and source style across modules).common,tools,launcher,report-tool, andmetrics, using--tests '*Test', one test fork, and-PtestRetryCount=0.ClientDeduplicationRecoveryTestin three separate runs with retries disabled, including two runs explicitly selecting-PtestGroups=broker-api. Each run executed the regression without skips.git diff --check.:tests:performance:launcher:networkCheckstopped at the existing capability probe withss -K destroyed no TCP socketon Docker Desktop (LinuxKit 7.0.14). A separate disposable network namespace reproduced zero destroyed sockets even with established connections. The capability failure remains fail-closed; no checks were weakened or skipped to obtain a pass.Does this pull request potentially affect one of the following parts: