Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cli/cmd/mcp_tools_runs_activity.go
Original file line number Diff line number Diff line change
Expand Up @@ -490,7 +490,7 @@ var mcpCaveatsRunsActivity = []mcpCaveat{
mcpJava + "disruption/DisruptionReportEntity.java:39-47",
mcpAnchoredRef(mcpJava+"disruption/DisruptionAnalysisResource.java:118-192",
`@Path("/compound")`, `"results", outcome.results()`),
mcpAnchoredRef(mcpJava+"resilience/ResilienceResource.java:107-137",
mcpAnchoredRef(mcpJava+"resilience/ResilienceResource.java:109-139",
"StreamingOutput executeWithKeepAlive(", "objectMapper.writeValue(os, payload);",
`"Failed to execute resilience test"`),
},
Expand Down
2 changes: 1 addition & 1 deletion docs/book/11-api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -948,7 +948,7 @@ Start a test run and inject one fault while it runs. The call is long-running: w
}
```

Optional fields: `probes` (steady-state probe definitions) and `maxRecoveryWaitSec` (default 120).
Optional fields: `probes` (steady-state probe definitions) and `maxRecoveryWaitSec` (default 120). A probe's `intervalSec`, the pause between two rounds of a `Continuous` probe, defaults to 10; under 1 is refused with a `400` that names the probe, as `probes[1].intervalSec`.

The workload has to be running when the fault lands. At 500 records per second the 180,000 records take 360 s, while the fault is triggered after `steadyStateSec` (30 s), lasts `chaosDurationSec` (30 s), and the run then waits up to `maxRecoveryWaitSec` for recovery. Without `throughput` the producer runs unthrottled and can finish before the fault. A run that has ended when `steadyStateSec` is up gets no fault, and the report is `ERROR`; one that ends after that, before the fault goes in, leaves both summaries describing a run the fault never touched. `testRequest` is the body of [POST /api/tests](#post-apitests), so the same fields apply: `throughput` is the rate limit, and LOAD runs one producer and one consumer, so the spec sets no producer count. `disruptionType` picks the fault: on the default `litmus-crd` chaos provider `POD_KILL` runs the LitmusChaos `pod-delete` experiment, and the outcome reports that name, while `experimentName` only names the ChaosEngine. Without `disruptionType`, Litmus runs the experiment that `experimentName` names, and the `kates-chaos` chart installs none called `kafka-pod-kill`. The selector adds `strimzi.io/broker-role=true` because `strimzi.io/component-type=kafka` alone also matches the KRaft controllers, and the random pick could then kill a controller instead of a broker.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package com.bmscomp.kates.resilience;

import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
Expand All @@ -21,6 +22,7 @@
import com.bmscomp.kates.audit.Audited;
import com.bmscomp.kates.chaos.FaultLimits;
import com.bmscomp.kates.chaos.FaultSpec;
import com.bmscomp.kates.chaos.ProbeSpec;
import com.bmscomp.kates.disruption.DisruptionConcurrencyGuard;
import com.bmscomp.kates.disruption.DisruptionPlan;
import com.bmscomp.kates.disruption.DisruptionSafetyGuard;
Expand Down Expand Up @@ -215,8 +217,8 @@ private Optional<Response> unsafeOrBusy(FaultSpec chaosSpec) {
responseCode = "400",
description = "Invalid request, including a testRequest with no type, a testRequest spec value outside"
+ " its limits, a testRequest spec field the test type or backend cannot apply, or a chaosSpec"
+ " parameter outside the chaos limits, where fieldErrors names each field; or a chaosSpec the"
+ " safety guard refuses, such as one that would hit every broker")
+ " parameter outside the chaos limits or a probe intervalSec under 1, where fieldErrors names"
+ " each field; or a chaosSpec the safety guard refuses, such as one that would hit every broker")
@APIResponse(
responseCode = "409",
description = "Another disruption is running against this cluster; no test is started")
Expand All @@ -240,6 +242,10 @@ public Response executeResilienceTest(ResilienceTestRequest request) {
if (outsideLimits.isPresent()) {
return outsideLimits.get();
}
Optional<Response> probesTooFast = probesTooFast(request.getProbes());
if (probesTooFast.isPresent()) {
return probesTooFast.get();
}
Optional<Response> unsafeOrBusy = unsafeOrBusy(request.getChaosSpec());
if (unsafeOrBusy.isPresent()) {
return unsafeOrBusy.get();
Expand All @@ -249,6 +255,32 @@ public Response executeResilienceTest(ResilienceTestRequest request) {
return Response.ok(stream).build();
}

/**
* A 400 naming each probe with an intervalSec under 1, or empty when there
* is none. The probe monitor sleeps that long between rounds, so at 0 it
* ran exec calls into the broker pods back to back for the whole fault.
*/
private static Optional<Response> probesTooFast(List<ProbeSpec> probes) {
if (probes == null) {
return Optional.empty();
}
Map<String, String> found = new LinkedHashMap<>();
for (int i = 0; i < probes.size(); i++) {
if (probes.get(i) != null && probes.get(i).intervalSec() < 1) {
found.put("probes[" + i + "].intervalSec", "must be at least 1");
}
}
if (found.isEmpty()) {
return Optional.empty();
}
String message = found.entrySet().stream()
.map(e -> e.getKey() + ": " + e.getValue())
.collect(Collectors.joining("; "));
return Optional.of(Response.status(400)
.entity(ApiError.validationFailed(message, found))
.build());
}

@GET
@Path("/scenarios")
@Operation(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,27 @@ void aSpecValueOutsideItsLimitsIsRefusedByName() {
.body("message", endsWith("; spec.topic: topic must be a legal Kafka topic name"));
}

/**
* The probe monitor sleeps a probe's intervalSec between rounds, so at 0
* it ran exec calls into the broker pods back to back for the whole fault.
*/
@Test
void aProbeIntervalUnderOneSecondIsRefusedByItsIndex() {
given().contentType("application/json")
.body("{\"testRequest\":{\"type\":\"LOAD\"},\"chaosSpec\":{\"experimentName\":\"test\"},"
+ "\"probes\":[{\"name\":\"ok\",\"mode\":\"Continuous\"},"
+ "{\"name\":\"spin\",\"mode\":\"Continuous\",\"intervalSec\":0}]}")
.when()
.post("/api/resilience")
.then()
.statusCode(400)
.body("error", is("Validation Failed"))
.body("fieldErrors", hasKey("probes[1].intervalSec"))
.body("fieldErrors", not(hasKey("probes[0].intervalSec")))
.body("message", is("probes[1].intervalSec: must be at least 1"));
verify(orchestrator, never()).execute(any());
}

/**
* The limits are those of POST /api/tests, and so is the fieldErrors that
* names a value outside them. As there, they are checked before the
Expand Down
Loading