fix(oracle): degrade the worker group on network outage faults instead of halting it

.agents/skills/verify-backwave/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: verify-backwave
-description: Launch the real BackWave Sample.Api (the sample host with the BackWave dashboard at /backwave) on a private port with its own SQLite file or Postgres container, create real jobs through the sample's HTTP endpoints, drive the dashboard in a private headless Chrome, capture screenshot, accessibility, API, and database evidence, and clean up. Use to prove a BackWave change works end to end, to reproduce a bug that a user or operator would see, or whenever you need to see or screenshot the dashboard running.
+description: Launch the real BackWave Sample.Api (the sample host with the BackWave dashboard at /backwave) on a private port with its own SQLite file, Postgres container, or Oracle container, create real jobs through the sample's HTTP endpoints, drive the dashboard in a private headless Chrome, capture screenshot, accessibility, API, and database evidence, and clean up. Use to prove a BackWave change works end to end, to reproduce a bug that a user or operator would see, or whenever you need to see or screenshot the dashboard running.
 ---
 
 # Verify BackWave
@@ -32,6 +32,7 @@ Each run owns these resources. All of them use the same run id (`YYYYMMDD-HHMMSS
 | HTTP port | `http://127.0.0.1:<port>` | A free port from the OS. `bwv` never uses the default `5283` from `appsettings.json`. |
 | Scratch directory | `${TMPDIR}/bw-verify-<run>/` | Holds the app copy and the SQLite file `backwave.db`. |
 | Postgres (only with `--store postgres`) | container `backwave-verify-<run>`, label `backwave.verify.run=<run>` | `postgres:17-alpine` on a free loopback port. It never uses the compose containers on `5398` or `14331`. |
+| Oracle (only with `--store oracle`) | container `backwave-verify-<run>`, label `backwave.verify.run=<run>` | `gvenzl/oracle-free:23-slim` on a fixed free loopback port. It never uses the compose container on `15210` or the sample compose container on `15211`. |
 | Browser | `CHROME_DEVTOOLS_AXI_SESSION=bw-verify-<run>` | A private bridge and an isolated headless Chrome profile. `bwv` never uses the `default` session. |
 | Evidence | `.verify-evidence/<run>/` at the worktree root | Gitignored. Cleanup never deletes it. |
 
@@ -44,14 +45,15 @@ Parallel runs in one worktree share `bin/` and `obj/`. For this reason, `bwv lau
 ```bash
 bwv launch                    # SQLite co-resident store (the default)
 bwv launch --store postgres   # a private Postgres container
+bwv launch --store oracle     # a private Oracle container (first boot 30-90 s)
 bwv launch --store inmemory   # no database, fastest start
 ```
 
 This command does these steps:
 
 1. Records the run in `.verify-evidence/<run>/run.env` and makes it the current run. It records each resource name before it creates the resource.
 2. Runs `dotnet build` on `samples/BackWave.Sample.Api` under the build lock, and copies `bin/Debug/net10.0` to the scratch directory.
-3. For `--store postgres`, starts the run's container and waits until Postgres accepts TCP connections.
+3. For `--store postgres`, starts the run's container and waits until Postgres accepts TCP connections. For `--store oracle`, starts the run's container and waits until its `healthcheck.sh` passes.
 4. Starts the app copy with `--urls http://127.0.0.1:<port>` and the store flags. It sets `DOTNET_hostBuilder__reloadConfigOnChange=false` (see [Gotchas](#gotchas)).
 5. Waits until `/backwave` returns 200, then makes sure that `app.log` names the expected store.
 
@@ -75,9 +77,9 @@ This is a read-only check. Run it after launch, before every proof, and first wh
 - The recorded PID still runs this run's `<scratch>/app/BackWave.Sample.Api.dll`.
 - That PID holds the listening socket on the run's port.
 - `/backwave` serves the page with the title `Overview · BackWave`.
-- `app.log` names the store: `SqliteCoResident`, `Postgres`, or `InMemory`.
+- `app.log` names the store: `SqliteCoResident`, `Postgres`, `Oracle`, or `InMemory`.
 - The build matches the worktree. The fingerprint covers the committed files, the uncommitted edits, and the untracked files. It ignores docs, `*.md`, `.agents`, and `.claude`. If you edit product code after the launch, this check fails.
-- The store exists: the SQLite file, or the Postgres container with the run's label.
+- The store exists: the SQLite file, or the Postgres or Oracle container with the run's label.
 - The state of the run's browser session.
 
 The exit code is not zero when a check fails. Do not drive a run that fails doctor. Correct the cause, or run `bwv cleanup` and then `bwv launch`.
@@ -166,7 +168,7 @@ Run cleanup after every failed iteration too. `bwv runs` lists every run in this
 
 | Helper | Invocation |
 |---|---|
-| `scripts/bwv` | `bwv launch [--store sqlite\|postgres\|inmemory] \| doctor \| url [path] \| api <METHOD> <path> [name] [json] \| wait-job <jobId> <State> [timeout-s] \| browser <args...> \| shot <name> \| sql <name> <query> \| cleanup \| runs`. Run `bwv` with no arguments to see the usage. |
+| `scripts/bwv` | `bwv launch [--store sqlite\|postgres\|oracle\|inmemory] \| doctor \| url [path] \| api <METHOD> <path> [name] [json] \| wait-job <jobId> <State> [timeout-s] \| browser <args...> \| shot <name> \| sql <name> <query> \| cleanup \| runs`. Run `bwv` with no arguments to see the usage. |
 | `chrome-devtools-axi` | Through `bwv browser` only, so that the run's session is always set. Run `chrome-devtools-axi --help` for the full command list. |
 | Swagger UI | `bwv url /swagger` prints the URL. The page lists every sample endpoint and its parameters. |
 
@@ -181,4 +183,6 @@ Run cleanup after every failed iteration too. `bwv runs` lists every run in this
 - **The Overview counts only queues that hold jobs.** The Queues page also lists `limited` (concurrency limit 1, set at startup by the actor `startup`).
 - **SQL Server is not wired into `bwv`.** The sample supports `--BackWave:Store=SqlServer`, but `bwv` does not start a private SQL Server container. Do not point a run at the compose container on port `14331`: other test runs share it.
 - **`sqlite3 -readonly` fails on this WAL database.** `bwv sql` uses `PRAGMA query_only=1` instead.
+- **To reproduce a store outage,** stop and start the run's own container: `docker stop -t 2 <CONTAINER>`, then `docker start <CONTAINER>`. `CONTAINER` is in `run.env`. The Oracle port is fixed, so the app finds the database again. On Oracle, `SHUTDOWN IMMEDIATE` and `STARTUP` through `docker exec -i <CONTAINER> sqlplus -S / as sysdba` give the restart codes (`ORA-01089`, `ORA-01109`). A host-run app cannot lose DNS, so `ORA-12545` needs the app in a container on a private Docker network.
+- **The Oracle store has no business table.** `POST /tx` answers 409, and `bwv sql` queries the `backwave` schema, for example `SELECT RAWTOHEX(job_id), state, attempt FROM jobs`.
 - **macOS has no `timeout` command.** Use `bwv wait-job` or a loop with `sleep` to wait.

.agents/skills/verify-backwave/features/README.md

@@ -6,7 +6,7 @@ This directory is the maintained source for verifying the user-facing behavior o
 
 - `bwv launch` printed `READY`, and `bwv doctor` passes every check on the run that you drive.
 - The run's Sample.Api starts with no jobs. Each recipe creates its own jobs through the sample's HTTP endpoints with `bwv api`.
-- The run's store is its own SQLite file (the default) or its own Postgres container (`--store postgres`). The queues are `critical`, `bulk`, `low`, and `limited` (concurrency limit 1).
+- The run's store is its own SQLite file (the default), its own Postgres container (`--store postgres`), or its own Oracle container (`--store oracle`). The queues are `critical`, `bulk`, `low`, and `limited` (concurrency limit 1).
 - The operator actor on the dashboard and on `/ops/*` is `sample-operator`. The startup limit on `limited` has the actor `startup`.
 - Never drive an app, a container, or a browser session that this run did not start.
 

.agents/skills/verify-backwave/scripts/bwv

@@ -6,12 +6,12 @@
 #   - a scratch dir          ${TMPDIR}/bw-verify-<run>/  (app copy + SQLite file)
 #   - a free port            http://127.0.0.1:<port>
 #   - a browser session      CHROME_DEVTOOLS_AXI_SESSION=bw-verify-<run>
-#   - (postgres only)        container backwave-verify-<run>, label backwave.verify.run=<run>
+#   - (postgres/oracle only) container backwave-verify-<run>, label backwave.verify.run=<run>
 #   - an evidence dir        .verify-evidence/<run>/  (gitignored, survives cleanup)
 # Nothing shared is reused or killed. See ../SKILL.md for the full contract.
 #
 # Usage:
-#   bwv launch [--store sqlite|postgres|inmemory]  new run: build, copy, start, wait for READY
+#   bwv launch [--store sqlite|postgres|oracle|inmemory]  new run: build, copy, start, wait for READY
 #   bwv doctor                         read-only health check of the current run
 #   bwv url [path]                     print the run's base URL (plus an optional path)
 #   bwv api <METHOD> <path> [name] [json-body]   call the Sample API -> <run>/api/<name>.json
@@ -33,6 +33,7 @@ PROJECT="$ROOT/samples/BackWave.Sample.Api/BackWave.Sample.Api.csproj"
 BUILD_OUT="$ROOT/samples/BackWave.Sample.Api/bin/Debug/net10.0"
 DLL="BackWave.Sample.Api.dll"
 PG_IMAGE="postgres:17-alpine"
+ORA_IMAGE="gvenzl/oracle-free:23-slim"
 STATES=(Scheduled AwaitingParent Leased Succeeded Cancelled DeadLettered Quarantined)
 
 log()  { printf "\033[1;34mbwv:\033[0m %s\n" "$*" >&2; }
@@ -141,14 +142,14 @@ cmd_launch() {
     local store=sqlite
     while (( $# )); do
         case "$1" in
-            --store) store="${2:?--store needs sqlite|postgres|inmemory}"; shift 2 ;;
+            --store) store="${2:?--store needs sqlite|postgres|oracle|inmemory}"; shift 2 ;;
             *) die "unknown launch flag '$1'" ;;
         esac
     done
-    [[ "$store" =~ ^(sqlite|postgres|inmemory)$ ]] || die "store must be sqlite, postgres or inmemory"
+    [[ "$store" =~ ^(sqlite|postgres|oracle|inmemory)$ ]] || die "store must be sqlite, postgres, oracle or inmemory"
     command -v dotnet >/dev/null || die "dotnet is not on PATH"
     command -v chrome-devtools-axi >/dev/null || die "chrome-devtools-axi is not on PATH"
-    if [[ "$store" == postgres ]]; then docker info >/dev/null 2>&1 || die "Docker is not running"; fi
+    if [[ "$store" =~ ^(postgres|oracle)$ ]]; then docker info >/dev/null 2>&1 || die "Docker is not running"; fi
 
     RUN="$(date +%Y%m%d-%H%M%S)-$$"
     RUN_DIR="$EVIDENCE_ROOT/$RUN"
@@ -170,6 +171,7 @@ cmd_launch() {
     set_var BROWSER_STARTED 0
     set_var CONTAINER ""
     set_var PG_PORT ""
+    set_var ORA_PORT ""
     set_var SQLITE_DB ""
     log "run $RUN ($store) - evidence at $RUN_DIR"
     mkdir -p "$SCRATCH"
@@ -203,6 +205,26 @@ cmd_launch() {
             store_args=(--BackWave:Store=Postgres
                 "--BackWave:Postgres:ConnectionString=Host=127.0.0.1;Port=$PG_PORT;Username=backwave;Password=backwave;Database=backwave_sample")
             set_var STORE_NAME Postgres ;;
+        oracle)
+            set_var CONTAINER "backwave-verify-$RUN"
+            # A fixed host port, so that the run can stop and start the container (a store outage)
+            # and the app finds the database again on the same port.
+            set_var ORA_PORT "$(free_port)"
+            log "starting $ORA_IMAGE as $CONTAINER on 127.0.0.1:$ORA_PORT (first boot takes 30-90 s)"
+            # Oracle needs far more shared memory than the 64 MB Docker default.
+            docker run -d --name "$CONTAINER" --label "backwave.verify.run=$RUN" --shm-size 2g \
+                -e ORACLE_PASSWORD=backwave -e APP_USER=backwave -e APP_USER_PASSWORD=backwave \
+                -p "127.0.0.1:$ORA_PORT:1521" "$ORA_IMAGE" > /dev/null \
+                || die "docker run failed - run 'bwv cleanup'"
+            local i
+            for i in $(seq 1 180); do
+                docker exec "$CONTAINER" healthcheck.sh >/dev/null 2>&1 && break
+                (( i == 180 )) && die "Oracle was not ready in 180 s - run 'bwv cleanup'"
+                sleep 1
+            done
+            store_args=(--BackWave:Store=Oracle
+                "--BackWave:Oracle:ConnectionString=User Id=backwave;Password=backwave;Data Source=127.0.0.1:$ORA_PORT/FREEPDB1")
+            set_var STORE_NAME Oracle ;;
     esac
 
     log "starting Sample.Api on $(base) (log: $RUN_DIR/app.log)"
@@ -264,9 +286,9 @@ cmd_doctor() {
     case "$STORE" in
         sqlite)
             [[ -f "$SQLITE_DB" ]] && ok "SQLite file $SQLITE_DB" || bad "SQLite file $SQLITE_DB missing" ;;
-        postgres)
+        postgres|oracle)
             if container_is_ours && [[ "$(docker inspect -f '{{.State.Running}}' "$CONTAINER")" == true ]]; then
-                ok "container $CONTAINER running on 127.0.0.1:$PG_PORT (label run=$RUN)"
+                ok "container $CONTAINER running on 127.0.0.1:${PG_PORT:-$ORA_PORT} (label run=$RUN)"
             else
                 bad "container $CONTAINER is not running with label backwave.verify.run=$RUN"
             fi ;;
@@ -368,6 +390,8 @@ cmd_sql() {
           sqlite) sqlite3 -header -column -cmd '.timeout 5000' "$SQLITE_DB" "PRAGMA query_only=1; $query" ;;
           postgres) docker exec -e PGOPTIONS='-c default_transaction_read_only=on' "$CONTAINER" \
                         psql -U backwave -d backwave_sample -v ON_ERROR_STOP=1 -c "$query" ;;
+          oracle) printf 'SET DEFINE OFF\nSET MARKUP CSV ON\nWHENEVER SQLERROR EXIT FAILURE\nSET TRANSACTION READ ONLY;\n%s;\nEXIT\n' "${query%;}" \
+                      | docker exec -i "$CONTAINER" sqlplus -S -L backwave/backwave@localhost/FREEPDB1 ;;
       esac
     } > "$out" 2>&1 || { cat "$out"; die "query failed (saved db/$name.txt)"; }
     cat "$out"

samples/BackWave.Sample.Api/BackWave.Sample.Api.csproj

@@ -31,6 +31,7 @@
     <ProjectReference Include="..\..\src\BackWave.Pro.Mcp\BackWave.Pro.Mcp.csproj" />
     <ProjectReference Include="..\..\src\BackWave.Postgres\BackWave.Postgres.csproj" />
     <ProjectReference Include="..\..\src\BackWave.SqlServer\BackWave.SqlServer.csproj" />
+    <ProjectReference Include="..\..\src\BackWave.Oracle\BackWave.Oracle.csproj" />
     <ProjectReference Include="..\..\src\BackWave.Sqlite\BackWave.Sqlite.csproj" />
     <ProjectReference Include="..\..\src\BackWave.EntityFrameworkCore\BackWave.EntityFrameworkCore.csproj" />
     <ProjectReference Include="..\..\src\BackWave.SourceGenerators\BackWave.SourceGenerators.csproj"

samples/BackWave.Sample.Api/Program.cs

@@ -23,7 +23,7 @@
 var builder = WebApplication.CreateBuilder(args);
 
 // ── Storage selection ────────────────────────────────────────────────────────
-// BackWave:Store = InMemory (default) | Postgres | SqlServer. In-Memory is F5-and-go with
+// BackWave:Store = InMemory (default) | Postgres | SqlServer | Oracle. In-Memory is F5-and-go with
 // zero infra; the relational stores need the sample's own `docker compose up -d` first. Each
 // kind carries its own store/DbContext/schema behavior (see StoreKind.cs).
 var store = StoreKind.FromConfiguration(builder.Configuration);

samples/BackWave.Sample.Api/README.md

@@ -27,7 +27,7 @@ Then open:
 ## Switching stores
 
 Storage is chosen by the `BackWave:Store` config key: `InMemory` (default), `Sqlite`,
-`SqliteDedicated`, `Postgres`, or `SqlServer`.
+`SqliteDedicated`, `Postgres`, `SqlServer`, or `Oracle`.
 
 | Store             | Infra needed           | Transactional Enqueue (`POST /tx`)         |
 | ----------------- | ---------------------- | ------------------------------------------ |
@@ -36,6 +36,7 @@ Storage is chosen by the `BackWave:Store` config key: `InMemory` (default), `Sql
 | `SqliteDedicated` | none — a local file    | returns **409** (BackWave on its own file) |
 | `Postgres`        | `docker compose up -d` | ✅ commits/rolls back atomically           |
 | `SqlServer`       | `docker compose up -d` | ✅ commits/rolls back atomically           |
+| `Oracle`          | `docker compose up -d` | returns **409** (no business table)        |
 
 The **SQLite Embedded Adapter** needs no server — just a local `.db` file (written under the content
 root). Two deployments demonstrate the spectrum:
@@ -65,10 +66,15 @@ BackWave__Store=Postgres dotnet run
 # SQL Server (amd64 image; slow under Rosetta on Apple Silicon)
 docker compose up -d sqlserver
 BackWave__Store=SqlServer dotnet run
+
+# Oracle (no business table, so POST /tx returns 409)
+docker compose up -d oracle
+BackWave__Store=Oracle dotnet run
 ```
 
 Connection strings default to the compose values and can be overridden via
-`BackWave:Postgres:ConnectionString` / `BackWave:SqlServer:ConnectionString`; the SQLite file paths
+`BackWave:Postgres:ConnectionString` / `BackWave:SqlServer:ConnectionString` /
+`BackWave:Oracle:ConnectionString`; the SQLite file paths
 via `BackWave:Sqlite:DataSource` (and `BackWave:Sqlite:BusinessDataSource` for the dedicated business
 file) — see `appsettings.json`.
 

samples/BackWave.Sample.Api/StoreKind.cs

@@ -1,3 +1,4 @@
+using BackWave.Oracle;
 using BackWave.Postgres;
 using BackWave.Sqlite;
 using BackWave.SqlServer;
@@ -20,6 +21,7 @@ internal abstract class StoreKind
     public static readonly StoreKind InMemory = new InMemoryStore();
     public static readonly StoreKind Postgres = new PostgresStore();
     public static readonly StoreKind SqlServer = new SqlServerStore();
+    public static readonly StoreKind Oracle = new OracleStore();
 
     /// <summary>The SQLite Embedded Adapter, co-resident: BackWave's tables live in the app's OWN file.</summary>
     public static readonly StoreKind Sqlite = new SqliteCoResidentStore();
@@ -33,6 +35,7 @@ public static StoreKind FromConfiguration(IConfiguration configuration)
         var configured = configuration["BackWave:Store"] ?? nameof(InMemory);
         return configured.Equals(nameof(Postgres), StringComparison.OrdinalIgnoreCase) ? Postgres
             : configured.Equals(nameof(SqlServer), StringComparison.OrdinalIgnoreCase) ? SqlServer
+            : configured.Equals(nameof(Oracle), StringComparison.OrdinalIgnoreCase) ? Oracle
             : configured.Equals(nameof(SqliteDedicated), StringComparison.OrdinalIgnoreCase) ? SqliteDedicated
             : configured.Equals(nameof(Sqlite), StringComparison.OrdinalIgnoreCase) ? Sqlite
             : InMemory;
@@ -155,6 +158,31 @@ private static string ConnectionString(IConfiguration configuration)
             => configuration["BackWave:SqlServer:ConnectionString"] ?? DefaultConnectionString;
     }
 
+    /// <summary>
+    /// The Oracle adapter. The sample registers no EF provider for Oracle, so it has no business table and
+    /// forgoes Transactional Enqueue: <c>POST /tx</c> answers 409.
+    /// </summary>
+    private sealed class OracleStore : StoreKind
+    {
+        private const string DefaultConnectionString =
+            "User Id=backwave;Password=backwave;Data Source=localhost:15211/FREEPDB1";
+
+        public override string Name => "Oracle";
+        public override bool SupportsTransactionalEnqueue => false;
+
+        public override IJobStore CreateStore(IConfiguration configuration, ILoggerFactory loggerFactory)
+            => new OracleJobStore(new OracleStoreOptions
+            {
+                ConnectionString = configuration["BackWave:Oracle:ConnectionString"] ?? DefaultConnectionString,
+                AutoMigrate = true,
+                LoggerFactory = loggerFactory, // opts in to the migration log; null (the default) is silent
+            });
+
+        public override void AddAdapterTracing(TracerProviderBuilder tracing) => tracing.AddBackWaveOracleInstrumentation();
+
+        public override void AddAdapterMetrics(MeterProviderBuilder metrics) => metrics.AddBackWaveOracleInstrumentation();
+    }
+
     /// <summary>
     /// The SQLite <b>Embedded Adapter</b> (no server, no Docker — just a local file). Two deployments:
     /// <see cref="SqliteCoResidentStore"/> puts BackWave's tables in the application's OWN database file

samples/BackWave.Sample.Api/docker-compose.yml

@@ -30,3 +30,19 @@ services:
       interval: 5s
       timeout: 5s
       retries: 60
+
+  oracle:
+    image: gvenzl/oracle-free:23-slim
+    # Oracle needs far more shared memory than Docker's 64 MB default.
+    shm_size: 2gb
+    environment:
+      ORACLE_PASSWORD: backwave
+      APP_USER: backwave
+      APP_USER_PASSWORD: backwave
+    ports:
+      - "15211:1521"
+    healthcheck:
+      test: ["CMD", "healthcheck.sh"]
+      interval: 5s
+      timeout: 5s
+      retries: 60