diff --git a/.agents/skills/verify-backwave/SKILL.md b/.agents/skills/verify-backwave/SKILL.md index 0fd02ea..67b07dc 100644 --- a/.agents/skills/verify-backwave/SKILL.md +++ b/.agents/skills/verify-backwave/SKILL.md @@ -1,6 +1,6 @@ --- name: verify-backwave -description: Launch the real BackWave Sample.Api (the sample host with the BackWave dashboard at /backwave) on a private port with its own SQLite file or Postgres container, create real jobs through the sample's HTTP endpoints, drive the dashboard in a private headless Chrome, capture screenshot, accessibility, API, and database evidence, and clean up. Use to prove a BackWave change works end to end, to reproduce a bug that a user or operator would see, or whenever you need to see or screenshot the dashboard running. +description: Launch the real BackWave Sample.Api (the sample host with the BackWave dashboard at /backwave) on a private port with its own SQLite file, Postgres container, or Oracle container, create real jobs through the sample's HTTP endpoints, drive the dashboard in a private headless Chrome, capture screenshot, accessibility, API, and database evidence, and clean up. Use to prove a BackWave change works end to end, to reproduce a bug that a user or operator would see, or whenever you need to see or screenshot the dashboard running. --- # Verify BackWave @@ -32,6 +32,7 @@ Each run owns these resources. All of them use the same run id (`YYYYMMDD-HHMMSS | HTTP port | `http://127.0.0.1:` | A free port from the OS. `bwv` never uses the default `5283` from `appsettings.json`. | | Scratch directory | `${TMPDIR}/bw-verify-/` | Holds the app copy and the SQLite file `backwave.db`. | | Postgres (only with `--store postgres`) | container `backwave-verify-`, label `backwave.verify.run=` | `postgres:17-alpine` on a free loopback port. It never uses the compose containers on `5398` or `14331`. | +| Oracle (only with `--store oracle`) | container `backwave-verify-`, label `backwave.verify.run=` | `gvenzl/oracle-free:23-slim` on a fixed free loopback port. It never uses the compose container on `15210` or the sample compose container on `15211`. | | Browser | `CHROME_DEVTOOLS_AXI_SESSION=bw-verify-` | A private bridge and an isolated headless Chrome profile. `bwv` never uses the `default` session. | | Evidence | `.verify-evidence//` at the worktree root | Gitignored. Cleanup never deletes it. | @@ -44,6 +45,7 @@ Parallel runs in one worktree share `bin/` and `obj/`. For this reason, `bwv lau ```bash bwv launch # SQLite co-resident store (the default) bwv launch --store postgres # a private Postgres container +bwv launch --store oracle # a private Oracle container (first boot 30-90 s) bwv launch --store inmemory # no database, fastest start ``` @@ -51,7 +53,7 @@ This command does these steps: 1. Records the run in `.verify-evidence//run.env` and makes it the current run. It records each resource name before it creates the resource. 2. Runs `dotnet build` on `samples/BackWave.Sample.Api` under the build lock, and copies `bin/Debug/net10.0` to the scratch directory. -3. For `--store postgres`, starts the run's container and waits until Postgres accepts TCP connections. +3. For `--store postgres`, starts the run's container and waits until Postgres accepts TCP connections. For `--store oracle`, starts the run's container and waits until its `healthcheck.sh` passes. 4. Starts the app copy with `--urls http://127.0.0.1:` and the store flags. It sets `DOTNET_hostBuilder__reloadConfigOnChange=false` (see [Gotchas](#gotchas)). 5. Waits until `/backwave` returns 200, then makes sure that `app.log` names the expected store. @@ -75,9 +77,9 @@ This is a read-only check. Run it after launch, before every proof, and first wh - The recorded PID still runs this run's `/app/BackWave.Sample.Api.dll`. - That PID holds the listening socket on the run's port. - `/backwave` serves the page with the title `Overview · BackWave`. -- `app.log` names the store: `SqliteCoResident`, `Postgres`, or `InMemory`. +- `app.log` names the store: `SqliteCoResident`, `Postgres`, `Oracle`, or `InMemory`. - The build matches the worktree. The fingerprint covers the committed files, the uncommitted edits, and the untracked files. It ignores docs, `*.md`, `.agents`, and `.claude`. If you edit product code after the launch, this check fails. -- The store exists: the SQLite file, or the Postgres container with the run's label. +- The store exists: the SQLite file, or the Postgres or Oracle container with the run's label. - The state of the run's browser session. The exit code is not zero when a check fails. Do not drive a run that fails doctor. Correct the cause, or run `bwv cleanup` and then `bwv launch`. @@ -166,7 +168,7 @@ Run cleanup after every failed iteration too. `bwv runs` lists every run in this | Helper | Invocation | |---|---| -| `scripts/bwv` | `bwv launch [--store sqlite\|postgres\|inmemory] \| doctor \| url [path] \| api [name] [json] \| wait-job [timeout-s] \| browser \| shot \| sql \| cleanup \| runs`. Run `bwv` with no arguments to see the usage. | +| `scripts/bwv` | `bwv launch [--store sqlite\|postgres\|oracle\|inmemory] \| doctor \| url [path] \| api [name] [json] \| wait-job [timeout-s] \| browser \| shot \| sql \| cleanup \| runs`. Run `bwv` with no arguments to see the usage. | | `chrome-devtools-axi` | Through `bwv browser` only, so that the run's session is always set. Run `chrome-devtools-axi --help` for the full command list. | | Swagger UI | `bwv url /swagger` prints the URL. The page lists every sample endpoint and its parameters. | @@ -181,4 +183,6 @@ Run cleanup after every failed iteration too. `bwv runs` lists every run in this - **The Overview counts only queues that hold jobs.** The Queues page also lists `limited` (concurrency limit 1, set at startup by the actor `startup`). - **SQL Server is not wired into `bwv`.** The sample supports `--BackWave:Store=SqlServer`, but `bwv` does not start a private SQL Server container. Do not point a run at the compose container on port `14331`: other test runs share it. - **`sqlite3 -readonly` fails on this WAL database.** `bwv sql` uses `PRAGMA query_only=1` instead. +- **To reproduce a store outage,** stop and start the run's own container: `docker stop -t 2 `, then `docker start `. `CONTAINER` is in `run.env`. The Oracle port is fixed, so the app finds the database again. On Oracle, `SHUTDOWN IMMEDIATE` and `STARTUP` through `docker exec -i sqlplus -S / as sysdba` give the restart codes (`ORA-01089`, `ORA-01109`). A host-run app cannot lose DNS, so `ORA-12545` needs the app in a container on a private Docker network. +- **The Oracle store has no business table.** `POST /tx` answers 409, and `bwv sql` queries the `backwave` schema, for example `SELECT RAWTOHEX(job_id), state, attempt FROM jobs`. - **macOS has no `timeout` command.** Use `bwv wait-job` or a loop with `sleep` to wait. diff --git a/.agents/skills/verify-backwave/features/README.md b/.agents/skills/verify-backwave/features/README.md index 7de14ce..2da2f0a 100644 --- a/.agents/skills/verify-backwave/features/README.md +++ b/.agents/skills/verify-backwave/features/README.md @@ -6,7 +6,7 @@ This directory is the maintained source for verifying the user-facing behavior o - `bwv launch` printed `READY`, and `bwv doctor` passes every check on the run that you drive. - The run's Sample.Api starts with no jobs. Each recipe creates its own jobs through the sample's HTTP endpoints with `bwv api`. -- The run's store is its own SQLite file (the default) or its own Postgres container (`--store postgres`). The queues are `critical`, `bulk`, `low`, and `limited` (concurrency limit 1). +- The run's store is its own SQLite file (the default), its own Postgres container (`--store postgres`), or its own Oracle container (`--store oracle`). The queues are `critical`, `bulk`, `low`, and `limited` (concurrency limit 1). - The operator actor on the dashboard and on `/ops/*` is `sample-operator`. The startup limit on `limited` has the actor `startup`. - Never drive an app, a container, or a browser session that this run did not start. diff --git a/.agents/skills/verify-backwave/scripts/bwv b/.agents/skills/verify-backwave/scripts/bwv index 3793ef6..53d42fb 100755 --- a/.agents/skills/verify-backwave/scripts/bwv +++ b/.agents/skills/verify-backwave/scripts/bwv @@ -6,12 +6,12 @@ # - a scratch dir ${TMPDIR}/bw-verify-/ (app copy + SQLite file) # - a free port http://127.0.0.1: # - a browser session CHROME_DEVTOOLS_AXI_SESSION=bw-verify- -# - (postgres only) container backwave-verify-, label backwave.verify.run= +# - (postgres/oracle only) container backwave-verify-, label backwave.verify.run= # - an evidence dir .verify-evidence// (gitignored, survives cleanup) # Nothing shared is reused or killed. See ../SKILL.md for the full contract. # # Usage: -# bwv launch [--store sqlite|postgres|inmemory] new run: build, copy, start, wait for READY +# bwv launch [--store sqlite|postgres|oracle|inmemory] new run: build, copy, start, wait for READY # bwv doctor read-only health check of the current run # bwv url [path] print the run's base URL (plus an optional path) # bwv api [name] [json-body] call the Sample API -> /api/.json @@ -33,6 +33,7 @@ PROJECT="$ROOT/samples/BackWave.Sample.Api/BackWave.Sample.Api.csproj" BUILD_OUT="$ROOT/samples/BackWave.Sample.Api/bin/Debug/net10.0" DLL="BackWave.Sample.Api.dll" PG_IMAGE="postgres:17-alpine" +ORA_IMAGE="gvenzl/oracle-free:23-slim" STATES=(Scheduled AwaitingParent Leased Succeeded Cancelled DeadLettered Quarantined) log() { printf "\033[1;34mbwv:\033[0m %s\n" "$*" >&2; } @@ -141,14 +142,14 @@ cmd_launch() { local store=sqlite while (( $# )); do case "$1" in - --store) store="${2:?--store needs sqlite|postgres|inmemory}"; shift 2 ;; + --store) store="${2:?--store needs sqlite|postgres|oracle|inmemory}"; shift 2 ;; *) die "unknown launch flag '$1'" ;; esac done - [[ "$store" =~ ^(sqlite|postgres|inmemory)$ ]] || die "store must be sqlite, postgres or inmemory" + [[ "$store" =~ ^(sqlite|postgres|oracle|inmemory)$ ]] || die "store must be sqlite, postgres, oracle or inmemory" command -v dotnet >/dev/null || die "dotnet is not on PATH" command -v chrome-devtools-axi >/dev/null || die "chrome-devtools-axi is not on PATH" - if [[ "$store" == postgres ]]; then docker info >/dev/null 2>&1 || die "Docker is not running"; fi + if [[ "$store" =~ ^(postgres|oracle)$ ]]; then docker info >/dev/null 2>&1 || die "Docker is not running"; fi RUN="$(date +%Y%m%d-%H%M%S)-$$" RUN_DIR="$EVIDENCE_ROOT/$RUN" @@ -170,6 +171,7 @@ cmd_launch() { set_var BROWSER_STARTED 0 set_var CONTAINER "" set_var PG_PORT "" + set_var ORA_PORT "" set_var SQLITE_DB "" log "run $RUN ($store) - evidence at $RUN_DIR" mkdir -p "$SCRATCH" @@ -203,6 +205,26 @@ cmd_launch() { store_args=(--BackWave:Store=Postgres "--BackWave:Postgres:ConnectionString=Host=127.0.0.1;Port=$PG_PORT;Username=backwave;Password=backwave;Database=backwave_sample") set_var STORE_NAME Postgres ;; + oracle) + set_var CONTAINER "backwave-verify-$RUN" + # A fixed host port, so that the run can stop and start the container (a store outage) + # and the app finds the database again on the same port. + set_var ORA_PORT "$(free_port)" + log "starting $ORA_IMAGE as $CONTAINER on 127.0.0.1:$ORA_PORT (first boot takes 30-90 s)" + # Oracle needs far more shared memory than the 64 MB Docker default. + docker run -d --name "$CONTAINER" --label "backwave.verify.run=$RUN" --shm-size 2g \ + -e ORACLE_PASSWORD=backwave -e APP_USER=backwave -e APP_USER_PASSWORD=backwave \ + -p "127.0.0.1:$ORA_PORT:1521" "$ORA_IMAGE" > /dev/null \ + || die "docker run failed - run 'bwv cleanup'" + local i + for i in $(seq 1 180); do + docker exec "$CONTAINER" healthcheck.sh >/dev/null 2>&1 && break + (( i == 180 )) && die "Oracle was not ready in 180 s - run 'bwv cleanup'" + sleep 1 + done + store_args=(--BackWave:Store=Oracle + "--BackWave:Oracle:ConnectionString=User Id=backwave;Password=backwave;Data Source=127.0.0.1:$ORA_PORT/FREEPDB1") + set_var STORE_NAME Oracle ;; esac log "starting Sample.Api on $(base) (log: $RUN_DIR/app.log)" @@ -264,9 +286,9 @@ cmd_doctor() { case "$STORE" in sqlite) [[ -f "$SQLITE_DB" ]] && ok "SQLite file $SQLITE_DB" || bad "SQLite file $SQLITE_DB missing" ;; - postgres) + postgres|oracle) if container_is_ours && [[ "$(docker inspect -f '{{.State.Running}}' "$CONTAINER")" == true ]]; then - ok "container $CONTAINER running on 127.0.0.1:$PG_PORT (label run=$RUN)" + ok "container $CONTAINER running on 127.0.0.1:${PG_PORT:-$ORA_PORT} (label run=$RUN)" else bad "container $CONTAINER is not running with label backwave.verify.run=$RUN" fi ;; @@ -368,6 +390,8 @@ cmd_sql() { sqlite) sqlite3 -header -column -cmd '.timeout 5000' "$SQLITE_DB" "PRAGMA query_only=1; $query" ;; postgres) docker exec -e PGOPTIONS='-c default_transaction_read_only=on' "$CONTAINER" \ psql -U backwave -d backwave_sample -v ON_ERROR_STOP=1 -c "$query" ;; + oracle) printf 'SET DEFINE OFF\nSET MARKUP CSV ON\nWHENEVER SQLERROR EXIT FAILURE\nSET TRANSACTION READ ONLY;\n%s;\nEXIT\n' "${query%;}" \ + | docker exec -i "$CONTAINER" sqlplus -S -L backwave/backwave@localhost/FREEPDB1 ;; esac } > "$out" 2>&1 || { cat "$out"; die "query failed (saved db/$name.txt)"; } cat "$out" diff --git a/samples/BackWave.Sample.Api/BackWave.Sample.Api.csproj b/samples/BackWave.Sample.Api/BackWave.Sample.Api.csproj index 43d995c..9a957c4 100644 --- a/samples/BackWave.Sample.Api/BackWave.Sample.Api.csproj +++ b/samples/BackWave.Sample.Api/BackWave.Sample.Api.csproj @@ -31,6 +31,7 @@ + The SQLite Embedded Adapter, co-resident: BackWave's tables live in the app's OWN file. public static readonly StoreKind Sqlite = new SqliteCoResidentStore(); @@ -33,6 +35,7 @@ public static StoreKind FromConfiguration(IConfiguration configuration) var configured = configuration["BackWave:Store"] ?? nameof(InMemory); return configured.Equals(nameof(Postgres), StringComparison.OrdinalIgnoreCase) ? Postgres : configured.Equals(nameof(SqlServer), StringComparison.OrdinalIgnoreCase) ? SqlServer + : configured.Equals(nameof(Oracle), StringComparison.OrdinalIgnoreCase) ? Oracle : configured.Equals(nameof(SqliteDedicated), StringComparison.OrdinalIgnoreCase) ? SqliteDedicated : configured.Equals(nameof(Sqlite), StringComparison.OrdinalIgnoreCase) ? Sqlite : InMemory; @@ -155,6 +158,31 @@ private static string ConnectionString(IConfiguration configuration) => configuration["BackWave:SqlServer:ConnectionString"] ?? DefaultConnectionString; } + /// + /// The Oracle adapter. The sample registers no EF provider for Oracle, so it has no business table and + /// forgoes Transactional Enqueue: POST /tx answers 409. + /// + private sealed class OracleStore : StoreKind + { + private const string DefaultConnectionString = + "User Id=backwave;Password=backwave;Data Source=localhost:15211/FREEPDB1"; + + public override string Name => "Oracle"; + public override bool SupportsTransactionalEnqueue => false; + + public override IJobStore CreateStore(IConfiguration configuration, ILoggerFactory loggerFactory) + => new OracleJobStore(new OracleStoreOptions + { + ConnectionString = configuration["BackWave:Oracle:ConnectionString"] ?? DefaultConnectionString, + AutoMigrate = true, + LoggerFactory = loggerFactory, // opts in to the migration log; null (the default) is silent + }); + + public override void AddAdapterTracing(TracerProviderBuilder tracing) => tracing.AddBackWaveOracleInstrumentation(); + + public override void AddAdapterMetrics(MeterProviderBuilder metrics) => metrics.AddBackWaveOracleInstrumentation(); + } + /// /// The SQLite Embedded Adapter (no server, no Docker — just a local file). Two deployments: /// puts BackWave's tables in the application's OWN database file diff --git a/samples/BackWave.Sample.Api/docker-compose.yml b/samples/BackWave.Sample.Api/docker-compose.yml index 3b80e02..4e36398 100644 --- a/samples/BackWave.Sample.Api/docker-compose.yml +++ b/samples/BackWave.Sample.Api/docker-compose.yml @@ -30,3 +30,19 @@ services: interval: 5s timeout: 5s retries: 60 + + oracle: + image: gvenzl/oracle-free:23-slim + # Oracle needs far more shared memory than Docker's 64 MB default. + shm_size: 2gb + environment: + ORACLE_PASSWORD: backwave + APP_USER: backwave + APP_USER_PASSWORD: backwave + ports: + - "15211:1521" + healthcheck: + test: ["CMD", "healthcheck.sh"] + interval: 5s + timeout: 5s + retries: 60 diff --git a/src/BackWave.Oracle/BackWave.Oracle.csproj b/src/BackWave.Oracle/BackWave.Oracle.csproj index f511fce..5b0f149 100644 --- a/src/BackWave.Oracle/BackWave.Oracle.csproj +++ b/src/BackWave.Oracle/BackWave.Oracle.csproj @@ -23,6 +23,14 @@ + + + + + + + diff --git a/src/BackWave.Oracle/OracleFaultCodes.cs b/src/BackWave.Oracle/OracleFaultCodes.cs index 7c52617..8865f42 100644 --- a/src/BackWave.Oracle/OracleFaultCodes.cs +++ b/src/BackWave.Oracle/OracleFaultCodes.cs @@ -2,17 +2,37 @@ namespace BackWave.Oracle; // The single source of truth for the transient Oracle connectivity/timeout error codes. A cold-booting // fleet can hit these listener/handshake-storm and connection-lost conditions without having caused them, -// so the migrator rides them out and the store tags them transient. Both reference this list so the two -// stay in lockstep. +// and a pump that loses its database in a store outage (network loss, a stop, a restart) sees the same +// codes until the database answers again. So the migrator rides them out and the store tags them +// transient: the worker group degrades and retries on the next poll instead of halting. Both reference +// this list so the two stay in lockstep. // 12170 TNS connect timeout 12541 no listener // 12514 listener does not know service 12518 listener could not hand off // 12537 connection closed 12570 packet reader failure // 3113 end-of-file on channel 3114 not connected // 28 session killed 1033 database initializing/shutting down +// 1089 shutdown in progress 1109 database not open (a restart opens it after the listener) // 12154 could not resolve connect id (transient during cold DNS/TNS) +// 12545 target host does not exist (the host name stops resolving in a network outage) +// ODP.NET managed raises its own codes when a connect fails: +// 50201 failed to connect to server or failed to parse connect string (connection refused while the +// database is down, or a service name the listener does not know during a restart) +// 50000 connection request timed out (the host does not answer) +// 50012 pooled connection request timed out (the pool has no free connection) +// A misconfigured Data Source raises some of these codes too, so it leaves the group degraded and never +// halts it: a mistyped host name (12545), a mistyped EZConnect string or TNS alias (12154), and a wrong +// service name or a descriptor that ODP.NET cannot parse (50201). A wrong attribute name in the +// connection string fails the open with an ArgumentException, and a wrong password raises 1017. Neither +// is in this list, so both halt the group. internal static class OracleFaultCodes { - // True when the ORA code is one of the transient connectivity/timeout faults above. + // True when the ORA or ODP.NET code is one of the transient connectivity/timeout faults above. internal static bool IsConnectivityFault(int code) => code is - 12170 or 12541 or 12514 or 12518 or 12537 or 12570 or 3113 or 3114 or 28 or 1033 or 12154; + 12170 or 12541 or 12514 or 12518 or 12537 or 12570 or 3113 or 3114 or 28 or 1033 or 12154 + or 1089 or 1109 or 12545 or 50201 or 50000 or 50012; + + // True when Oracle rolled back one statement to break a deadlock: 60 on rows, 4020 on the library cache + // (two nodes that create the same object at the same time). A new attempt can succeed, so the migrator + // retries both and the store tags both transient. + internal static bool IsDeadlock(int code) => code is 60 or 4020; } diff --git a/src/BackWave.Oracle/OracleJobStore.cs b/src/BackWave.Oracle/OracleJobStore.cs index d0955c3..49834b0 100644 --- a/src/BackWave.Oracle/OracleJobStore.cs +++ b/src/BackWave.Oracle/OracleJobStore.cs @@ -187,7 +187,9 @@ private async ValueTask EnsureReadyAsync(CancellationToken cancellationToken) } if (options.AutoMigrate) { - await OracleMigrator.MigrateAsync(options.ConnectionString, options.SchemaName, options.CoordinateMigration, cancellationToken).ConfigureAwait(false); + await OracleMigrator.MigrateCoreAsync( + options.ConnectionString, options.SchemaName, options.CoordinateMigration, + OracleMigrator.StoreRetryBudget, cancellationToken).ConfigureAwait(false); BackWaveLog.MigrationApplied(_logger, "oracle"); } await OracleMigrator.VerifySchemaVersionAsync(options.ConnectionString, options.SchemaName, _logger, cancellationToken).ConfigureAwait(false); @@ -222,12 +224,12 @@ private Task FailpointAsync(string name, CancellationToken cancellationToken) // Classifies a store fault for the backwave.store.faults metric tag, mirroring the host's own // transient/terminal split: a bare TimeoutException plus the shared connectivity/timeout set and the - // ORA-00060 deadlock victim are the whole transient set. Emit-only - the host still makes the real - // retry/fail-stop decision from the rethrown exception. + // deadlock victims (ORA-00060, ORA-04020) are the whole transient set. Emit-only - the host still makes + // the real retry/fail-stop decision from the rethrown exception. private static bool IsTransientStoreFault(Exception exception) => exception is TimeoutException || (exception is OracleException o - && (OracleFaultCodes.IsConnectivityFault(o.Number) || o.Number == 60)); + && (OracleFaultCodes.IsConnectivityFault(o.Number) || OracleFaultCodes.IsDeadlock(o.Number))); /// public bool IsTransientFault(Exception exception) => IsTransientStoreFault(exception); diff --git a/src/BackWave.Oracle/OracleMigrator.cs b/src/BackWave.Oracle/OracleMigrator.cs index 7aa206a..ac00e99 100644 --- a/src/BackWave.Oracle/OracleMigrator.cs +++ b/src/BackWave.Oracle/OracleMigrator.cs @@ -1,3 +1,4 @@ +using System.Diagnostics; using BackWave.Diagnostics; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; @@ -19,16 +20,17 @@ public static class OracleMigrator /// The schema version this build of the adapter requires the database to be at. public const int ExpectedSchemaVersion = 2; - // Transient connection faults a cold-booting fleet can hit that the bounded retry should ride out - // rather than surface: the shared listener/handshake-storm and connection-lost connectivity set. - private static bool IsTransient(OracleException exception) - => OracleFaultCodes.IsConnectivityFault(exception.Number); + // Transient faults a cold-booting fleet can hit that the bounded retry should ride out rather than + // surface: the shared listener/handshake-storm and connection-lost connectivity set, and a deadlock. + // The scripts are idempotent, so a new attempt converges. + internal static bool IsTransient(int number) + => OracleFaultCodes.IsConnectivityFault(number) || OracleFaultCodes.IsDeadlock(number); /// /// Runs every schema script in version order, bringing the database up to the version this adapter /// requires. Idempotent - safe to run against an already-current database. Transient connection faults - /// from a fleet cold-boot are retried internally with a bounded backoff; a fault that persists past - /// those attempts propagates. + /// and deadlocks (ORA-00060, ORA-04020) from a fleet cold-boot are retried internally with a bounded + /// backoff; a fault that persists past those attempts propagates. /// /// The ODP.NET connection string for the target database. /// Token to cancel the migration. @@ -51,18 +53,30 @@ public static Task MigrateAsync(string connectionString, CancellationToken cance /// /// /// When (the default), a cold-booting fleet rides out the transient connection - /// faults of a first-boot handshake storm with a bounded retry. The Oracle schema script is idempotent - /// and concurrency-safe on its own, so no database lock is taken either way. Setting it to - /// runs the script once with no transient retry; it is safe when the deployment - /// already serializes migration itself. + /// faults of a first-boot handshake storm and the deadlocks (ORA-00060, ORA-04020) of nodes that create + /// the same object at once, with a bounded retry. The Oracle schema script is idempotent, so no database + /// lock is taken either way. Setting it to runs the script once with no retry; + /// it is safe when the deployment already serializes migration itself. /// /// Token to cancel the migration. /// A task that completes once every script has run. /// is not a valid identifier. /// The token was cancelled. - public static async Task MigrateAsync( + public static Task MigrateAsync( string connectionString, string schemaName, bool coordinate = true, CancellationToken cancellationToken = default) + => MigrateCoreAsync(connectionString, schemaName, coordinate, retryBudget: null, cancellationToken); + + // The time limit on the transient retry when the store migrates on first use. The store holds its + // ready gate for the whole migration, so every other store call waits, and the pump migrates again on + // its next poll anyway. + internal static readonly TimeSpan StoreRetryBudget = TimeSpan.FromSeconds(5); + + // The migration itself. A null retry budget gives the full bounded attempts with no time limit, for a + // caller that migrates once and has no next poll. + internal static async Task MigrateCoreAsync( + string connectionString, string schemaName, bool coordinate, TimeSpan? retryBudget, + CancellationToken cancellationToken) { var rewriter = new SchemaRewriter(schemaName); @@ -78,8 +92,10 @@ public static async Task MigrateAsync( // listener/handshake faults it did nothing to cause. The schema script is idempotent and // concurrency-safe by construction (guarded DDL + DDL serialized on each object by Oracle), so // re-running it converges; ride out a bounded number of transient faults instead of crashing - // startup. On exhaustion the last fault propagates (fail-stop preserved). - const int maxAttempts = 8; + // startup. A retry budget bounds the retry by time as well, because a host that does not answer + // costs a full connect timeout on each attempt. On exhaustion the last fault propagates (fail-stop + // preserved). + var started = Stopwatch.GetTimestamp(); for (var attempt = 1; ; attempt++) { try @@ -87,13 +103,21 @@ public static async Task MigrateAsync( await ApplyScriptsAsync(connectionString, rewriter, cancellationToken).ConfigureAwait(false); return; } - catch (OracleException exception) when (attempt < maxAttempts && IsTransient(exception)) + catch (OracleException exception) when (IsTransient(exception.Number) + && ShouldRetry(attempt, Stopwatch.GetElapsedTime(started), retryBudget)) { await Task.Delay(TimeSpan.FromMilliseconds(50 * attempt), cancellationToken).ConfigureAwait(false); } } } + // True when a transient fault on this attempt gets another attempt. + internal static bool ShouldRetry(int attempt, TimeSpan elapsed, TimeSpan? retryBudget) + { + const int maxAttempts = 8; + return attempt < maxAttempts && (retryBudget is not { } budget || elapsed < budget); + } + // Runs every embedded schema script in version order. Each script is a single anonymous PL/SQL block // (guarded object-by-object), so one ExecuteNonQuery per script is correct and Oracle's implicit DDL // commit needs no explicit transaction. diff --git a/src/BackWave.SqlServer/BackWave.SqlServer.csproj b/src/BackWave.SqlServer/BackWave.SqlServer.csproj index ddd085a..25b1a70 100644 --- a/src/BackWave.SqlServer/BackWave.SqlServer.csproj +++ b/src/BackWave.SqlServer/BackWave.SqlServer.csproj @@ -19,6 +19,14 @@ + + + + + + + diff --git a/tests/BackWave.Dashboard.Tests/NoConfigReload.cs b/tests/BackWave.Dashboard.Tests/NoConfigReload.cs new file mode 100644 index 0000000..ba33847 --- /dev/null +++ b/tests/BackWave.Dashboard.Tests/NoConfigReload.cs @@ -0,0 +1,12 @@ +using System.Runtime.CompilerServices; + +namespace BackWave.Dashboard.Tests; + +internal static class NoConfigReload +{ + // WebApplication.CreateBuilder watches appsettings.json for changes. On macOS, starting that + // file watcher sometimes blocks forever, and the whole run hangs. No test here reloads config. + [ModuleInitializer] + internal static void Initialize() => + Environment.SetEnvironmentVariable("DOTNET_hostBuilder__reloadConfigOnChange", "false"); +} diff --git a/tests/BackWave.Oracle.Tests/NoConfigReload.cs b/tests/BackWave.Oracle.Tests/NoConfigReload.cs new file mode 100644 index 0000000..7fcbd18 --- /dev/null +++ b/tests/BackWave.Oracle.Tests/NoConfigReload.cs @@ -0,0 +1,12 @@ +using System.Runtime.CompilerServices; + +namespace BackWave.Oracle.Tests; + +internal static class NoConfigReload +{ + // WebApplication.CreateBuilder watches appsettings.json for changes. On macOS, starting that + // file watcher sometimes blocks forever, and the whole run hangs. No test here reloads config. + [ModuleInitializer] + internal static void Initialize() => + Environment.SetEnvironmentVariable("DOTNET_hostBuilder__reloadConfigOnChange", "false"); +} diff --git a/tests/BackWave.Oracle.Tests/OracleCoordinatedMigrationTests.cs b/tests/BackWave.Oracle.Tests/OracleCoordinatedMigrationTests.cs index c853b00..5543058 100644 --- a/tests/BackWave.Oracle.Tests/OracleCoordinatedMigrationTests.cs +++ b/tests/BackWave.Oracle.Tests/OracleCoordinatedMigrationTests.cs @@ -49,7 +49,15 @@ public sealed class OracleCoordinatedMigrationTests : IAsyncLifetime barrier.SignalAndWait(); try { - await MigrateRidingOutConnectivityFaultsAsync(coordinate); + // The coordinated path calls the migrator directly, so the storm proves its own retry. + if (coordinate) + { + await OracleMigrator.MigrateAsync(MigrationConnectionString, Schema); + } + else + { + await MigrateUncoordinatedRidingOutTransientFaultsAsync(); + } return (Exception?)null; } catch (Exception exception) @@ -61,23 +69,23 @@ public sealed class OracleCoordinatedMigrationTests : IAsyncLifetime return await Task.WhenAll(tasks); } - // The coordinate:false path strips the migrator's own transient-connectivity retry, so a cold-boot - // listener-handoff fault (ORA-12518, ORA-12570, and the rest of the connectivity set) can surface here - - // unrelated to the guarded-DDL convergence this test proves. Ride those out with a bounded retry. Any - // non-connectivity error (a real convergence fault such as ORA-00955 or ORA-00001) propagates unchanged - // and fails the storm as before. - private static async Task MigrateRidingOutConnectivityFaultsAsync(bool coordinate) + // The coordinate:false path strips the migrator's own transient retry, so a cold-boot listener-handoff + // fault (ORA-12518, ORA-12570, and the rest of the connectivity set) or a deadlock between two nodes + // that create the same object (ORA-04020) can surface here - unrelated to the guarded-DDL convergence + // this test proves. Ride those out with a bounded retry, as the coordinated path does. Any other error + // (a real convergence fault such as ORA-00955 or ORA-00001) propagates unchanged and fails the storm. + private static async Task MigrateUncoordinatedRidingOutTransientFaultsAsync() { const int maxAttempts = 10; for (var attempt = 1; ; attempt++) { try { - await OracleMigrator.MigrateAsync(MigrationConnectionString, Schema, coordinate); + await OracleMigrator.MigrateAsync(MigrationConnectionString, Schema, coordinate: false); return; } catch (OracleException exception) - when (OracleFaultCodes.IsConnectivityFault(exception.Number) && attempt < maxAttempts) + when (OracleMigrator.IsTransient(exception.Number) && attempt < maxAttempts) { await Task.Delay(250); } diff --git a/tests/BackWave.Oracle.Tests/OracleOutageRecoveryTests.cs b/tests/BackWave.Oracle.Tests/OracleOutageRecoveryTests.cs new file mode 100644 index 0000000..1a01437 --- /dev/null +++ b/tests/BackWave.Oracle.Tests/OracleOutageRecoveryTests.cs @@ -0,0 +1,170 @@ +using System.Text.Json.Serialization; +using BackWave.Core; +using BackWave.Hosting; +using BackWave.Jobs; +using BackWave.Storage; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Oracle.ManagedDataAccess.Client; + +namespace BackWave.Oracle.Tests; + +/// +/// After an outage a worker group on Oracle must claim and run jobs again without a process +/// restart. The outage is the database killing every session of the group while the pool still holds +/// them, so the next claim runs on a dead connection. The group runs as a private bw_recover user, so the +/// kill reaches this test's sessions and no other user's. +/// +[Collection("oracle")] +public sealed class OracleOutageRecoveryTests(Xunit.Abstractions.ITestOutputHelper output) : IAsyncLifetime +{ + private const string Schema = "bw_recover"; + private const string Group = "recovery-workers"; + + // Pooling stays on (the default): the dead pooled session is the outage this test models. A minimum + // pool size of zero stops the pool from opening a fresh session after ClearPool, which would block + // the DROP USER at teardown. + private static readonly string SchemaConnectionString = + new OracleConnectionStringBuilder(OracleTestDatabase.ConnectionString) + { + UserID = Schema, + Password = Schema, + MinPoolSize = 0, + }.ConnectionString; + + public Task InitializeAsync() => OracleTestDatabase.ResetUserAsync(Schema); + + public async Task DisposeAsync() + { + // Lock the user, so the pool cannot open a new session, then end every session it still has, so + // the DROP USER does not wait on them. + await OutageTestSupport.LockUserAsync(Schema); + OracleConnection.ClearPool(new OracleConnection(SchemaConnectionString)); + await OutageTestSupport.DisconnectSessionsAsync(Schema); + await OracleTestDatabase.DropUserAsync(Schema); + } + + [Fact] + public async Task A_worker_group_claims_and_runs_jobs_again_after_its_pooled_sessions_are_killed() + { + var store = new OracleJobStore(new OracleStoreOptions + { + ConnectionString = SchemaConnectionString, + SchemaName = Schema, + AutoMigrate = true, + }); + var builder = Host.CreateApplicationBuilder(); + builder.Services.AddTransient, RecoveryHandler>(); + builder.Services.AddBackWave(backwave => backwave + .UseStore(store) + .UseRegistry(new JobRegistry( + [ + JobRegistration.Create("recovery", OracleRecoveryJsonContext.Default.RecoveryJob), + ])) + .AddWorkerGroup(new WorkerGroupOptions + { + Name = Group, + Policy = new DispatchPolicy.Strict(["default"]), + PollInterval = TimeSpan.FromMilliseconds(200), + MaxPollInterval = TimeSpan.Zero, + LeaseDuration = TimeSpan.FromSeconds(30), + })); + using var host = builder.Build(); + var health = host.Services.GetRequiredService(); + var client = host.Services.GetRequiredService(); + + await host.StartAsync(); + try + { + // Baseline: the group runs a job before the outage. + var before = await client.EnqueueAsync(new RecoveryJob(1), dueTime: DateTimeOffset.UtcNow); + await WaitForSucceededAsync(store, health, before); + + var killed = await KillSessionsAsync(); + output.WriteLine($"killed {killed} {Schema} sessions"); + Assert.True(killed > 0, $"the group must hold at least one {Schema} session to kill"); + + // The next poll takes a killed session from the pool. The group must degrade, not halt. + var cause = await OutageTestSupport.WaitForDegradedAsync(health, Group, "the killed sessions to mark the group degraded"); + output.WriteLine($"degraded: {cause}"); + Assert.True(OracleTransientFaultTests.NamesTransientCode(cause), $"the degraded cause must name a transient code: {cause}"); + + // Each failed poll drops one dead session from the pool. Wait until none is left, so the + // enqueue below cannot take one either. + await OutageTestSupport.WaitForAsync(async () => await CountKilledSessionsAsync() == 0, health, Group, + "the pool to drop every killed session"); + + // Recovery: a job enqueued after the outage is claimed and run by the same group. + var after = await client.EnqueueAsync(new RecoveryJob(2), dueTime: DateTimeOffset.UtcNow); + await WaitForSucceededAsync(store, health, after); + OutageTestSupport.AssertNotHalted(health, Group); + } + finally + { + await host.StopAsync(); + } + } + + // Kills every session of the private user as SYSTEM. The filter is the username alone, so no session + // of another user or test is touched. The outage kill is not IMMEDIATE on purpose: IMMEDIATE closes + // the socket, and ODP.NET then drops the session from the pool before it hands it out. A plain kill + // keeps the socket open, so the pool hands out the dead session and its next call raises the fault. + private static async Task KillSessionsAsync() + { + await using var admin = new OracleConnection(OracleTestDatabase.SystemConnectionString); + await admin.OpenAsync(); + + var sessions = new List(); + await using (var select = admin.CreateCommand()) + { + select.CommandText = "SELECT sid, serial# FROM v$session WHERE username = :username AND status <> 'KILLED'"; + select.Parameters.Add(new OracleParameter("username", Schema.ToUpperInvariant())); + await using var reader = await select.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + { + sessions.Add($"{reader.GetInt32(0)},{reader.GetInt32(1)}"); + } + } + + foreach (var session in sessions) + { + await using var kill = admin.CreateCommand(); + kill.CommandText = $"ALTER SYSTEM KILL SESSION '{session}'"; + try + { + await kill.ExecuteNonQueryAsync(); + } + // ORA-00031: the session is busy, so Oracle marks it and ends it on its next call. + // ORA-00030: the session ended after the select. + catch (OracleException exception) when (exception.Number is 31 or 30) + { + } + } + return sessions.Count; + } + + // A killed session stays in v$session until its client makes a call and is told it is dead. + private static async Task CountKilledSessionsAsync() + { + await using var admin = new OracleConnection(OracleTestDatabase.SystemConnectionString); + await admin.OpenAsync(); + await using var count = admin.CreateCommand(); + count.CommandText = "SELECT count(*) FROM v$session WHERE username = :username AND status = 'KILLED'"; + count.Parameters.Add(new OracleParameter("username", Schema.ToUpperInvariant())); + return Convert.ToInt32(await count.ExecuteScalarAsync()); + } + + private static Task WaitForSucceededAsync(OracleJobStore store, BackWaveHealth health, Guid jobId) => + OutageTestSupport.WaitForAsync(async () => (await store.GetJobAsync(jobId))?.State == JobState.Succeeded, health, + Group, $"job {jobId} to reach Succeeded"); +} + +public sealed record RecoveryJob(int Index); + +public sealed class RecoveryHandler : IJobHandler +{ + public Task HandleAsync(RecoveryJob job, JobContext context, CancellationToken cancellationToken) => Task.CompletedTask; +} + +[JsonSerializable(typeof(RecoveryJob))] +internal sealed partial class OracleRecoveryJsonContext : JsonSerializerContext; diff --git a/tests/BackWave.Oracle.Tests/OraclePoolExhaustionFaultTests.cs b/tests/BackWave.Oracle.Tests/OraclePoolExhaustionFaultTests.cs new file mode 100644 index 0000000..381063e --- /dev/null +++ b/tests/BackWave.Oracle.Tests/OraclePoolExhaustionFaultTests.cs @@ -0,0 +1,38 @@ +using BackWave.Storage; +using Oracle.ManagedDataAccess.Client; + +namespace BackWave.Oracle.Tests; + +/// +/// A pump that waits for a pooled connection and does not get one in time must see a transient +/// fault, not a fail-stop. A busy node can drain its pool without any fault in the database. This is an +/// integration test: it needs "docker compose up -d oracle". +/// +[Collection("oracle")] +public sealed class OraclePoolExhaustionFaultTests(Xunit.Abstractions.ITestOutputHelper output) +{ + [Fact] + public async Task A_pool_with_no_free_connection_raises_a_fault_the_adapter_classifies_transient() + { + await OracleTestDatabase.CreateFreshStoreAsync(); + + // A pool of one connection, keyed by this exact string, so no other test shares it. + var connectionString = new OracleConnectionStringBuilder(OracleTestDatabase.ConnectionString) + { + MaxPoolSize = 1, + ConnectionTimeout = 2, + }.ConnectionString; + var store = new OracleJobStore(new OracleStoreOptions { ConnectionString = connectionString }); + + // Hold the one pooled connection, so the store's next open must wait for it. + await using var held = new OracleConnection(connectionString); + await held.OpenAsync(); + + var fault = await Assert.ThrowsAsync(async () => + await store.ClaimAsync(new ClaimRequest("w1", ["default"], 10, TimeSpan.FromMinutes(1), DateTimeOffset.UtcNow))); + output.WriteLine($"ORA-{fault.Number}: {fault.Message} (IsTransient={fault.IsTransient})"); + + Assert.Equal(50012, fault.Number); + Assert.True(store.IsTransientFault(fault), $"ORA-{fault.Number} must classify transient"); + } +} diff --git a/tests/BackWave.Oracle.Tests/OracleRefusedConnectionRecoveryTests.cs b/tests/BackWave.Oracle.Tests/OracleRefusedConnectionRecoveryTests.cs new file mode 100644 index 0000000..48083e0 --- /dev/null +++ b/tests/BackWave.Oracle.Tests/OracleRefusedConnectionRecoveryTests.cs @@ -0,0 +1,292 @@ +using System.Collections.Concurrent; +using System.Net; +using System.Net.Sockets; +using System.Text.Json.Serialization; +using BackWave.Core; +using BackWave.Hosting; +using BackWave.Jobs; +using BackWave.Storage; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Oracle.ManagedDataAccess.Client; + +namespace BackWave.Oracle.Tests; + +/// +/// A worker group on Oracle that boots while its database refuses connections must degrade, not halt, and +/// run jobs once the database answers. The store connects through a loopback port that nothing listens on, +/// so ODP.NET gets a refused connect. A TCP forwarder then opens that port in front of the real database, +/// which models the database coming back. The group runs as a private bw_refused user, so the teardown +/// reaches this test's sessions and no other user's. +/// +[Collection("oracle")] +public sealed class OracleRefusedConnectionRecoveryTests(Xunit.Abstractions.ITestOutputHelper output) : IAsyncLifetime +{ + private const string Schema = "bw_refused"; + private const string Group = "refused-workers"; + + private static readonly TimeSpan PollInterval = TimeSpan.FromMilliseconds(200); + + private static readonly int Port = ReserveFreePort(); + + // The store reaches the database only through the forwarder port. A minimum pool size of zero stops the + // pool from opening a fresh session after ClearPool, which would block the DROP USER at teardown. The + // short connect timeout keeps a connect that hangs from outlasting the test. + private static readonly string SchemaConnectionString = + new OracleConnectionStringBuilder(OracleTestDatabase.ConnectionString) + { + UserID = Schema, + Password = Schema, + DataSource = $"127.0.0.1:{Port}/{Upstream.Service}", + MinPoolSize = 0, + ConnectionTimeout = 3, + }.ConnectionString; + + private TcpForwarder? _forwarder; + + public Task InitializeAsync() => OracleTestDatabase.ResetUserAsync(Schema); + + public async Task DisposeAsync() + { + // Lock the user, so the pool cannot open a new session, then end every session it still has, so + // the DROP USER does not wait on them. + await OutageTestSupport.LockUserAsync(Schema); + OracleConnection.ClearPool(new OracleConnection(SchemaConnectionString)); + await OutageTestSupport.DisconnectSessionsAsync(Schema); + await OracleTestDatabase.DropUserAsync(Schema); + if (_forwarder is not null) + { + await _forwarder.DisposeAsync(); + } + } + + [Fact] + public async Task A_worker_group_that_boots_while_connections_are_refused_runs_jobs_once_the_database_answers() + { + var store = new OracleJobStore(new OracleStoreOptions + { + ConnectionString = SchemaConnectionString, + SchemaName = Schema, + AutoMigrate = true, + }); + var builder = Host.CreateApplicationBuilder(); + builder.Services.AddTransient, RefusedHandler>(); + builder.Services.AddBackWave(backwave => backwave + .UseStore(store) + .UseRegistry(new JobRegistry( + [ + JobRegistration.Create("refused", OracleRefusedJsonContext.Default.RefusedJob), + ])) + .AddWorkerGroup(new WorkerGroupOptions + { + Name = Group, + Policy = new DispatchPolicy.Strict(["default"]), + PollInterval = PollInterval, + MaxPollInterval = TimeSpan.Zero, + LeaseDuration = TimeSpan.FromSeconds(30), + })); + using var host = builder.Build(); + var health = host.Services.GetRequiredService(); + var client = host.Services.GetRequiredService(); + + await host.StartAsync(); + try + { + // The port is closed, so the first poll's migration gets a refused connect. The group must + // degrade, not halt: the host degrades a group only on a fault the store calls transient. + var cause = await OutageTestSupport.WaitForDegradedAsync(health, Group, "the refused connect to mark the group degraded"); + output.WriteLine($"degraded while refused: {cause}"); + Assert.Matches(@"ORA-\d+", cause); + + // An enqueue while the port is closed raises the same fault to the caller. The store must call it + // transient, so the caller can retry it as the group does. + var fault = await Assert.ThrowsAsync( + async () => await client.EnqueueAsync(new RefusedJob(0), dueTime: DateTimeOffset.UtcNow)); + output.WriteLine($"enqueue while refused: ORA-{fault.Number:D5}"); + Assert.True(store.IsTransientFault(fault), $"ORA-{fault.Number:D5} must be transient: {fault.Message}"); + + // Each poll runs the migration again and gets refused again. The enqueue above outlasted at + // least one more of those polls. None of them may halt the group. + await WatchNotHaltedAsync(health, 3 * PollInterval); + + // The database answers again. + _forwarder = TcpForwarder.Start(Port, Upstream.Host, Upstream.Port); + + // Recovery: a job enqueued after the database answers is claimed and run by the same group. + var first = await client.EnqueueAsync(new RefusedJob(1), dueTime: DateTimeOffset.UtcNow); + await WaitForSucceededAsync(store, health, first); + OutageTestSupport.AssertNotHalted(health, Group); + + // A drop mid-session: the forwarder closes every live connection, so no pooled session is + // usable. The job is due after the drop, so the group must claim it on a fresh session. The pool + // discards an idle connection with a closed socket when it hands it out, so the drop can pass + // without a fault; a call in flight at the drop gets ORA-12537. + var second = await client.EnqueueAsync(new RefusedJob(2), dueTime: DateTimeOffset.UtcNow.AddSeconds(1)); + var dropped = _forwarder.DropConnections(); + output.WriteLine($"dropped {dropped} live connections"); + Assert.True(dropped > 0, "the group must hold at least one live connection to drop"); + var dropCause = await WaitForSucceededAsync(store, health, second); + output.WriteLine($"degraded after drop: {dropCause ?? "never"}"); + OutageTestSupport.AssertNotHalted(health, Group); + } + finally + { + await host.StopAsync(); + } + } + + // Binds a listener on port 0 to get a free loopback port, then stops it, so nothing listens there until + // the forwarder starts. Only a port that nothing holds gets a refused connect (ORA-50201): on macOS a + // bound socket that does not listen makes the connect wait for its timeout (ORA-50000). Another + // process can take the port in the meantime, but then the connect is not refused or the forwarder + // cannot bind, so the test fails and never passes by mistake. + private static int ReserveFreePort() + { + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + var port = ((IPEndPoint)listener.LocalEndpoint).Port; + listener.Stop(); + return port; + } + + // The real database endpoint, from a host:port/service data source. + private static class Upstream + { + private static readonly string DataSource = + new OracleConnectionStringBuilder(OracleTestDatabase.ConnectionString).DataSource; + + public static readonly string Host = DataSource[..DataSource.IndexOf(':')]; + public static readonly int Port = + int.Parse(DataSource[(DataSource.IndexOf(':') + 1)..DataSource.IndexOf('/')]); + public static readonly string Service = DataSource[(DataSource.IndexOf('/') + 1)..]; + } + + private static async Task WatchNotHaltedAsync(BackWaveHealth health, TimeSpan duration) + { + var deadline = DateTime.UtcNow + duration; + while (DateTime.UtcNow < deadline) + { + OutageTestSupport.AssertNotHalted(health, Group); + await Task.Delay(TimeSpan.FromMilliseconds(20)); + } + OutageTestSupport.AssertNotHalted(health, Group); + } + + // Returns the first degraded cause seen while waiting, or null when the group never degraded. + private static async Task WaitForSucceededAsync(OracleJobStore store, BackWaveHealth health, Guid jobId) + { + string? cause = null; + await OutageTestSupport.WaitForAsync(async () => + { + if (cause is null && health.DegradedGroups.TryGetValue(Group, out var degraded)) + { + cause = degraded; + } + return (await store.GetJobAsync(jobId))?.State == JobState.Succeeded; + }, health, Group, $"job {jobId} to reach Succeeded", TimeSpan.FromMilliseconds(5)); + return cause; + } + + // Accepts each client on a loopback port and pumps its bytes both ways to the upstream endpoint. + private sealed class TcpForwarder : IAsyncDisposable + { + private readonly TcpListener _listener; + private readonly string _upstreamHost; + private readonly int _upstreamPort; + private readonly CancellationTokenSource _stopping = new(); + private readonly ConcurrentDictionary _connections = new(); + private readonly Task _accepting; + + private TcpForwarder(int port, string upstreamHost, int upstreamPort) + { + _upstreamHost = upstreamHost; + _upstreamPort = upstreamPort; + _listener = new TcpListener(IPAddress.Loopback, port); + _listener.Start(); + _accepting = AcceptAsync(); + } + + public static TcpForwarder Start(int port, string upstreamHost, int upstreamPort) => + new(port, upstreamHost, upstreamPort); + + // Closes every live connection and returns how many there were. The listener stays open, so a + // new connect still reaches the database. + public int DropConnections() + { + var live = _connections.Values.ToList(); + foreach (var (client, upstream) in live) + { + client.Dispose(); + upstream.Dispose(); + } + return live.Count; + } + + public async ValueTask DisposeAsync() + { + _stopping.Cancel(); + _listener.Stop(); + DropConnections(); + await Task.WhenAll(_connections.Keys.Append(_accepting)); + _stopping.Dispose(); + } + + private async Task AcceptAsync() + { + try + { + while (true) + { + var client = await _listener.AcceptTcpClientAsync(_stopping.Token); + var upstream = new TcpClient(); + var completed = new TaskCompletionSource(); + _connections[completed.Task] = (client, upstream); + _ = PumpAsync(client, upstream, completed); + } + } + catch (Exception exception) when (exception is OperationCanceledException or SocketException or ObjectDisposedException) + { + } + } + + // Pumps both ways until either side closes, then closes both, so the other side sees the drop too. + private async Task PumpAsync(TcpClient client, TcpClient upstream, TaskCompletionSource completed) + { + try + { + await upstream.ConnectAsync(_upstreamHost, _upstreamPort, _stopping.Token); + var clientStream = client.GetStream(); + var upstreamStream = upstream.GetStream(); + var copies = new[] + { + clientStream.CopyToAsync(upstreamStream, _stopping.Token), + upstreamStream.CopyToAsync(clientStream, _stopping.Token), + }; + await Task.WhenAny(copies); + client.Dispose(); + upstream.Dispose(); + await Task.WhenAll(copies); + } + catch (Exception exception) when (exception is OperationCanceledException or IOException or SocketException or ObjectDisposedException) + { + } + finally + { + client.Dispose(); + upstream.Dispose(); + _connections.TryRemove(completed.Task, out _); + completed.SetResult(); + } + } + } +} + +public sealed record RefusedJob(int Index); + +public sealed class RefusedHandler : IJobHandler +{ + public Task HandleAsync(RefusedJob job, JobContext context, CancellationToken cancellationToken) => Task.CompletedTask; +} + +[JsonSerializable(typeof(RefusedJob))] +internal sealed partial class OracleRefusedJsonContext : JsonSerializerContext; diff --git a/tests/BackWave.Oracle.Tests/OracleRejectedLoginTests.cs b/tests/BackWave.Oracle.Tests/OracleRejectedLoginTests.cs new file mode 100644 index 0000000..8bb2e2d --- /dev/null +++ b/tests/BackWave.Oracle.Tests/OracleRejectedLoginTests.cs @@ -0,0 +1,53 @@ +using BackWave.Hosting; +using Microsoft.Extensions.DependencyInjection; +using Oracle.ManagedDataAccess.Client; + +namespace BackWave.Oracle.Tests; + +/// +/// The negative case for the outage tests: a fault that is not an outage must still halt the group. A user +/// that does not exist raises ORA-01017 on the live listener. The test does not use a wrong password for a +/// real user, because repeated failed logins can lock that account. This is an integration test: it needs +/// "docker compose up -d oracle". +/// +[Collection("oracle")] +public sealed class OracleRejectedLoginTests(Xunit.Abstractions.ITestOutputHelper output) +{ + private const string Group = OracleTransientFaultTests.Group; + + [Fact] + public async Task A_rejected_login_halts_the_worker_group() + { + var store = new OracleJobStore(new OracleStoreOptions + { + ConnectionString = new OracleConnectionStringBuilder(OracleTestDatabase.ConnectionString) + { + UserID = "bw_no_such_user", + Password = "bw_no_such_user", + ConnectionTimeout = 3, + }.ConnectionString, + }); + using var host = OracleTransientFaultTests.BuildHost(store); + var health = host.Services.GetRequiredService(); + + await host.StartAsync(); + try + { + // A listener fault on an early poll only degrades the group, so the wait ignores degrades and + // ends at the halt. + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(15); + while (DateTime.UtcNow < deadline && !health.HaltedGroups.ContainsKey(Group)) + { + await Task.Delay(50); + } + + Assert.True(health.HaltedGroups.TryGetValue(Group, out var halt), "a rejected login must halt the group"); + output.WriteLine($"halted: {halt.ExceptionType}: {halt.Message}"); + Assert.Contains("ORA-01017", halt.Message); + } + finally + { + await host.StopAsync(); + } + } +} diff --git a/tests/BackWave.Oracle.Tests/OracleTransientFaultTests.cs b/tests/BackWave.Oracle.Tests/OracleTransientFaultTests.cs new file mode 100644 index 0000000..8e957f6 --- /dev/null +++ b/tests/BackWave.Oracle.Tests/OracleTransientFaultTests.cs @@ -0,0 +1,197 @@ +using System.Diagnostics; +using System.Text.RegularExpressions; +using BackWave.Core; +using BackWave.Hosting; +using BackWave.Jobs; +using BackWave.Storage; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Oracle.ManagedDataAccess.Client; + +namespace BackWave.Oracle.Tests; + +/// +/// A network outage between a worker group and its Oracle database must degrade the group, not +/// fail-stop it. ODP.NET raises these faults while it opens a connection, so each case points a real store +/// at an endpoint that cannot answer and needs no database. +/// +public sealed class OracleTransientFaultTests(Xunit.Abstractions.ITestOutputHelper output) +{ + // Each endpoint reproduces one outage the way ODP.NET reports it to the pump. The code in each comment + // is the one measured here. A network can report the same outage with another code, so the tests + // assert that the code is transient, not that it is this one. + public static TheoryData Outages => new() + { + // The host name stops resolving (a DNS or network outage). DNS never resolves the .invalid domain. + // ORA-12545. + Descriptor("bw-no-such-host.invalid", 1521), + // The host answers but nothing listens on the port (the database is down). ORA-50201. + Descriptor("127.0.0.1", 1), + // The host does not answer at all, so the connect times out. 192.0.2.1 is in TEST-NET-1, which no + // network routes. ORA-50000. + Descriptor("192.0.2.1", 1521), + }; + + [Theory] + [MemberData(nameof(Outages))] + public async Task A_network_outage_raises_a_fault_the_adapter_classifies_transient(string dataSource) + { + var store = Store(dataSource); + + var fault = await Assert.ThrowsAsync(async () => + await store.ClaimAsync(new ClaimRequest("w1", ["default"], 10, TimeSpan.FromMinutes(1), DateTimeOffset.UtcNow))); + + output.WriteLine($"ORA-{fault.Number}: {fault.Message}"); + Assert.True(store.IsTransientFault(fault), $"ORA-{fault.Number} must classify transient"); + } + + [Theory] + [MemberData(nameof(Outages))] + public async Task A_network_outage_degrades_the_worker_group_instead_of_halting_it(string dataSource) + { + using var host = BuildHost(Store(dataSource)); + var health = host.Services.GetRequiredService(); + + await host.StartAsync(); + try + { + await WaitForDegradedOrHaltedAsync(health); + + OutageTestSupport.AssertNotHalted(health, Group); + Assert.True(health.DegradedGroups.TryGetValue(Group, out var cause), "the outage must mark the group degraded"); + output.WriteLine($"degraded: {cause}"); + Assert.True(NamesTransientCode(cause), $"the degraded cause must name a transient code: {cause}"); + + // Later polls hit the same outage. Watch three poll intervals: none of them may halt the group. + var watchEnd = DateTime.UtcNow + 3 * PollInterval; + while (DateTime.UtcNow < watchEnd) + { + OutageTestSupport.AssertNotHalted(health, Group); + await Task.Delay(50); + } + OutageTestSupport.AssertNotHalted(health, Group); + } + finally + { + await host.StopAsync(); + } + } + + // A host that does not answer costs a full connect timeout on every attempt. The store holds its ready + // gate while it migrates, so an enqueue waits for the claim's migration and then runs its own. The + // retry budget bounds both: without it each call waits out all eight attempts (about 17 s here). + [Fact] + public async Task An_unreachable_host_holds_the_store_ready_gate_only_for_the_retry_budget() + { + var store = new OracleJobStore(new OracleStoreOptions + { + ConnectionString = $"User Id=backwave;Password=backwave;Connection Timeout=2;Data Source={Descriptor("192.0.2.1", 1521)}", + AutoMigrate = true, + }); + var elapsed = Stopwatch.StartNew(); + + var claim = Assert.ThrowsAsync(async () => + await store.ClaimAsync(new ClaimRequest("w1", ["default"], 10, TimeSpan.FromMinutes(1), DateTimeOffset.UtcNow))); + var enqueue = Assert.ThrowsAsync(async () => + await store.EnqueueAsync(new NewJob(Guid.NewGuid(), "gate-test", "{}"u8.ToArray(), "default", DateTimeOffset.UtcNow), DateTimeOffset.UtcNow)); + var faults = await Task.WhenAll(claim, enqueue); + + output.WriteLine($"both faulted after {elapsed.Elapsed.TotalSeconds:F1} s: ORA-{faults[0].Number}, ORA-{faults[1].Number}"); + Assert.All(faults, fault => Assert.True(store.IsTransientFault(fault), $"ORA-{fault.Number} must classify transient")); + Assert.True(elapsed.Elapsed < TimeSpan.FromSeconds(20), $"the enqueue waited {elapsed.Elapsed.TotalSeconds:F1} s"); + } + + // On the store path the pump migrates again on its next poll, so the budget stops the retry. A caller + // that migrates once has no next poll, so the public migrator keeps every attempt with no time limit. + [Theory] + [InlineData(1, 15, null, true)] + [InlineData(7, 100, null, true)] + [InlineData(8, 0, null, false)] + [InlineData(1, 0, 5, true)] + [InlineData(2, 6, 5, false)] + [InlineData(8, 0, 5, false)] + public void The_migrator_bounds_its_retry_by_time_only_when_a_budget_is_given( + int attempt, int elapsedSeconds, int? budgetSeconds, bool retries) + { + var budget = budgetSeconds is { } seconds ? TimeSpan.FromSeconds(seconds) : (TimeSpan?)null; + Assert.Equal(retries, OracleMigrator.ShouldRetry(attempt, TimeSpan.FromSeconds(elapsedSeconds), budget)); + } + + // Two migrators that create the same index at the same time can deadlock in the library cache. Oracle + // rolls back one statement with ORA-04020, and the measured 16-node storm hit it about 1 run in 3. A + // row deadlock (ORA-00060) gets the same retry. A real schema fault such as ORA-00942 still stops the + // migration. + [Theory] + [InlineData(4020, true)] + [InlineData(60, true)] + [InlineData(12545, true)] + [InlineData(942, false)] + [InlineData(1031, false)] + public void The_migrator_retries_deadlocks_and_connectivity_faults_only(int number, bool retries) + => Assert.Equal(retries, OracleMigrator.IsTransient(number)); + + // The store migrates on first use with a 5 s retry budget, and one DDL lock wait can last 30 s, so a + // deadlock can reach the pump after the budget. The store must then mark it transient, or the group + // halts. The database raises each code here, so the exception is the one ODP.NET gives. + [Theory] + [InlineData(4020)] + [InlineData(60)] + public async Task The_store_classifies_a_deadlock_transient(int number) + { + await using var connection = new OracleConnection(OracleTestDatabase.ConnectionString); + await connection.OpenAsync(); + await using var raise = connection.CreateCommand(); + raise.CommandText = $"DECLARE e EXCEPTION; PRAGMA EXCEPTION_INIT(e, -{number}); BEGIN RAISE e; END;"; + var fault = await Assert.ThrowsAsync(() => raise.ExecuteNonQueryAsync()); + + var store = new OracleJobStore(new OracleStoreOptions { ConnectionString = OracleTestDatabase.ConnectionString }); + Assert.Equal(number, fault.Number); + Assert.True(store.IsTransientFault(fault), $"ORA-{fault.Number} must classify transient"); + } + + internal const string Group = "outage-workers"; + private static readonly TimeSpan PollInterval = TimeSpan.FromMilliseconds(200); + + internal static IHost BuildHost(OracleJobStore store) + { + var builder = Host.CreateApplicationBuilder(); + builder.Services.AddBackWave(backwave => backwave + .UseStore(store) + .UseRegistry(new JobRegistry([])) + .AddWorkerGroup(new WorkerGroupOptions + { + Name = Group, + Policy = new DispatchPolicy.Strict(["default"]), + PollInterval = PollInterval, + LeaseDuration = TimeSpan.FromSeconds(30), + })); + return builder.Build(); + } + + private static async Task WaitForDegradedOrHaltedAsync(BackWaveHealth health) + { + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(15); + while (DateTime.UtcNow < deadline + && !health.DegradedGroups.ContainsKey(Group) + && !health.HaltedGroups.ContainsKey(Group)) + { + await Task.Delay(50); + } + } + + // True when the cause names an ORA or ODP.NET code that the adapter classifies transient. + internal static bool NamesTransientCode(string cause) + { + var code = Regex.Match(cause, @"ORA-(\d+)"); + return code.Success && OracleFaultCodes.IsConnectivityFault(int.Parse(code.Groups[1].Value)); + } + + private static string Descriptor(string host, int port) => + $"(DESCRIPTION=(CONNECT_TIMEOUT=2)(ADDRESS=(PROTOCOL=TCP)(HOST={host})(PORT={port}))" + + "(CONNECT_DATA=(SERVICE_NAME=FREEPDB1)))"; + + private static OracleJobStore Store(string dataSource) => new(new OracleStoreOptions + { + ConnectionString = $"User Id=backwave;Password=backwave;Connection Timeout=3;Data Source={dataSource}", + }); +} diff --git a/tests/BackWave.Oracle.Tests/OracleWakeUpHintTests.cs b/tests/BackWave.Oracle.Tests/OracleWakeUpHintTests.cs index 84a84fd..d075c47 100644 --- a/tests/BackWave.Oracle.Tests/OracleWakeUpHintTests.cs +++ b/tests/BackWave.Oracle.Tests/OracleWakeUpHintTests.cs @@ -33,16 +33,21 @@ public async Task Enqueue_wakes_an_idle_pump_far_faster_than_the_poll_interval() var recorder = new ConcurrencyRecorder(); // The consumer node: its pump waits on DBMS_ALERT while it polls every 5 s. + var registered = await AlertRegistrationsAsync(); var consumerStore = NewStore(enableWakeUpHints: true); using var consumer = BuildHost(consumerStore, recorder, WakeGroup()); await consumer.StartAsync(); - // Let the pump reach its idle state and the DBMS_ALERT waiter register before the signal fires. - await Task.Delay(TimeSpan.FromMilliseconds(750)); + // A signal before the waiter registers is lost, so wait for the consumer's registration. A fixed + // delay was not enough when the whole suite loaded the machine. + await WaitForNewRegistrationAsync(registered); - // The producer node: a separate store (separate connections) enqueues the probe and signals. - using var producer = BuildHost(NewStore(enableWakeUpHints: true), new ConcurrencyRecorder(), group: null); + // The producer node: a separate store (separate connections) enqueues the probe and signals. Its + // first call opens its connections and checks the schema, so that call runs before the stopwatch. + var producerStore = NewStore(enableWakeUpHints: true); + using var producer = BuildHost(producerStore, new ConcurrencyRecorder(), group: null); var producerClient = producer.Services.GetRequiredService(); + await producerStore.GetJobAsync(Guid.NewGuid()); recorder.ArmProbe(); var stopwatch = Stopwatch.StartNew(); @@ -255,6 +260,39 @@ private static async Task RevokeAlertAccessAsync() await revoke.ExecuteNonQueryAsync(); } + // The session ids that hold a registration on the store's alert. Oracle keeps a row for a session that + // has ended, so a new registration is a session id that an earlier read did not return. + private static async Task> AlertRegistrationsAsync() + { + await using var admin = new OracleConnection(OracleTestDatabase.SystemConnectionString); + await admin.OpenAsync(); + await using var select = admin.CreateCommand(); + select.CommandText = "SELECT sid FROM sys.dbms_alert_info WHERE name = :name"; + select.Parameters.Add(new OracleParameter("name", + new SchemaRewriter(SchemaRewriter.DefaultSchema).HintAlertName.ToUpperInvariant())); + var sessions = new HashSet(); + await using var reader = await select.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + { + sessions.Add(reader.GetString(0)); + } + return sessions; + } + + private static async Task WaitForNewRegistrationAsync(HashSet before) + { + var deadline = DateTimeOffset.UtcNow + TimeSpan.FromSeconds(10); + while (DateTimeOffset.UtcNow < deadline) + { + if ((await AlertRegistrationsAsync()).Except(before).Any()) + { + return; + } + await Task.Delay(50); + } + Assert.Fail("Timed out waiting for the consumer to register its DBMS_ALERT waiter"); + } + private static async Task WaitForAsync(Func condition, TimeSpan timeout, string description) { var deadline = DateTimeOffset.UtcNow + timeout; diff --git a/tests/BackWave.Oracle.Tests/OutageTestSupport.cs b/tests/BackWave.Oracle.Tests/OutageTestSupport.cs new file mode 100644 index 0000000..0997c20 --- /dev/null +++ b/tests/BackWave.Oracle.Tests/OutageTestSupport.cs @@ -0,0 +1,97 @@ +using BackWave.Hosting; +using Oracle.ManagedDataAccess.Client; + +namespace BackWave.Oracle.Tests; + +/// The health waits and the private-user teardown that the outage tests share. +internal static class OutageTestSupport +{ + // A halt of one pump counts too, so the check reads the partly halted groups as well. + public static void AssertNotHalted(BackWaveHealth health, string group) + { + var halted = health.HaltedGroups.TryGetValue(group, out var halt) + || health.PartiallyHaltedGroups.TryGetValue(group, out halt); + Assert.False(halted, $"the outage fail-stopped the group: {halt?.ExceptionType}: {halt?.Message}"); + } + + // Polls the condition and fails the test the moment the group halts, so a fail-stop reports its cause + // instead of a timeout. + public static async Task WaitForAsync( + Func> condition, BackWaveHealth health, string group, string description, + TimeSpan? interval = null) + { + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10); + while (DateTime.UtcNow < deadline) + { + AssertNotHalted(health, group); + if (await condition()) + { + return; + } + await Task.Delay(interval ?? TimeSpan.FromMilliseconds(50)); + } + AssertNotHalted(health, group); + Assert.Fail($"Timed out waiting for: {description}"); + } + + // Samples the health state until the group is degraded. The next clean poll clears the mark, so the + // sample interval stays well under the poll interval. + public static async Task WaitForDegradedAsync(BackWaveHealth health, string group, string description) + { + var cause = ""; + await WaitForAsync(() => Task.FromResult(health.DegradedGroups.TryGetValue(group, out cause!)), health, + group, description, TimeSpan.FromMilliseconds(5)); + return cause; + } + + public static async Task LockUserAsync(string user) + { + await using var admin = new OracleConnection(OracleTestDatabase.SystemConnectionString); + await admin.OpenAsync(); + await using var lockUser = admin.CreateCommand(); + lockUser.CommandText = $"ALTER USER {user} ACCOUNT LOCK"; + await lockUser.ExecuteNonQueryAsync(); + } + + // Ends every session of the private user, a killed one too, and waits until none is left. A killed + // session holds its process until its client calls again, and the client is gone at teardown. + public static async Task DisconnectSessionsAsync(string user) + { + await using var admin = new OracleConnection(OracleTestDatabase.SystemConnectionString); + await admin.OpenAsync(); + + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30); + while (true) + { + var sessions = new List(); + await using (var select = admin.CreateCommand()) + { + select.CommandText = "SELECT sid, serial# FROM v$session WHERE username = :username"; + select.Parameters.Add(new OracleParameter("username", user.ToUpperInvariant())); + await using var reader = await select.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + { + sessions.Add($"{reader.GetInt32(0)},{reader.GetInt32(1)}"); + } + } + if (sessions.Count == 0 || DateTime.UtcNow > deadline) + { + return; + } + + foreach (var session in sessions) + { + await using var disconnect = admin.CreateCommand(); + disconnect.CommandText = $"ALTER SYSTEM DISCONNECT SESSION '{session}' IMMEDIATE"; + try + { + await disconnect.ExecuteNonQueryAsync(); + } + catch (OracleException exception) when (exception.Number is 31 or 30) + { + } + } + await Task.Delay(200); + } + } +} diff --git a/tests/BackWave.Pro.Dashboard.Tests/NoConfigReload.cs b/tests/BackWave.Pro.Dashboard.Tests/NoConfigReload.cs new file mode 100644 index 0000000..19dd97e --- /dev/null +++ b/tests/BackWave.Pro.Dashboard.Tests/NoConfigReload.cs @@ -0,0 +1,12 @@ +using System.Runtime.CompilerServices; + +namespace BackWave.Pro.Dashboard.Tests; + +internal static class NoConfigReload +{ + // WebApplication.CreateBuilder watches appsettings.json for changes. On macOS, starting that + // file watcher sometimes blocks forever, and the whole run hangs. No test here reloads config. + [ModuleInitializer] + internal static void Initialize() => + Environment.SetEnvironmentVariable("DOTNET_hostBuilder__reloadConfigOnChange", "false"); +} diff --git a/tests/BackWave.Pro.Mcp.Tests/NoConfigReload.cs b/tests/BackWave.Pro.Mcp.Tests/NoConfigReload.cs new file mode 100644 index 0000000..bec24f9 --- /dev/null +++ b/tests/BackWave.Pro.Mcp.Tests/NoConfigReload.cs @@ -0,0 +1,12 @@ +using System.Runtime.CompilerServices; + +namespace BackWave.Pro.Mcp.Tests; + +internal static class NoConfigReload +{ + // WebApplication.CreateBuilder watches appsettings.json for changes. On macOS, starting that + // file watcher sometimes blocks forever, and the whole run hangs. No test here reloads config. + [ModuleInitializer] + internal static void Initialize() => + Environment.SetEnvironmentVariable("DOTNET_hostBuilder__reloadConfigOnChange", "false"); +} diff --git a/tests/BackWave.Tests/NoConfigReload.cs b/tests/BackWave.Tests/NoConfigReload.cs new file mode 100644 index 0000000..2d12e8d --- /dev/null +++ b/tests/BackWave.Tests/NoConfigReload.cs @@ -0,0 +1,12 @@ +using System.Runtime.CompilerServices; + +namespace BackWave.Tests; + +internal static class NoConfigReload +{ + // WebApplication.CreateBuilder watches appsettings.json for changes. On macOS, starting that + // file watcher sometimes blocks forever, and the whole run hangs. No test here reloads config. + [ModuleInitializer] + internal static void Initialize() => + Environment.SetEnvironmentVariable("DOTNET_hostBuilder__reloadConfigOnChange", "false"); +}