Repository navigation
fix(oracle): degrade the worker group on network outage faults instead of halting it - #14
Merged
Merged
Conversation
…erver and Oracle adapters SqlClient 6.1.1 and Oracle.ManagedDataAccess.Core 23.9.1 bring in Pkcs 8.0.1 and 9.0.4. The advisories GHSA-846f-f764-76mx (high) and GHSA-2fmv-rv2c-59x7 (moderate) make restore fail with NU1903 and NU1902. Each target framework now pins the patched version of its own line.
Dashboard.Tests hung for more than 25 minutes in FileSystemWatcher.StartRaisingEvents on macOS. This is the same hang that Hosting.Tests fixed. Dashboard, Pro.Dashboard, Pro.Mcp, Oracle, and the core tests now set the same switch.
A worker group on Oracle fail-stopped on the first connect that failed during a store outage, and it did not claim again until the process restarted. ODP.NET raises codes for these faults that the transient list did not have: - ORA-12545 when the database host name stops resolving - ORA-50201 when the connection is refused (the database is down) - ORA-50000 when the connect times out - ORA-01089 and ORA-01109 while the database shuts down and opens again These codes now degrade the group, and it retries on the next poll. The pump runs the migration again on each poll, so a migration that fails on one of these codes also degrades the group. A wrong Data Source can give the same codes. A mistyped host name gives ORA-12545, and a wrong service name gives ORA-50201. A worker group with such a store now stays degraded and does not halt. The health check reports it as Degraded, not Unhealthy, so a readiness probe that accepts Degraded reports the deployment as ready. The release notes must state this tradeoff. The Postgres adapter already behaves this way. Fixes #12
The sample accepts BackWave:Store=Oracle, and bwv launch --store oracle starts a private Oracle container on a fixed loopback port. With a fixed port, a run can stop and start the container to reproduce a store outage.
The DNS case used a single-label host name. A DNS search domain or a resolver that rewrites NXDOMAIN can make that name resolve, and then the test gets ORA-50000. The timeout case used 10.255.255.1. A VPN or a cloud network that routes 10/8 can reject it fast with a different code. The tests now use bw-no-such-host.invalid (RFC 6761) and 192.0.2.1 (TEST-NET-1, RFC 5737).
The Oracle store defaulted to localhost:15210, which is the shared test database of the repo-root compose file. The sample compose file now has an Oracle service on port 15211, and the default points at it. The sample README lists the Oracle store.
A pool with no free connection raises ORA-50012, not ORA-50000. ODP.NET 23.9.1 sets IsTransient to false on it, so a busy node halted its worker group. Add 50012 to the transient connectivity codes and fix the comment on 50000. A live test holds the one connection of a pool of one and checks the store classifies the fault transient.
The header now covers a store outage too: a pump that loses its database degrades and retries. The 50xxx codes come from ODP.NET, so the list is of ORA or ODP.NET codes. A wrong service name on a live listener also raises ORA-50201 (measured against the test database). A mistyped host name raises ORA-12545, so it leaves the group degraded and never halts it.
… are killed The outage tests only showed that a group degrades instead of halting. No test showed that it claims and runs jobs again, which is the symptom in issue #12. The new test runs a real host as a private bw_recover user. It runs one job, kills every session of that user as SYSTEM, and checks that the group degrades with ORA-00028 and never halts. Then a new job must reach Succeeded on the same group. The kill is not IMMEDIATE. ODP.NET drops a session with a closed socket from the pool before it hands it out, so only a plain kill makes the next claim run on a dead pooled session. At teardown the test locks the user, so the pool cannot open a new session, and disconnects every session of the user before DROP USER.
The coordinated migration retries a transient connect fault up to eight times. Since the connect-timeout code is transient, a host that does not answer costs a full Connection Timeout on each attempt. With the default 15 s timeout one migration call took about 2 min to fail, and a permanent misconfiguration was reported late. The retry now also stops once 5 s have passed, and the last fault propagates. The eight attempts still ride out a cold-boot handshake storm, because those faults come back fast and the backoff adds only 1.4 s. An auto-migrating store runs the migration again on its next poll, so an outage still retries at the pump.
…budget applies One slow transient fault, for example a TNS connect timeout with the default 15 s Connection Timeout, used the whole 5 s budget on the first attempt, so the migrator did not retry at all. An auto-migrating store tries again on its next poll, but a caller that runs MigrateAsync once at startup has no next poll, and that caller lost the cold-boot retry. The first three attempts now always retry. The 5 s budget applies after them, and the limit of eight attempts stays. The rule is one internal method, so a test covers it without a slow connect.
…ave degraded A probe with ODP.NET 23.9.1 showed what each kind of typo in a connection string raises: - A descriptor with a bad keyword raises ORA-50201 "failed to parse connect string". - A mistyped EZConnect string or TNS alias raises ORA-12154. - A wrong attribute name fails the open with an ArgumentException (ORA-50008). - A wrong password raises ORA-01017. The first two are transient, so they leave the group degraded. The last two halt it. The comment now lists all of them in one place. OracleConnectionStringBuilder catches only the attribute-name typos, and those already halt, so the store does not parse the string up front.
… one exact code The outage tests expected one exact code per endpoint. A VPN or a CI runner can answer 192.0.2.1 with ICMP unreachable, and a kill on a busy session can surface as 3113 or 3135 instead of 28. The classification is still correct then, but the tests failed. The tests now write the code to the test output and assert that the adapter classifies it transient. A code that is not in the list still fails them. The comments keep the code measured for each endpoint.
…jected login halts it The degrade test stopped at the first degraded mark and checked for a halt only once, so a later poll that halted the group went undetected. The test now watches three poll intervals after the mark, and it counts a halt of one pump too. A new negative test points the store at the live listener with a user that does not exist. The login fails with ORA-01017, which is not an outage, so the group must halt. The test fails if 1017 is added to the transient list. It does not use a wrong password for a real user, because repeated failed logins can lock that account.
The summaries make sense without it, and the code does not cite tracker numbers.
…dy path The store runs the migration while it holds its ready gate, and every store call that waits at the gate runs the migration again after it. The three-attempt minimum let one call hold the gate for three full connect timeouts, which is 45 s with the default 15 s Connection Timeout. Each waiting enqueue then added the same time again. The 5 s retry budget now applies only when the store migrates on first use. The pump migrates again on its next poll, so the store path does not need more attempts. The public MigrateAsync has no next poll, so it keeps all eight attempts with no time limit, as it did before this branch. A test points a store at a host that does not answer and runs a claim and an enqueue together. Both now fail in about 12 s. Without the budget the enqueue waited 35.8 s.
… refused The session-kill recovery test raises ORA-00028, which was transient before the outage fix, so it passes without that fix. This test raises a code the fix adds. The store connects through a free loopback port that nothing listens on, so ODP.NET gets a refused connect (ORA-50201). The group must degrade and not halt, and an enqueue in that window must fail with a transient fault. Then an in-process TCP forwarder opens the port in front of the real database. A job enqueued after that must succeed on the same group. Last, the forwarder closes every live connection, and one more job must succeed. Without the fix the group halts on ORA-50201 at the first poll.
The hinted-wake test waited a fixed 750 ms for the consumer to register its DBMS_ALERT waiter. A signal before the registration is lost, and under load 750 ms was not always enough. The test now waits until a new session appears in sys.dbms_alert_info for the store's alert. The producer store also opens its connections and checks the schema before the stopwatch starts, so the latency measures the hint and not a cold start. The test passed 40 of 40 runs alone on a private Oracle container. The earlier timeouts occurred only while another worktree ran the Oracle suite on the shared database.
Two nodes that create the same index at the same time can deadlock in the library cache. Oracle rolls back one statement with ORA-04020, and the migrator retried only connectivity faults, so that node failed its start. The 16-node first-boot storm test failed about 1 run in 3 with this error on a private container. The scripts are idempotent, so the coordinated migrator now retries ORA-04020 like a connectivity fault. The uncoordinated path is still caller-serialized and does not retry. Its storm test rides out the same codes. The storm test class passed 15 of 15 runs after the change.
The store migrates on first use with a 5 s retry budget, but one DDL lock wait can last 30 s (ddl_lock_timeout). So an ORA-04020 can arrive after the budget and go out through EnsureReadyAsync. The store classifier accepted only ORA-00060 as a deadlock, so the worker group halted. OracleFaultCodes.IsDeadlock now names 60 and 4020, and the migrator and the store classifier both use it. A new test raises each code in the database and makes sure that the store marks it transient. It failed for 4020 before this change.
The two recovery test classes had their own copies of the halt check, the wait loops, and the private-user teardown. Their halt checks did not read PartiallyHaltedGroups, so a halt of one pump went unseen. The copy in OracleTransientFaultTests read it. OutageTestSupport now holds one copy of each helper, and its halt check reads both halted sets.
…est database The test hard-coded localhost:15210 and ignored BACKWAVE_ORACLE_DSN, so on any other database it got ORA-50201 and failed. It stopped at the first degrade, so a listener fault on an early poll failed it too. Its class has no collection, so it ran in parallel with the 16-node migration storm. The test now takes its endpoint from OracleTestDatabase.ConnectionString with a user that does not exist, waits until the group halts, and lives in its own class in the oracle collection. The outage tests that need no database still run in parallel.
A review asked to keep the port bound and not accept connects. On macOS a bound socket that does not listen makes the connect wait for its timeout, so the test got ORA-50000 instead of ORA-50201 and failed 20 of 20 runs. Only a port that nothing holds gets a refused connect. The comment now records this. If another process takes the port, the test fails and never passes by mistake.
The coordinate doc named only the connection faults, and it called the script concurrency-safe on its own. The coordinated path also retries ORA-00060 and ORA-04020, and the uncoordinated path can surface them.
An & in a query started a substitution prompt. sqlplus then took the next input line as the value, so SELECT 'a&b' FROM dual returned aEXIT. SET DEFINE OFF now runs first, and the same query returns a&b.
A bwv run uses neither the shared test container on 15210 nor the sample compose container on 15211. The store sentence in features/README.md missed a comma.
The storm helper rode out transient faults for both paths, so the coordinated storm passed even with no retry in the migrator. The coordinated path now calls MigrateAsync directly, and only the uncoordinated path keeps the helper retry. With the deadlock retry removed from the migrator, the storm class failed on its second run with ORA-04020. With the retry, it passed 12 of 12 runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12.
An Oracle worker group halted (fail-stop) when the database became unreachable, for example with ORA-12545 when the host name stopped resolving. ODP.NET sets
IsTransient=falseon these faults, and the adapter's classifier did not list the codes. Now the group degrades and polls again, and it runs jobs when the database answers.Changes
OracleFaultCodeslists the outage and restart codes, each one measured with ODP.NET 23.9.1: 12545, 50201 (connection refused), 50000 (connect timeout), 50012 (pool exhausted), 1089 and 1109 (restart).OracleFaultCodes.IsDeadlock(ORA-00060, ORA-04020) is used by the migrator and the store. A 16-node first-boot migration failed about 1 run in 3 with a library cache deadlock (ORA-04020). This fault was already on main.MigrateAsynccall keeps 8 attempts.bwv launch --store oracleand a sample Oracle service on port 15211.Tradeoff
A wrong Data Source gives some of the same codes (12545, 12154, 50201). So a misconfiguration leaves the group degraded and does not halt it. The Postgres adapter behaves the same way. A rejected login (ORA-01017) and a wrong connection string attribute still halt the group. The release notes must state this.
Also in this PR
Two commits from local main that CI needs:
7adf3b1: pins the patched System.Security.Cryptography.Pkcs. Without the pin, restore fails with NU1903 (GHSA-846f-f764-76mx).4ba2135: turns off config reload in every test project that builds a host.Tests
OracleRefusedConnectionRecoveryTests: this is the regression test for WorkerGroupService does not classify OracleException during pump #12. The group boots while connects are refused (ORA-50201), degrades, and runs jobs after a TCP forwarder starts, and again after the forwarder drops every connection. It fails without the fix.