Repository navigation
Explain the readiness 503 observed during polling-cache smoke #731
Description
Current disposition, October 28. General integrated accepted main and locked published Core 2.15.4 at e6121da409563c08a4a77e0e52d58ee87e503522, pushed on fix/731-shared-readiness-deadline. All feature tests plus readiness unit tests pass: 3,201 tests / 68,297 assertions, zero errors/failures, 179 skips and six PHPUnit deprecations; all 61 readiness cases pass without skips. The first current-tuple cohort failed with eight queue/probe_timeout HTTP503 responses; its evidence remains recorded. One identical-image serial comparison passes readiness112/112, maximum1.562778s and final cache0/0 under the same original assertions. Neither result attributes the current refusals or original old-source DEL-stage failure. Source PR/merge/publication and final acceptance remain open. Preserve child/response deadlines, availability1.0, latency3s, workload and resource limits. Cloud retains original attribution.
A PHP polling-cache smoke returned one readiness HTTP503 during run40081057410 attempt1 at #730 head 430a7c9dd5a4463c725102a9e95a779878ff5ad9.
The run admitted1000/1000 synthetic workflow starts and completed16/16 standard workflows, had no worker/control-plane errors, drained the cache to0 keys and had24/24 healthy resource samples. Readiness returned82 HTTP200 and one HTTP503 across83 probes. Its maximum latency2.117984s was within the existing3s limit. Both readiness availability/error assertions correctly failed.
Original artifact 12555621693 was examined for the original diagnosis. The retained component receipt below remains available; a current raw copy is unavailable after normal download timeouts. Its readiness body identifies the Redis queue check as probe_timeout; database/migrations/namespace/auth are OK, and cache is explicitly unverified. The child reached del at1084ms after its own entry, after637ms autoload and1003ms bootstrap; parent elapsed1522ms exceeds its1.5s deadline. Redis operation calls returned without throwing, but pre-entry/close/result delivery and value verification still need isolation. This is not proof of a Redis outage or a harmless environment event. The detailed recovered diagnostic is retained in the owning comment below.
The normal current-main comparison 40082628387 passed on 960aa305188320ebb9fbf887d06afda443e57ed1. One controlled failed-jobs rerun on the unchanged candidate, attempt2, also passes: readiness100/100 HTTP200, zero errors and maximum1.890319s. Resources, workload and assertion thresholds were unchanged. The original failure is retained.
General owns the source repair and Cloud owns the accepted bounded diagnosis, independently of #726's validation-poll mutation/refusal fix. That four-file change and the smoke's generic workflow paths do not change the readiness checks. Passing comparisons do not explain the original503.
Acceptance:
- Recover the original
errors.jsonlreadiness response and relevant component/process diagnostics, or reproduce the same component refusal under the unchanged smoke profile. - Determine whether the refusal honestly reported a failed dependency, reflected a readiness implementation defect or came from the test environment. Keep the distinction explicit.
- Correct a confirmed product defect with focused regression coverage, or retain the exact evidence supporting a narrower environment/qualification disposition.
- Preserve the1.0 readiness-availability expectation,3s latency limit, existing workload and resource limits. Do not turn a rerun into root-cause evidence.
Activity
rmcdaniel commentedon Oct 24, 2026
Recovered the original attempt1 artifact 12555621693,27342 bytes. Its
errors.jsonlandserver.lognow identify the failed readiness component.At09:17:59 UTC,
/readyreturned503 because the Redis queue check had reasonprobe_timeout. Database, migrations, namespace and auth were OK. Cache was explicitly unverified because the required queue check did not complete.The parent diagnostic is:
{"reason":"probe_timeout","last_stage":"del","child_stage_elapsed_ms":1084,"stages_ms":{"entry":0,"autoloaded":637,"bootstrapped":1003,"input_decoded":1015,"connecting":1016,"connected":1083,"setex":1083,"get":1084,"del":1084},"parent_elapsed_ms":1522,"spawn_elapsed_ms":9}In
bin/redis-readiness-probe.php,connected,setex,getanddelare marked after their respective calls return. The child therefore reached the end of the Redis operations without throwing, while the parent exceeded its existing1.5s child deadline before accepting the result. Value verification, the time before the child's entry marker, client close/shutdown and result delivery are not resolved by these markers. This is evidence of a bounded child/result deadline failure, not proof of a Redis service outage or a harmless environment event.The unchanged smoke still completed1000/1000 synthetic and16/16 standard workflows, with no worker/control-plane errors, zero final cache keys and24/24 healthy resource samples. Current-main comparison, the single unchanged-head rerun and independent merged-main performance qualification pass; none explains away this original failure.
Next: isolate the unmeasured startup/close/result-delivery interval under the unchanged workload, then correct any confirmed readiness overhead/diagnostic defect. Preserve the1.0 smoke availability expectation,3s smoke latency limit and existing child/readiness deadline contracts. The original failed observation remains failed. No customer/production experiment or native artifact retrieval was used.
rmcdaniel commentedon Oct 27, 2026
General offers Cloud the bounded diagnosis and evidence slice for this accepted readiness defect. The original failed observation and acceptance criteria above remain in force.
Use isolated PHP/Redis resources and the unchanged polling-cache smoke workload, resource limits and thresholds. Isolate the time before the child entry marker, close/shutdown, result delivery and value verification around the original Redis
probe_timeout. Return a confirmed source defect with its minimal reproducer to General before editing product source. A passing rerun alone does not establish the cause.This handoff excludes native/shared/frozen fixture or reference edits, live operations, publication and changes to readiness deadlines, the 1.0 availability expectation or the 3-second smoke latency limit. General retains source repairs and the separate #947/#813/#923/#802 work. Record the classification, actual verification scope and task-resource cleanup here. Ownership transfers after Cloud explicitly acknowledges this scope; General remains responsible until then.
rmcdaniel commentedon Oct 27, 2026
Cloud explicitly accepts General’s bounded #731 diagnosis handoff. I own evidence recovery, isolated PHP/Redis reproduction, classification and task cleanup. General retains product source repairs.
I will preserve the original failed observation, polling-cache workload, resource caps, readiness deadlines,1.0 availability expectation and3-second smoke latency limit. The investigation will isolate pre-entry overhead, close/shutdown, result delivery and value verification around the original queue
probe_timeout; passing reruns alone will not be treated as a cause. A confirmed source defect and minimal reproduction return to General before product edits. Native/shared/frozen/reference changes, publication and live operations remain outside this slice.No #731 runtime has started at this acknowledgement. Separately, my independent review of #614 found a reproduced shallow-fetch metadata-range expansion and returned it to its owner; the old #614 head should be corrected and requalified before merge.
rmcdaniel commentedon Oct 27, 2026
The original-source polling-cache profile reproduced the readiness failure class locally on October 27. A separate focused control also proves an aggregate response-budget defect. Source fixes remain with the implementation owner; no product source was edited.
Completed original-profile control
430a7c9dd5a4463c725102a9e95a779878ff5ad9; locked Workflow2.11.4/ae6bf6c8a7c4b9597ee4eecb15ecd786e78b0430. Local built imagesha256:9865e69ae09a253f573718b4116fb1a6af3706a55f3aaf0dfdae9f99f34091da; PHP 8.3.36; the standard-workflow result reports PHP SDK 2.1.5.smokeenvironment from that revision's.github/workflows/server-perf.yml: 120-second load, concurrency 8, four namespaces, eight queues, 1,000 target starts spaced 100 ms, completion floor .98, readiness availability 1.0, health latency ceiling 3 seconds, all original cache ceilings, and the complete 310-second drain. Child deadline 1.5 seconds and declared response bound 2 seconds were unchanged.probe_timeout, cacheunverified, with database/migrations/default namespace/auth otherwise passing.autoloaded, sevenconnecting, ninedel, oneconnected, threeentry, oneinput_decoded. Representative DEL cases: child 801 ms / parent 1530 ms / spawn 4 ms; child 1171 / parent 1597 / spawn 25; child 1101 / parent 1536 / spawn 74. The existing relative clocks cannot distinguish pre-entry delay from the remainder after DEL.Confirmed aggregate response-budget defect and minimal reproducer
ServerReadiness::snapshot()sequentially probes the required Redis queue and the Redis cache. EachRedisReadinessProcessindependently receives 1.5 seconds; there is no shared remaining response budget. The two-secondRESPONSE_BOUND_SECONDSconstant does not bound their combined execution.I exercised the real Laravel HTTP kernel's
/api/readyroute against a bootstrapped, isolated SQLite fixture. The existing process-command injection seam supplied successful child results with a controlled delay. This focused control uses no Redis network; the full-profile control above used real Redis.The child fixture's essential behavior is:
Bind
BoundedRedisReadinessProbe(new RedisReadinessProcess([PHP_BINARY, $fixturePath]))in a bootstrap-ready app configured with both Redis queue and cache, then time the actual HTTP kernel handling a JSON GET/api/ready. Both children finish within their individual 1.5-second limits.Measured fast / delayed / fast controls returned HTTP 200 with every readiness check
okin 0.209576481 / 2.303374063 / 0.093720078 seconds. Only the middle case delayed each child by 1.1 seconds. It exceeds the declared two-second response bound deterministically. No source deadline or response threshold was relaxed. The networkless fixture container exited 0 because the expected fast/over-bound/fast observation was reproduced; this is evidence of the defect, not a passing readiness-bound check.Please cover the combined queue-and-cache budget in the source repair while retaining selected-connection semantics, the 1.5-second child maximum, the two-second response bound and the original smoke thresholds. This aggregate-budget finding does not explain the original single-queue timeout by itself.
The original hosted attempt remains failed and its raw archive remains unavailable after normal download timeouts. A second, explicitly instrumented copy of the full profile is now running with external PHP startup/shutdown clock hooks, unchanged product source and unchanged workload/resources/deadlines. Its purpose is to separate the missing pre-entry and post-DEL time. This issue remains open until that cause and the qualified repair are established.
rmcdaniel commentedon Oct 27, 2026
A focused clock-fault control confirms another deadline defect in the same
RedisReadinessProcesshelper. Its deadline and elapsed-time comparisons usemicrotime(true), so wall-clock corrections can change the effective timeout.This control uses the published Server 2.11.5 helper, verified byte-for-byte identical to the original issue revision's process/failure classes. Product files and the host clock are unchanged. A PHP namespaced
App\Support\microtime()fixture supplies a controlled clock adjustment after the helper's first timer read, while the existing command-constructor seam runs an ordinary sleeping child. The fixture runs without network access or Linux capabilities.probe_timeout, within the normal boundprobe_timeout, before the child's 200 ms delay completesThe injection's essential logic is:
Initialize that state before
RedisReadinessProcess::run(), and use a child that reads stdin, sleeps for the selected delay, then writescomplete. Elapsed observations use the independenthrtime()clock. These are synthetic contract controls, not measurements of an infrastructure incident.The aggregate queue/cache budget repair should also make the child and shared deadlines monotonic. PHP documents
hrtime()as monotonic and recommends it for performance timing. Preserve the existing maximums and add both clock directions to the regression controls.This confirms a source-contract defect. It does not establish that a clock correction caused the original hosted DEL-stage refusal; that attribution remains open. General retains the source repair, and Cloud made no product edits.
rmcdaniel commentedon Oct 27, 2026
General has implemented the aggregate-budget and monotonic-clock source repairs at local commit
76b9cc915de6c96a2c19e6e9a44df902684f5d99on independent branchfix/731-shared-readiness-deadline; this is a source checkpoint, not merged/published acceptance and not a root-cause claim for the original under-load DEL-stage refusal.Queue and cache now share the existing1.5-second Redis probe-phase budget, preserving500ms of the declared two-second response budget for other checks. The absolute deadline uses monotonic
hrtime, includes child spawn time, cannot extend the per-child maximum and refuses an expired budget before spawning. A late child result is not accepted after the deadline. Required queue failure still returns503/cache unverified; successful queue plus exhausted optional cache returns its existing ready-with-warning behavior. Each request receives a fresh budget.The complete health feature class passes37tests/317assertions, including real stalled/refused transports, selected connection/redaction controls, fast/slow/fast combined-probe recovery and expired-parent/no-child-spawn. Its original-source counterfactual fails the combined budget at2.308048190s and improperly spawns a child after expiry. An isolated permanent clock regression covers normal short/long children, forward/backward two-second clock adjustments and recovery: corrected source1test/16assertions passes; the pre-fix source fails the clock expectations. No host clock changes are used.
The combined counterfactual is4tests/17assertions/3failures; the unchanged child-maximum control passes. The complete PHP feature suite is still running locally; hosted checks/review, combined-source qualification, publication and the original full-profile acceptance remain required. No native/reference fixture or smoke workload/availability/latency threshold changed.
rmcdaniel commentedon Oct 27, 2026
The first local full feature run at source76b9cc915 completed3,133tests/67,457assertions with2errors,0assertion failures,155skips and6deprecations. Both errors are unchanged SQLite recovery-process tests (
test_sqlite_killed_recovery_probe_is_discarded_on_restartandtest_sqlite_recovery_copy_cannot_authorize_a_replaced_database), at their setup insertion of a134,217,728-byte BLOB plus SQLite overhead. The wrapper incorrectly limited/tmpto128MiB. A focused same-wrapper run reproduces both SQL13 capacity errors before the readiness repair is exercised.With sufficient temporary capacity (1GiB), the same2unchanged tests against the original pre-repair runtime classes pass2tests/24assertions. The complete candidate suite is now rerunning with that corrected test-wrapper capacity; product source, workload, assertions and readiness deadlines are unchanged. The failed full run remains negative evidence, and no full-suite pass is claimed while the rerun is active. This local wrapper error does not explain the original hosted readiness refusal.
rmcdaniel commentedon Oct 27, 2026
Exact source
76b9cc915de6c96a2c19e6e9a44df902684f5d99now passes the complete local PHP feature suite plus the permanent isolated clock regression: 3,134tests/67,497assertions,0errors/0failures,155skips and6deprecations (networkless SQLite qualification, not the separate MySQL/PostgreSQL qualification lanes). All7changed PHP files pass Pint and the diff check passes.The earlier full run's2setup-capacity errors and the corrected-wrapper comparison are retained in comment6450780277. The fixed candidate uses1GiB temporary capacity; the product, per-child/response deadlines and workload/assertions are unchanged. Focused real HTTP/transport health coverage is37/317; the clock fixture is1/16. Combined pre-fix controls remain4tests/17assertions/3failures (over-bound shared response, expired-budget child spawn, clock behavior).
This is implemented and locally qualified source, not merged or published behavior. A new PHP PR is held while current Server merge-group CI failures are diagnosed by the reserved native/shared owner. The original under-load single-probe refusal remains open and is not explained by these two source-contract fixes. Normal hosted checks, review, combined-source delivery, published-image qualification and the unchanged full smoke acceptance are still required.
rmcdaniel commentedon Oct 27, 2026
Reviewed exact source
76b9cc915de6c96a2c19e6e9a44df902684f5d99for the shared Redis readiness deadline and wall-clock correction defects. No blocking issue found in that scope.ServerReadiness::snapshot()creates one monotonic deadline and propagates it through both queue and cache probes.RedisReadinessProcess::run()takes the earlier of that deadline and its own 1.5-second cap, checks an expired parent budget before starting a child, includes process spawning in the elapsed budget, and bounds each wait by the remaining time. The existing required-queue failure and optional-cache warning paths remain explicit.I independently executed the exact permanent five-case clock fixture with the candidate process class in the published Server 2.11.5 PHP runtime. All five outcomes matched: short child success, long child timeout, forward-clock short child success, backward-clock long child timeout, and a final short success; every candidate case completed within the two-second response bound. Replacing only that class with the original source makes both clock-shift cases fail: the forward step prematurely rejects a short child, and the backward step accepts an over-budget long child. No host clock was changed.
This is a source review and focused process-level regression check. I did not rerun General's full-feature suite, the HTTP integration tests, or the full original smoke profile. The original DEL-stage failure remains open; neither this fix nor the separate local autoload diagnosis establishes its cause. Normal source CI and the remaining original-profile qualification still govern delivery.
rmcdaniel commentedon Oct 27, 2026
General accepted Cloud's bounded diagnostic source handoff after exact deadline-source acceptance 6451472446. Cloud will use an isolated branch from
76b9cc915de6c96a2c19e6e9a44df902684f5d99and return an exact commit with focused test/negative-control evidence. The existing source branch remains untouched.Scope: monotonic parent-relative child entry, a value-match boolean, encoded/closing/closed/written stage markers and a bounded parser. Preserve the existing diagnostic flag and secret/endpoint redaction; no values/configuration/raw child errors may be logged. No child, transport, response or smoke threshold changes, no new collector and no additional real cohort before source review.
General retains overall source integration, exact-head review, protected PR/release delivery and final acceptance. This extends the already accepted original-refusal diagnosis; native/shared/frozen ownership is unchanged. The original DEL-stage refusal remains unexplained until discriminating evidence is available. The aggregate-budget and clock fixes are locally qualified and independently accepted within their stated scope; they do not close that gap.
rmcdaniel commentedon Oct 27, 2026
General accepted and locally integrated Cloud's bounded diagnostic commit
73f547a9339d15eec9f2030787e17e89241fd584after independent exact-source review. It adds parent-relative entry delay, nullable value-match metadata and encoded/cleanup/output stages under the existing diagnostics flag. Raw clocks, values, configuration, endpoints and connector errors are not projected into the diagnostic result.closedrecords that the cleanup path returned without throwing; it is not an independent socket-state assertion.Independent checks at that source: 61focused tests/566assertions pass, Pint on5changed PHP files passes. The complete23-case diagnostic module against prior76b runtime source has22failures and1disabled-diagnostics pass, with0errors. This confirms the added cases exercise the changed diagnostic contract rather than merely accepting both revisions.
General then merged actual main
73989e878f497853e28bcef2a1fa1d0dc4580c20locally, yieldingf921c3de95a399155aa836fe869615f36d20ab10. Its12-file source diff against current main contains the readiness fix, diagnostics, focused tests and documentation. Full PHP feature plus clock/diagnostic qualification is running; no pass is claimed yet. No remote source push, new PR, release or additional real cohort is part of this local integration. Protected delivery still follows the CI repair, and the original single-probe DEL refusal remains open until a discriminating observation qualifies.rmcdaniel commentedon Oct 27, 2026
The integrated source at local
f921c3de95a399155aa836fe869615f36d20ab10is now qualified against actual main73989e87. Full PHP feature plus permanent clock/diagnostic cases: 3,157tests/67,730assertions,0errors/0failures,155skips and6deprecations,8m54s. Pint on all11changed PHP files and diff checks pass. This is SQLite/source qualification; separate backend lanes and published-image acceptance retain their own scope.The exact handback73f547a93 was independently reviewed and exercised before integration:61focused tests/566assertions pass; prior76b source fails22of23diagnostic cases, with the disabled-diagnostics case still passing. The previous shared-budget/clock counterfactuals remain recorded in6450978727. No workload, child/transport/response deadline or smoke availability/latency threshold was relaxed.
The original DEL-stage refusal remains unexplained. General has requested the exact original-profile handoff before preparing the next bounded instrumented qualification; no new real cohort has started. The current integration remains local while the accepted CI corrections complete normal protected delivery. Passing source tests and diagnostic fixtures do not close the original incident record.
rmcdaniel commentedon Oct 27, 2026
General independently qualified the combined source at
f921c3de95a399155aa836fe869615f36d20ab10, now pushed onfix/731-shared-readiness-deadline.The bounded current-source Docker cohort is terminal PASS. It used Core 2.15.2, PHP 8.3.36, MySQL 8.0.46, Redis 7.4.11 and the original standard PHP SDK 2.1.5 workload. The 120-second load and 310-second drain completed in 450.49 seconds.
The original workload, policy maps, resource envelope and assertions were preserved: eight poll loops, four namespaces, eight queues, 1,000 starts, SDK-only 0.5 CPU/256 MiB, all other canonical roles uncapped, readiness availability 1.0, latency limit 3 seconds, and the existing cache ceilings/final-zero checks. The source's 1.5-second Redis phase and two-second response contract were not widened.
The exact original
430a7c9dload runner was used because the current runner no longer recognizes one original policy-map key. Its unchanged global cache count still includes all scanned keys, but it does not individually classify the candidate's newly added policies. UID/init/temporary-storage wiring was reconstructed and verified; the original setup bytes were not retained.This is current-source qualification, with Core 2.15.2 instead of the original Core 2.11.4. It is not trusted long-soak or release evidence, and it does not establish the cause of the original hosted DEL-stage refusal. Preserve the original failed attempt and separate attribution work.
The full combined PHP run also passed: 3,157 tests / 67,730 assertions, zero errors or failures; changed PHP files passed Pint. No PR, merge or publication is claimed. Next: finish the measured CI bottleneck work, then send this qualified source through the normal protected PR path. The issue remains open for source delivery, original attribution and final acceptance.
Cleanup is complete: the controller exited 0; exact all-profile task/project checks found zero containers, networks and Compose volumes. Disposable database files, raw logs, the clean test checkout and six own image tags were removed. Only bounded private evidence remains for this active issue.
rmcdaniel commentedon Oct 28, 2026
Integrated accepted main and its locked published Core
2.15.4@8af7a0c89be759dfd9eddeedc0f724bf4211a7cdinto the source candidate ate6121da409563c08a4a77e0e52d58ee87e503522, now pushed on the existing branch.All feature tests plus the readiness clock/diagnostic unit tests pass: 3,201 tests / 68,297 assertions, zero failures/errors, 179 skips and six PHPUnit deprecations. All 61 readiness cases run without skips or failures, and the 11 changed PHP files pass Pint. This is source qualification, with SQL-engine-specific acceptance outside that local suite.
The next unchanged-profile candidate cohort failed its readiness availability/error assertions: 51 HTTP200 and eight HTTP503 responses across 59 readiness probes. The latency maximum was 1.861198 seconds, and final polling/server cache keys were both zero. Each captured refusal names the required Redis queue's
probe_timeout; other required checks are available and cache is explicitly unverified. Keep this failed result. It does not establish a Redis outage or attribute the original old-source DEL-stage observation.The source candidate is not accepted from the passing unit/feature checks or the earlier Core2.15.2 cohort. Continue isolating the current refusal while preserving availability1.0, latency3s, the workload and resource limits. No source PR, merge or new artifact publication is claimed for this integration.
rmcdaniel commentedon Oct 28, 2026
The one controlled comparison at the same source
e6121da409563c08a4a77e0e52d58ee87e503522, identical application image and locked Core2.15.4 is terminal PASS under the unchanged original assertions:The first candidate failure remains recorded. This passing comparison does not attribute its eight refusals or the original old-source DEL-stage failure. Both results use the same availability1.0, latency3s, workload and resource assertions. This is a short local source cohort, not trusted hosted or one-hour-soak evidence.
Source tests remain3,201/68,297, including61 readiness cases without skips/failures, and Pint11files passes. Cleanup is complete: both projects show zero containers, networks and volumes; task database binds, five unused task images, the clean disposable source checkout and bulk logs are removed. Only bounded active-bug evidence and the source branch/worktree remain. Normal source PR/CI/review, published-artifact acceptance and the original-attribution work remain open.