Repository navigation
Description
General accepts the PHP investigation handed off by the native Server owner. This is an intermittent published-PHP qualification failure; its root cause is not yet established.
The owner handoff identifies update-validation / approved_update returning HTTP500 instead of the expected202 on the PHP side of the MariaDB10.11 corpus in #822 run40578777402, job122454294740, using published Server2.11.5. The earlier #816 run40561089085, MariaDB job122392116543 reported the same response shape on published2.11.4. Neighboring full corpus runs pass; that does not explain either500.
The retained PHP log is a tail and does not retain the correlated request exception. A later external-payload exception belongs to a different fixture and must not be used as this failure's diagnosis. This is distinct from a demonstrated native-source regression. The normal #822 failed-job rerun is owned by the native maintainer; General will retain the original failures regardless of that rerun's result.
Current evidence: the owner correlated a later state-updates HTTP 500 with SQLSTATE 40001 / InnoDB 1213 while inserting workflow_tasks, reported in the request-specific receipt. The original approved-update 500s above remain uncorrelated. Both original published MariaDB probe contracts passed once in a controlled baseline, which does not explain the intermittent failures.
General pushed 517e2647512e7f94722050ace5d33fecceb48ef6 with owned-root MySQL/MariaDB admission retries after rollback. Twenty real-engine controls and 82 related tests pass; source and qualification receipt. Follow-up head adaad79ac327fda10d40dd4df96f5ce5e7b7dc1f also fences the selected run inside retried admission. A real deadlock concurrent with continue-as-new previously let the direct path record an unknown-update rejection after the requested run changed. All 24 real-engine controls, 78 related tests and 54 controller tests pass; the strengthened final four race cases pass with 140 assertions. See follow-up qualification. The approved path already refused stale validation. Task-owned database fixtures/network have been removed.
Merged PR: #828, main 759abaa3b4eb7d931cf256b1aab2950768c2c0a9. The combined PR head passed 16 checks with 7 conditional skips; the normal merge group passed all 43 Shared Server jobs and its full PHP feature qualification. Source review and the specific Sonar maintainability decision are recorded. The admission correction is included in published 2.11.7; the exact full-corpus scope and remaining historical uncertainty are described below.
Published follow-through: Server 2.11.7 / Core 2.15.7 completed the first finite PHP-only MariaDB corpus with 74 of 82 cases passing and eight differences; update-validation and many-updates pass. The first record is retained. Subsequent classification separates four masked-route-cache fixture differences, missing writable /state for storage-test, the existing MariaDB #569 health GROUPBY defect whose allowance needs the truthful database-image hint, a default-prefix fixture assumption, and the #770 reference correction accepted in #867. Corrected observations and targeted follow-through preserve each scope. In particular, the operator tuple's third field is health namespace, not task namespace; metrics remain200. No task-isolation finding or repaired-health claim is supported.
The explicit PHP timing-helper handoff is implemented in PR #870, current combined head c108ed328de8360aa05589f01a66eed35c331977: elapsed time now covers the logical waiting request across lock-pressure attempts/backoff using a monotonic clock. Actual HTTP tests fail both retry-duration cases on the old helper and pass all four on the fix; focused Node26/26 and existing PHP retry6 tests/70 assertions pass, as do lint/actionlint. Frozen timing bounds, retry policy and product runtime are unchanged. Source review is accepted. The prior combined a9f head passed the full 43-job Shared Server run and entered the normal queue. Main #874 then merged, requiring one comparator-command conflict resolution. Current c108 merges main9670, retains the exact test argv union with one argument per line, and passes114 focused Node tests plus actionlint. Fresh current-head qualification is green: Shared40978243986 succeeds and all57 PR checks complete without failures. PR870 merged normally as6c1e63dd at01:14:27 UTC after successful protected merge-group Shared/PHP checks. Normal-main Shared40984638681/PHP40984638686/Sonar40984638635 all succeed on6c1e63dd. The historical500 disposition remains separate and this issue stays open. Timer helper/test bytes remain unchanged from5edd.
Fresh full follow-through is now 83/83 PASS, 2026-10-29 20:50:00.262–20:57:15.518 UTC, using PR870 runner5edd, actual published2.11.7/2.15.7, a fresh database/default prefix, stock container layout and truthful MariaDB hint. Exact full-record receipt and limits. All82 old cases remain (81 old fixture hashes unchanged; accepted#867 updates waterline-import); the already-merged child-descriptions fixture is the83rd. Known published-defect allowances and MariaDB#569 health500 remain modeled; metrics are200. No lock-pressure retries were reported in this full run.
Next action: preserve the accepted first/full records and complete cleanup of PR870's task resources now that protected merge and normal-main Shared/PHP/Sonar qualification have passed. Retain a narrowly scoped disposition for the original historical approved-update500s, whose exception remains missing. A passing rerun does not explain their cause; this issue remains open.
Acceptance:
- Preserve the original response, exact published PHP/Core tuple, database version, fixture runner and workload; qualify whether the input and expected202 remain valid.
- Reproduce or retain correlated evidence that distinguishes product storage/validation failure from fixture/environment failure. Do not assign a cause from a neighboring log entry.
- Correct any confirmed product defect with focused regression coverage, normal source qualification and published-consumer follow-through. Keep required checks and original failing evidence.
- Verify the affected full MariaDB corpus after repair; a passing rerun alone does not close this investigation.
Activity
rmcdaniel commentedon Oct 27, 2026
General accepted the published-PHP investigation from the Rust owner. Original artifact metadata independently confirms artifact
12773370526,mysql-execution-mariadb-10.11-corpus, run40578777402, at source0e7e955244d7a30fb19694bb04c263b90c44e582. The archive is 15,382 bytes and is not expired. Its normal GitHub download timed out, so I have not independently inspected its contents or correlated the HTTP 500 with an exception.The reports for published PHP 2.11.4 and 2.11.5 remain separate evidence. The unrelated payload exception in the retained log tail is not a root-cause finding. A passing failed-job rerun would not establish the cause. Next: recover the original request/case correlation and exercise that published-PHP MariaDB path independently, retaining any first failure before changing code.
rmcdaniel commentedon Oct 27, 2026
The Rust owner supplied a later, request-correlated report from run 40578777402, attempt 2, job 122482789008: the published PHP 2.11.5 record returned HTTP 500 for the
state-updatesfixture's selected-runset_payloadupdate at 21:40:40 UTC. The reported PHP exception was SQLSTATE40001, MariaDB error1213, on insertion intoworkflow_tasks, with maintenance running. The published-PHP recording failed before the native recording ran. This is an owner-reported correlation; General has not independently recovered that log. It does not establish the exception behind the earlierapproved_updatefailures.Source inspection traced remote workflow updates through Server's
ExternalWorkflowUpdateAdmission, including the outer approval transaction inValidatedExternalWorkflowUpdateAdmission. Their defaultControlPlaneMutationRetrierpath only retries SQLite, so the reported MariaDB path is a candidate retry gap. Core's embedded update admission also uses a single-attempt transaction, but it is a separate path and is not yet implicated in this remote fixture failure. Existing Core command-deadlock coverage exercises MySQL signal/cancellation cases and explicitly skips MariaDB. No reproduced root cause or delivered fix is claimed.Next: exercise the exact original probe inputs against published PHP 2.11.5/Core 2.15.1 on isolated MariaDB, retain correlated request and exception evidence, then qualify a scoped retry correction with rollback, idempotency, caller-owned transaction, and unrelated-error controls. Passing reruns do not close this issue.
rmcdaniel commentedon Oct 27, 2026
The isolated published-PHP reproduction harness is working. Original
state-updatesandupdate-validationprobes from runner source0e7e955244d7a30fb19694bb04c263b90c44e582each completed once and passed the canonicalcheckObservationcontract. The tuple is published Server 2.11.5, Core 2.15.1, normal locked PHP SDK 2.2.6, MariaDB 10.11.19. The database was bootstrapped with the frozen published 2.5.14 layout and migrated with the published current image. The repair/schedule maintenance loop stayed active; the isolated server's deadlock counter remained zero.This first pair did not reproduce either intermittent HTTP 500. It is not a root-cause finding or closure evidence. The exact selected images/input hashes and small original observations are retained only for this active investigation; disposable data, raw logs and task resources will be removed after the correlated diagnostic receipt.
The next useful check is a deterministic rollback/deadlock control at the actual Server external-admission boundary. A correction must retry an owned root transaction, preserve a caller-owned transaction, and avoid repeating validator execution or committed effects. The validated path has an outer approval transaction around parent admission, so blindly enabling nested retries would be insufficient. No product source has changed for this issue yet.
rmcdaniel commentedon Oct 27, 2026
Implemented and pushed the scoped correction at
517e2647512e7f94722050ace5d33fecceb48ef6, including actual merged #824. The application change isb8c685e73onfix/825-external-update-deadlocks; only the two external-update admission classes and a regression test differ from current main.Server now asks Laravel to retry owned root MySQL/MariaDB admission transactions using the existing configured attempt budget. Nested approval admission remains owned by its outer approval transaction. Caller-owned transactions, other storage connections, unrelated failures and after-commit failures retain their original behavior. Validator approval remains outside the retried admission, and the retained approval is rechecked on each attempt.
Before/after evidence:
1213while accessingworkflow_tasks.This reproduces and fixes the root-transaction retry gap at the actual Server external-admission boundary. It does not independently identify the exception behind the earlier uncorrelated
approved_update500s, or establish acceptance of a newly published Server image. Those failures remain retained. The original published 2.11.5 baseline still needs replacement only after a qualified publication and coordination with the native fixture owner.No PR or release has been created yet while CI throughput repair #826 is under qualification. Next: defensive exact-source review, protected checks/merge, qualified publication and published-artifact verification of the reported corpus cases.
rmcdaniel commentedon Oct 28, 2026
General pushed follow-up head
adaad79ac327fda10d40dd4df96f5ce5e7b7dc1fonfix/825-external-update-deadlocks.A controlled real InnoDB deadlock now also exercises a worker's actual continue-as-new operation between rollback and retry. Before the run guard, MariaDB's direct case retried twice, answered
unknown_updateHTTP 404 and left one durable update row after the requested run changed. The approved case already refused stale validation. The first loose probe allowed admission to commit before continuation and was discarded as an invalid interleaving; the retained negative control holds the history range so rollback actually precedes continuation.The controller now passes the selected run ID into external admission. Fresh admission rechecks it while holding the instance/run locks on each retry, and refuses a changed run with the existing
historical_run_command_rejectedHTTP 409. The guard runs after retained-key recovery, preserving committed receipts, and before validation can start for a different selected run. Ordinary successful response metadata is unchanged. No native or frozen-fixture source changed.Validation on the source logic: all 24 real-engine controls passed on MySQL 8.0.46 and MariaDB 10.11.19 (604 assertions), all 78 related request/validation/routing/fence/retrier tests passed (934 assertions), and all 54 control-plane/controller tests passed (819 assertions). After strengthening the race assertions, its four final cases passed with 140 assertions: one rolled-back insert attempt, correct original run/reason, and zero admitted command/update/history rows. Pint passed all four changed PHP files. This supplements the original rollback-only retry proof; it does not correlate the original approved-update HTTP 500 reports.
Next: exact-source review, normal CI/PR progression after the current CI bottleneck repair, then published-image acceptance and the full affected MariaDB corpus. No PR, tag or release is claimed yet.
rmcdaniel commentedon Oct 28, 2026
Merged #828 through the normal queue as
759abaa3b4eb7d931cf256b1aab2950768c2c0a9. All 43 merge-group Shared Server jobs and the merge-group PHP feature qualification pass. The PR head had 16 successful checks and 7 conditional skips, with its source/Sonar review recorded in the PR.This puts the demonstrated owned-transaction retry and selected-run corrections on main. It is not a published-image acceptance result. Next: qualify main, publish through the protected release path with the appropriate qualified Core dependency, and repeat the full affected MariaDB corpus against the exact published image. The originally reported approved-update HTTP 500s remain uncorrelated and this issue stays open.
rmcdaniel commentedon Oct 29, 2026
Handoff from #854's merge-queue run 40841285248 (MariaDB 10.11 corpus lane, job 123339409484). General owns the PHP follow-through.
tests/Fixtures/ServerParityBaselines/php-2.11.5.json; linux/amd64durableworkflow/server@sha256:bcc672859a399d1df234fc304531ab06b7bfa8acad7cb2c9da95f8e7ab5c98b4).many-updatesprobe sent its 16thPOST /api/workflows/{id}/update/set(wait_for: accepted, with arequest_id) while the installed maintenance loop (workflow:v2:repair-passthenschedule:evaluate) was running. The answer was an uncaught500with body{"message":"Server Error"}.SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction, on aselect * from workflow_tasks where workflow_run_id = …in the update admission.request_id) again, at most three times in all. Each retry writes a value-free note (many_updates_submit_retry: index, attempt, status) that the recorder keeps even when the probe passes. So a recovered 500 stays visible as a defect signal, not error-free qualification.rmcdaniel commentedon Oct 29, 2026
Correction, October 29: I initially misread the third operator comparison field as a task namespace. It is
health_namespace; the corrected interpretation appears below. The original recorded values are unchanged.Published PHP-only MariaDB follow-through completed on October 29, 17:56:11–18:02:42 UTC. This is the first retained record: 74 of 82 selected cases pass; eight differences remain unclassified, recorder exit 1 /
product-fail. No full-pass or historicalapproved_update500 root-cause claim follows from this run.update-validationandmany-updatespass in this record; the original intermittent failures above remain open.Exact tuple: published Server 2.11.7, source/runner f55f90e, immutable Linux/amd64 child
sha256:be7c2288ff935396262d743590530b204fac11b76582e56a525cc81a8da840d3; Core 2.15.7 /3aa98b8473b130dbc141457f76ccc32e4aa58062; normal Composer PHP SDK 2.2.6 /290fa427a76ffc6ea226bcd59749374aa7a89efa; published CLI 2.2.0, SHA256fce532e6a49737aef9cb8a137fd66219928b2595764462be33b12a0fdf5abc7c. Real MariaDB 10.11.19 used a fixed 1 CPU / 1 GiB memory-and-swap limit, 128 MiB InnoDB buffer, TLS fixture, and no host ports. The database was initialized through the published layout and normal bootstrap path. Real repair/schedule maintenance ran during the finite corpus. This was a PHP-only HTTP profile, without native Server execution.The eight differences are:
operator-metrics: the health response namespace is null because health returns HTTP 500 after an activity exists. This matches the known MariaDB PHP Server health returns 500 on MariaDB (ONLY_FULL_GROUP_BY in task queue build-id snapshot) #569 health defect; the metrics response itself retains HTTP 200 and the correct namespace. The client omittedMYSQL_IMAGE, so the probe did not identify the MariaDB family for its existing documented disposition.schedule-catalog,schedule-fixed-rate-one-occurrence,schedule-manual-lifecycle,schedule-policies: four schedule audit route-name differences.start-search-attributes: supplied custom workflow-ID prefix differs from a hardcoded reference prefix.storage-test: storage roundtrip returns HTTP 500 rather than 200.waterline-import: recorded history bytes are 4713 rather than 5431; reference/meter intent belongs to the existing coordinated reference work.Next: preserve and classify these differences using the first observations and correlated diagnostics, distinguish fixture/reference assumptions from product behavior, and obtain request-correlated evidence for the storage 500. No rerun has replaced this record. A passing repeat alone will not close either historical admission failure. This does not infer an authorization/isolation failure from null metadata.
The finite client is removed. Task-owned diagnostics and the isolated database are retained for classification; maintenance has been stopped after preserving the run's application log. Cleanup checkpoint is completion of classification and durable evidence retention. The normal public 2.11.7 release and its specific Bridge acceptance are already documented in #861; that acceptance does not imply this full corpus passed.
rmcdaniel commentedon Oct 29, 2026
General also accepted the native owner's explicit PHP helper handoff:
update-validation-probe.phpmust measure the complete logical request, including lock-pressure retries, while preserving frozen timing bounds and product runtime behavior. Add focused retry and cached-receipt controls. This helper correction is separate from identifying the original intermittent admission 500. It follows the current Laravel security-floor qualification; native reference/frozen changes remain with their owner.rmcdaniel commentedon Oct 29, 2026
The
storage-testHTTP 500 in the October 29 first corpus now has request-correlated evidence. BothPOST /api/storage/testrequests at 18:00:23 UTC match two application exceptions at the same time fromRuntimeLocalExternalPayloadStorage::__construct, reached throughNamespaceExternalPayloadStorage::driverForandStorageController::test: the configured external payload root could not be created.The runner's
storage-test-probe.phpexplicitly configuresfile:///state/storage-test-<target>/<namespace>. The normalserver-parity.ymlPHP container provides writable/state. This local disposable fixture omitted that mount, so its unprivileged application could not create the configured root. The first record remains retained unchanged; a corrected-envelope targeted repeat is still pending. This explains the fixture setup difference and does not establish a storage runtime regression.This is separate from the historical
approved_updatefailures: those still have no request-correlated exception, and neither these storage exceptions nor a later unrelated external-payload exception is their diagnosis. The other seven first-record differences remain for classification.The finite maintenance process is stopped. The isolated database and bounded request/access/error diagnostics remain only until classification and durable evidence retention are complete; then remove the task-owned fixture.
rmcdaniel commentedon Oct 29, 2026
The first 82-case record remains 74 pass / eight differences. Targeted follow-through on the same published Server 2.11.7 / Core 2.15.7 / runner
f55f90e54a213872f1c164a1b22c082d859c18aanow classifies those observations more precisely:/statemakes the same namespace's real storage diagnostic return HTTP 200, with both 64-byte and 70,000-byte roundtrips passing. The first run's two 500s are correlated with failure to create that configured storage root.health_namespace. Metrics is HTTP 200 with the correct namespace throughout. Health becomes 500 after an activity exists, and the actual exception is MariaDB 1055 /workflow_tasks.compatibilitynot in GROUP BY, the existing #569 defect. My client omitted theMYSQL_IMAGEhint used by the probe to identify MariaDB. Supplying the exact inspected database image makes the unchanged operator case pass its existing known-defect disposition at 19:52:45–19:52:56 UTC. This does not repair health; PHP Server health returns 500 on MariaDB (ONLY_FULL_GROUP_BY in task queue build-id snapshot) #569 remains open. The earlier “task namespace” interpretation was incorrect and is corrected in the first receipt.parity-v1prefix. The earlier custom prefix exposed its hardcoded expected workflow ID, a fixture comparison limitation rather than an incorrect returned workflow ID.These selected records do not replace the first full record or constitute a clean full-corpus qualification. Next: qualify the whole finite PHP MariaDB profile with the normal image envelope, truthful database-family metadata, fresh database/default prefix, and the accepted reference alignment. Preserve the historical approved-update 500s without assigning either this storage exception or #569's health exception as their cause.
The selected clients are removed and maintenance is stopped after each finite record. The isolated task database and small diagnostics remain for full follow-through; remove them once that result and remaining dispositions are durable.
rmcdaniel commentedon Oct 29, 2026
Fresh full published-PHP MariaDB qualification passes 83/83 cases, 2026-10-29 20:50:00.262–20:57:15.518 UTC, exit0. This is PHP-only acceptance of the current reviewed corpus, including its explicit published-defect allowances.
Exact tuple: Server2.11.7 source
f55f90e54a213872f1c164a1b22c082d859c18aa, actual stock amd64 imagesha256:be7c2288ff935396262d743590530b204fac11b76582e56a525cc81a8da840d3, installed Core2.15.7/3aa98b8473b130dbc141457f76ccc32e4aa58062, PHP SDK2.2.6/290fa427a76ffc6ea226bcd59749374aa7a89efa; visibility verifies published CLI2.2.0 PHAR SHAfce532e6a49737aef9cb8a137fd66219928b2595764462be33b12a0fdf5abc7c. Runner is PR870 head5eddc262dea97b3c99f5188ba9577eb15c692a48, based on accepted current reference6af963db. Real MariaDB10.11.19 uses pinned imagesha256:7db29378d4fdab73f8123bbc2b48905c90d1a4b00cf848b028f1e81e623257f2, inspected memory+swap1GiB/1GiB. Fresh database was bootstrapped using the normal published2.5.14 layout then2.11.7 bootstrap; defaultparity-v1prefix, truthful actual MYSQL_IMAGE, normal stock container layout with writable/state.The original74/82 record is unchanged (SHA
7da1483736cecfba3e755ab65f2fc7d3953377d9b958bc58f5d11922a76023e4). Current full record SHA8a950d2bbea3bfc3f0dcc58b9c3a39d413822aa2ae129e4d8829b061b0e1b41c. Both artifact tuples are identical. All82 original cases remain;81 original fixture hashes are identical, while the owning Rust maintainer's accepted#867 canonical-meter change updates waterline-import. Its already-merged child-descriptions fixture adds the83rd case. General changed no frozen fixture or native source.All eight earlier differences now have bounded dispositions supported by the retained first record and targeted controls: four masked-route-cache fixture differences; missing writable
/state; the omitted MariaDB image hint for the existing#569 allowance; a custom-prefix fixture assumption; and the accepted#770 canonical-meter reference update. Whole-profile schedule/storage/start-search/update-validation/waterline checks now pass. Approved update is202 (0.314183821s logical request); this run reported no lock-pressure retry notes, so retry timing itself is proved by PR870's actual-child real-HTTP negative/positive controls, not inferred from this no-retry run.Known PHP defects remain modeled: cancellation-scopes, record-local-activity, workflow-import#765 and worker-sessions. In operator-metrics, metrics are200 in every stage; health is200 in the three pre-activity stages and still500/null health namespace in the four later stages under the documented MariaDB#569 allowance. This is no repaired-health or task-isolation claim. Bridge checks exercise the published2.11.7 correction. The normal hosted PR870 qualification remains pending.
The original historical approved-update500s remain uncorrelated and this issue stays open. A passing full record does not supply their missing exception or prove a cause. Next: finish PR870 hosted qualification/merge, preserve the reviewed evidence and retain a narrowly scoped historical-failure disposition. The finite client was reaped and its maintenance process stopped; task-only resource cleanup follows review. Retained local records are for this active investigation, with removal reconsidered after the historical disposition and qualification receipts are accepted.
rmcdaniel commentedon Nov 2, 2026
Closing: the HTTP 500 was published-PHP-only, PHP Server is frozen (operator direction, November 1), and native 3.0.1 can't produce it.
workflow_tasks. PHP's fix (Retry owned external update admissions after InnoDB deadlocks #828) shipped in 2.11.7. Native recordings in those runs passed. The MariaDB lane of merge-group run 41514975042 (the source ofnative-3.0.1) recordedoutcome: passfor PHP, embedded and native.923d45b3: update admission (admit_update,rust/src/runtime/updates.rs) is one transaction. The validator wait runs outside it, and the approved path re-enters admission and rechecks the run and the approval. Any database error, a deadlock included, rolls back.RuntimeError's response mapping (rust/src/runtime/mod.rs) then answers it as503 storage_unavailable, never 500, with a redactedruntime_storage_unavailablelog line carrying the driver code.{"many_updates_submit_retry":…"status":500}lines in PHPUnit logs come fromServerParityManyUpdatesRetryTest's fake 500s, not from a server.Weaker relative in native, not a 500: a deadlock on update admission isn't retried in-process and isn't answered with the typed, retryable
backend_lock_pressure+Retry-Aftersome other routes use. Clients get a plain 503. There's also no native two-connection InnoDB deadlock test for update admission. If that becomes worth doing, it's a native issue of its own: retry or typed lock pressure on admission, plus a deadlock test proving rollback and no duplicate under onerequest_id.rmcdaniel commentedon Nov 2, 2026
The concurrency test noted above is tracked in #960.