Repository navigation
Open
Labels
addieIssues related to Addie (via any channel)Issues related to Addie (via any channel)bugSomething isn't workingSomething isn't workingclaude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.Issue has been triaged by the Claude Code triage routine. Remove to re-triage.compliance-suitepriority:P0Immediate priority: user/revenue/security/reliability bug to work nowImmediate priority: user/revenue/security/reliability bug to work nowseverity:critical
Milestone
Description
Summary
An agent's comply scheduler history stops updating (no new entries, no error) while health checks continue running normally. Supplemental diagnostic work points to a fail-closed trust check as the likely cause.
Observed case
- Affected:
sell.nofluffadvisory.com/mcp— comply history stopped after the 2026-08-19T18:47Z heartbeat; health checks unaffected. - Control:
adcp.signal-stack.io— still running comply twice-daily on 3.1.20, no stall.
Declaration diff between the two
- Affected agent declares
supported_versionsat BOTH top level AND underadcp.supported_versions: ["3.1"], plusadcp.major_versions: [3]. - Control agent declares ONLY
adcp.supported_versions: ["3.0","3.1"], no top-level field.
Hypothesis
The target resolver and the fail-closed trust check from #6426 (merged 2026-08-12T03:31Z — refuse to publish against an untrusted fallback target, discard a cached target the live profile contradicts) may read different locations for supported_versions or handle a single-entry list differently than the two-entry control case. For the affected agent, this could cause a silent publish refusal: no comply history entry is written, and no error is surfaced, while health checks (a separate path) continue unaffected.
Timeline
- 15 successful heartbeats occurred between the fix(compliance): avoid untrusted fallback verdicts #6426 merge (2026-08-12) and the stall (2026-08-19), consistent with a registry deployment of the change landing around 2026-08-19/20 rather than at merge time.
Suggested investigation
- Check what the target resolver produced for
sell.nofluffadvisory.com/mcpat the first missed heartbeat (~2026-08-20T06:00Z). - Check whether a fail-closed refusal was logged internally without a corresponding comply history entry being written.
Ruled out
- The affected agent's own version handling is confirmed correct and unrelated: via
get_adcp_capabilities, it acceptsadcp_versionpins 3.1/3.1.13/3.1.20 and correctly rejects 3.0 and 3.2.0-rc.1 withVERSION_UNSUPPORTED.
Reported via
Metadata
Metadata
Assignees
Labels
addieIssues related to Addie (via any channel)Issues related to Addie (via any channel)bugSomething isn't workingSomething isn't workingclaude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.Issue has been triaged by the Claude Code triage routine. Remove to re-triage.compliance-suitepriority:P0Immediate priority: user/revenue/security/reliability bug to work nowImmediate priority: user/revenue/security/reliability bug to work nowseverity:critical
Activity
bokelley commentedon Sep 7, 2026
Triage
Classification: Bug (regression)
Bucket(s): compliance-suite, addie
Status: ready-to-implement
Priority: P0 — silent production failure for an external adopter, no error surfaced
What the experts said:
My take: The issue is well-evidenced: PR #6426 ("fix(compliance): avoid untrusted fallback verdicts", merged 2026-08-12, server/Addie-only) introduced a fail-closed trust check that refuses comply history writes when the target can't be trusted. The affected agent declares
supported_versionsat both the top level and underadcp.supported_versions; the control agent declares onlyadcp.supported_versions. The 15-heartbeat gap between merge and stall is consistent with a delayed registry deployment, not a code deployment, landing ~2026-08-19/20. Health checks are unaffected because they run through a separate path. The symptom — comply history silently stops, no error, health checks continue — is exactly what a fail-closed silent refusal would produce.Implementation brief:
supported_versionsfrom only one location (either top-level oradcp.*-namespaced). An agent declaring both causes a contradiction between what the pre-discovery step cached and what the live profile asserts, triggering the "discard a cached target the live profile contradicts" refusal. A single-entry list (["3.1"]) vs. a two-entry list (["3.0","3.1"]) may also be handled differently.supported_versionsbefore the trust evaluation step — merge top-level andadcp.supported_versions/adcp.major_versionsinto a single canonical set so both declaration styles produce the same result. Ensure the fail-closed refusal writes a log entry (or comply history stub with a failure reason) so silent stalls are detectable.sell.nofluffadvisory.com/mcpat the first missed heartbeat (~2026-08-20T06:00Z); check whether a fail-closed refusal log line exists without a corresponding comply history write.main.Triaged by Claude Code. Session: https://claude.ai/code/session_01RAqvGrz4MH9t2feuxbuc95
Generated by Claude Code
EvgenyAndroid commentedon Sep 15, 2026
I own the affected agent. Your hypothesis is falsifiable from outside, and I think the data falsifies it — but the symptom you reported is real, and it is back.
The declaration shape is not the cause.
sell.nofluffadvisory.com/mcpstill returns exactly the shape this issue names — top-levelsupported_versions: ["3.1"]alongsideadcp.supported_versions: ["3.1"]andadcp.major_versions: [3](re-probed 2026-09-15T12:45Z). Nothing about it has changed since you filed. Yet the agent was heartbeated ten times between 2026-09-08T04:26:43.320Z and 2026-09-13T02:43:07.298Z, on cadence, at 8.4–15.8 h intervals. If the #6426 fail-closed trust check refused to publish because of that declaration, it would not have published ten times with the declaration unchanged.A cause that fits the window better, and it is mine. Our publisher
adagents.jsonlisted the sales agent at the bare originhttps://sell.nofluffadvisory.cominstead of its registered/mcpendpoint, from 2026-08-08T16:45Z. I corrected it and it went live 2026-09-06T23:52:57Z. The agent resumed 2026-09-08T04:26:43.320Z — 28 h 34 min later. The stall window sits entirely inside that stale span.One thing that does not fit cleanly: the URL went stale 2026-08-08T16:45Z but the stall began 2026-08-19T18:47:59.122Z, eleven days later. A periodic re-verification cadence would explain the lag; I cannot see that from outside. So I am offering this as the better-fitting hypothesis, not as a conclusion.
Your actual complaint survives either way. Whichever mechanism it was, it was silent: no history row, no error surfaced, health checks unaffected. If a publisher-side authorization mismatch can park an agent's compliance history indefinitely, that state belongs on the card as a visible reason rather than as an absence of rows. That is the part I would keep at P0 regardless of which hypothesis wins.
Current state, for the record. No heartbeat row since 2026-09-13T02:43:07.298Z. Six
owner_testruns that afternoon (13:02:03.421Z through 14:46:01.285Z, the last of them a maintainer verification under #7428) reset the next scheduled check to 2026-09-14T02:46:01Z, and it is now 34 h past that.I am deliberately not claiming this is the same defect, and I would rather not muddy this issue with it, because the circumstances differ:
adagents.jsonis correct and unchanged this time, and I deployed the worker seven times on 2026-09-13, one of which narrowed the advertisedcompliance_testing.scenariosfrom twelve to five about fourteen minutes before the last row. Those are my confounders to rule out, not yours.Worth noting for scope: current staleness is not specific to this agent. Across the 31 non-opted-out registry agents publishing a
last_checked_at, the median is 4.5x the agent's own declaredcheck_interval_hoursas of 2026-09-15T12:45Z, with 26 past 2x and 19 past 4x. Mine is 3.8x, rank 21 of 31.Happy to run anything against the agent — it is live, and
adcp.signal-stack.iois mine as well, which makes it a usable control.bokelley commentedon Sep 23, 2026
Queue audit update: the declaration-shape hypothesis is disproved by the reporter\x27s unchanged-profile successful runs, so this issue should no longer be implemented as top-level/nested
supported_versionsnormalization. The silent-failure/liveness portion has since gained concrete safeguards onmainvia #7453/#7556/#7622: incomplete runs are audit-only, authoritative grades are preserved, heartbeat admission/execution is bounded, zero-authoritative batches fail scheduler health, and operational diagnostics expose skips/errors. #7648 (pending behind RC6) separately fixes explicit refreshes being starved by ordinary heartbeat ordering. Keeping this issue open for post-deploy evidence of any remaining broad scheduled-heartbeat staleness; its title/root-cause framing should be updated rather than closing it as fully resolved.[-]Comply history silently stalls for agents declaring top-level + nested supported_versions[/-][+]Compliance heartbeat scheduling can silently stall across agents[/+]garvitkaushik-123 commentedon Sep 24, 2026
Pulled the public compliance history API to check post deploy status after #7453, #7556, #7622, and #7648.
sell.nofluffadvisory.com/mcp, the agent from this issue: last run with triggered_by heartbeat was 2026-09-13T02:43Z. Every run after that (10 of them, through 09-17) has triggered_by owner_test. No heartbeat-triggered run shows up in the history for 11 days.
adcp.signal-stack.io/mcp, the control agent cited as not stalling in the original writeup: last heartbeat-triggered run was 2026-09-14T20:03Z. Same pattern after that, owner_test only. 10 days with no heartbeat-triggered run in the history.
Checked one more agent from the registry at random, sales-agent.addiscovery.ai/mcp: last heartbeat 2026-09-15T07:03Z, nothing since.
Important caveat on all of this: /compliance/history only returns is_authoritative true rows (that filter is from #7453). If the heartbeat is firing every 6 to 12h like these agents are configured for but the runs are timing out or erroring, they would land as is_authoritative false and just not show up here, while getting silently retried. So this data can't tell the difference between the scheduler not running and the scheduler running but never completing for these agents. Both would look identical from this endpoint.
Also #7648 merged this morning but bokelley flagged it as pending behind RC6, so it is not deployed yet and this data does not test it.
What I can say with confidence: no authoritative heartbeat grade has landed for sell.nofluffadvisory.com/mcp or adcp.signal-stack.io/mcp in over a week, on agents configured for 6 to 12h checks. What I can't tell from the public endpoint is whether that means the scheduler stopped or the runs are happening and failing to complete. Whoever has access to the audit_only skip counts or eligible_backlog for these agents would be able to tell which one it is faster than I can from outside.
Raw data (tested_at, triggered_by):
sell.nofluffadvisory.com/mcp
2026-09-17T11:41:54.909Z owner_test
2026-09-17T11:32:30.768Z owner_test
2026-09-16T09:55:09.448Z owner_test
2026-09-16T00:28:17.342Z owner_test
2026-09-13T14:46:01.285Z owner_test
2026-09-13T14:30:09.442Z owner_test
2026-09-13T14:11:23.852Z owner_test
2026-09-13T13:45:46.180Z owner_test
2026-09-13T13:35:54.316Z owner_test
2026-09-13T13:02:03.421Z owner_test
2026-09-13T02:43:07.298Z heartbeat <- last heartbeat-triggered run
adcp.signal-stack.io/mcp
2026-09-16T09:52:10.051Z owner_test
2026-09-16T00:24:43.111Z owner_test
2026-09-14T20:03:40.370Z heartbeat <- last heartbeat-triggered run
Pulled from GET /api/registry/agents/:encodedUrl/compliance/history on agenticadvertising.org, the public endpoint, no db access.
bokelley commentedon Sep 24, 2026
Triage (updated 2026-09-24 — see @EvgenyAndroid follow-up comments)
Classification: Bug (confirmed code regression — scope corrected)
Bucket(s): compliance-suite
Status: ready-for-human
Milestone: P0 Bugs
Correction to prior escalation: @EvgenyAndroid's follow-up directly disproves the "globally inoperative scheduler" conclusion. The scheduler is running: 11 authoritative
triggered_by='heartbeat'rows confirmed strictly after 2026-09-15T07:03Z across 11 distinct agents, with session IDs decoding to live runs. The "universal cutoff" inference was built on a biased sample — the two focal agents are among the most owner-tested in the registry, and one of the four checked agents last heartbeated on 2026-08-11, a separate cohort.Confirmed code bug —
compliance-db.ts:937:recordComplianceRun'sschedule_nextCTE advancesnext_compliance_check_atfor every authoritative run without atriggered_bypredicate. After #7453 changed heartbeat due-selection from the deriveds.last_checked_atpredicate to storedm.next_compliance_check_at, anyowner_testrun silently defers that agent's next automatic heartbeat by a full interval. Agents with heavy owner-testing are systematically underserved by the scheduler.Fix scope (corrected): The
triggered_bygate onnext_compliance_check_atis the correct and only fix — advance it only on heartbeat-triggered runs. The "at minimum don't clearrequeued_atonowner_test" fallback in the prior version of this comment is wrong and should not be implemented: clearingrequeued_aton publish is intentional permigrations/608_agent_registry_requeued_at.sql("cleared when an authoritative run is published"), and keeping it non-null without re-arming the eligibility gate (next_compliance_check_at IS NULL OR < NOW()at:1552) would permanently sort affected agents to the front of every subsequent heartbeat batch viaORDER BY requeued_at ASC NULLS LAST(:1561) without fixing the 12h deferral.requeued_at = NULLstays exactly as is. The same invariant is already enforced one function away indeferComplianceCheckAfterInconclusiveTarget(:1443-1452) with anAND next_compliance_check_at > NOW()guard. This fix is already filed as #7680 with repro and line anchors.This issue vs. #7680: Let #7680 carry the CTE fix. #7355 stays on the broader throughput question — the post-deploy data assembled here (fleet-wide heartbeat collapse from 2026-09-14, the
agent_compliance_runssplit query) is the tracker for remaining fleet-wide degradation.Still open (needs DB access): Whether remaining fleet degradation is scheduler not firing vs. runs firing but landing non-authoritative (
is_authoritative=falseon timeout). The query from @EvgenyAndroid's comment separates the two cases definitively.Triaged by Claude Code. Session: https://claude.ai/code/session_01C2avokMhSKNNL5t19HjFLW
Generated by Claude Code
EvgenyAndroid commentedon Sep 24, 2026
The data here is sound but the conclusion drawn from it is not, and the part that got escalated is the part that is wrong. I went looking to confirm the opposite of this and failed, so here is what actually holds up.
"Zero scheduled runs since" is false. Same public endpoint, no DB access:
Eleven authoritative
triggered_by='heartbeat'rows exist strictly after 2026-09-15T07:03Z, across eleven distinct agents. The session ids decode to 35s–8.5min before their owntested_at, so these are live runs, not backfills. The 2026-09-18T00:07–00:43Z cluster spans four unrelated operators (selleragent.pubx.ai,webmd.…optimera.nyc,vastlint.org,adcp-tviq.teqblaze.io) with sequential session-start epochs, which is an autonomous batch walking the queue, not one operator requeueing their own fleet.So the scheduler is firing.
job-definitions.ts:573-602still registerscompliance-heartbeaton a 1h interval, andworker-watchdog.tscold-restarts a dead worker every 60s. A worker silently down for 11 days would have to survive that loop.The collapse is real, though, and the date is 2026-09-14T03:01Z, not the evening of 09-13. Heartbeat rows/day fleet-wide:
09-13 ran a normal full day through 23:04:39Z, plus 02:12Z and 03:01Z on 09-14. First anomalous gap is 09-14T03:01:29Z → 14:13:50Z. Re-tallying at
limit=100leaves 09-14 onward unchanged and only inflates the earlier days, so the row cap biases against this finding rather than creating it.@bokelley, to your question about a deploy on or just before 09-13: yes, and it is the stack you set aside. The provenance cutover brackets it to the minute. Last null-
provenance_jsonrun fleet-wide is 2026-09-13T14:46:01.285Z; first populated is 2026-09-13T17:24:15.976Z; no exceptions across 2,832 rows. That window is #7453 (1834f1c5, merged 14:06:14Z) and #7454 (dd6b0496, 14:16:36Z). It is ~15h later in the day than the 02:43Z timestamp this issue anchors on, which is probably why a search near 02:43Z came up empty.1834f1c5moved heartbeat due-selection froms.last_checked_at < NOW() - check_interval_hourstom.next_compliance_check_at < NOW()(compliance-db.ts:1552). That replaces a derived predicate with stored state, and stored state can be written by things that are not a heartbeat.One of those writers is a real bug, independent of everything above.
compliance-db.ts:936-942, insiderecordComplianceRun:No
triggered_bypredicate. Every authoritative run defers the next automatic check by a full interval, includingowner_test. An owner clicking "Recheck & retest" suppresses that agent's own heartbeat for 12h, and clearsrequeued_aton the way through.That is worth separating from the scheduler question because it is self-inflicted and it skews the sample. Both agents at the centre of this issue are among the most owner-tested in the registry, which makes them the worst possible choices for measuring scheduler health. My own agents, and I have been generating a lot of that
owner_testtraffic.Two corrections to the framing, offered plainly.
adcp.4dvertible.comdoes not belong in the 09-13..09-15 window; its last heartbeat is 2026-08-11T23:37:42.384Z, 33 days earlier, part of a separate ~10-agent cohort that stalled within about ten hours on 08-11/12. That makes the pattern 3 of 4, not 4 of 4, and "universal cutoff in a ~60-hour window" is doing more work than the data supports. And #7648 merged today at 12:49:38Z while the last fleet-wide heartbeat is 2026-09-22T02:41Z, so there is no observation window in which it could have changed anything yet.What I could not settle from outside. Whether the missing runs are not firing, or firing and landing non-publishable.
run-publication.ts:5-9publishes onlyis_authoritative !== false && completeness === 'complete', andcompliance-testing.ts:1479setsis_authoritativefalse ontimed_out. WithHOSTED_FULL_COMPLIANCE_TIMEOUT_MS = 600_000against a worst case around 1139s, a slow agent's heartbeat becomes invisible rather than absent. One query separates the two:If
totalholds near ~28/day since 09-14, this is a publication regression and the fix is a diagnostic surface. Iftotalcollapses withvisible, the scheduling gate is the target, starting with the all-10 batch lock atcompliance-heartbeat.ts:144-153and the CTE above.There is also a separate live finding in the meantime: eleven agents currently show
last_checked_at19–32 days newer than their newest visible run, all written before the 14:06Z commit. Something was advancing those cards with no publishable row on the old code, which is the silent-publish-failure class #7453/#7556/#7622 were aimed at, still happening.Method note for anyone re-running this. A malformed agent URL returns HTTP 200 with
{"runs":[],"count":0}rather than a 404. Appending/mcpto an agent that has no/mcpproduces a convincing total outage. It caught me once in this pass.EvgenyAndroid commentedon Sep 24, 2026
Thanks for the re-scope. Two notes on the fix.
It is already filed as #7680, with the repro and line anchors, so the scheduling defect has somewhere to live that is not this issue. Probably worth keeping #7355 on the throughput question and letting #7680 carry the CTE fix, otherwise the DB query below gets lost under a fix that is already scoped.
The "at minimum don't clear
requeued_atonowner_test" fallback is the wrong half, and it is worse than doing nothing. Clearingrequeued_aton publish is deliberate —migrations/608_agent_registry_requeued_at.sqlsays so in its own comment: "Record the requeue time so the heartbeat can serve explicit requeues first; cleared when an authoritative run is published."The defect is re-arming the eligibility gate. Eligibility is
next_compliance_check_at IS NULL OR < NOW()(compliance-db.ts:1552); ordering byrequeued_at(:1561) only ranks agents that already passed that gate. Implement the fallback alone and you get both failure modes at once: the agent is still ineligible for 12h becausenext_compliance_check_atwas still advanced, and itsrequeued_atnow persists through every subsequent run, so underORDER BY requeued_at ASC NULLS LASTit sorts ahead of every non-requeued agent permanently.The
triggered_bygate onnext_compliance_check_atis the fix.requeued_at = NULLshould stay exactly as it is.Minor: the CTE opens at
:937, not:936.One precedent for the patch, since it settles the intent question:
deferComplianceCheckAfterInconclusiveTarget(:1443-1452) already enforces this rule one function away, with the comment "A manual requeue (NULL) must win over deferral" and anAND next_compliance_check_at > NOW()guard. The CTE is the same invariant, unguarded.bokelley commentedon Oct 13, 2026
Re-scoping this issue. The #7680 CTE defect is fixed (#7969, verified on sell.nofluffadvisory.com before/after). Heartbeats recovered on Oct 1 with #7844.
Remaining problem: heartbeat throughput is declining since Oct 3, plus multi-day per-agent gaps after #7844.
Fleet heartbeats per day (public compliance-history endpoint, all 36 public agents):
Multi-day gaps: adcp.signal-stack.io 102h (Oct 8 16:52 → Oct 12 23:08), signals.purrsonality 206h, teqblaze 123h, addiscovery 105h. bidmachine, aiootech and seller.purrsonality have no run in 6–9 days.
Cannot yet distinguish retry backoff, lapsed discovery authorization, or starvation from the public API. The ~10 agents last run on 2026-08-11/12 look deliberately excluded (all failing or partial), not stalled.
Next step: run #7844's queue diagnostic against prod to separate eligibility, backoff and starvation for the agents above.
bokelley commentedon Oct 16, 2026
Correction and resolution update. The per-day figures in my earlier re-scope comment came from the public compliance-history endpoint and undercount; these are from the database (
agent_compliance_runs,triggered_by = 'heartbeat').What actually happened
timed_outruns across 13 agents (average 654 s against the 600 s budget, max 1,343 s), plus 33not_completedruns from one agent whose declaredsponsored-intelligencespecialism wasn't in the grader's compliance cache. Recovery began Sep 29 and completed with fix(compliance): bound heartbeat retries and prioritize current agents #7844 on Oct 1.get_productswith a 17.8 MB response; the worker was OOM-killed mid-batch, and because the batch claim was never released, every agent in that batch (six of them) starved for days with no run recorded and no backoff. Fixed by fix(compliance): bound agent response size during hosted compliance runs #8187 (4 MiB response cap on hosted compliance calls) and fix(compliance-heartbeat): release orphaned batch claims without losing crash backoff #8188 (claim recovery and crash backoff). Since deploy, the affected agents are getting heartbeats again on their normal cadence.#7680 (owner tests deferring heartbeats) was a separate defect, fixed by #7969 and closed.
Still open, tracked separately
sponsored-intelligence(defined only from 3.1) has its whole run voided.I'll close this after one more per-day check shows several clean days.