Skip to content

feat(node): keep refusal counts across restarts - #593

Open
LKSNDRTMLKV wants to merge 2 commits into
feat/138-authenticated-health-detailfrom
fix/590-persist-process-counters
Open

LKSNDRTMLKV wants to merge 2 commits into
feat/138-authenticated-health-detailfrom
fix/590-persist-process-counters

Conversation

@LKSNDRTMLKV

Copy link
Copy Markdown
Member

Closes #590

Summary

The node-health route reports two refusal counts, signingFailuresTotal and rulesetLoadFailuresTotal. They were in-process counters, so a restart reset them: a node whose signing failed for a week could report 0 after one restart. They are now kept in odal.node_failure_counter (migration 0048), so a restart does not clear them.

Design

  • dpp_types::node_failure::NodeFailureCounter is the port. PgNodeFailureRepo records each refusal with one upsert, so two refusals racing each other cannot lose one.
  • Each refused signature in publish awaits the durable count. If the count cannot be written, the failure is logged and the publish still returns its signing error.
  • Refused ruleset bundles are counted durably on all three paths: boot, the poller, and the admin reload route. The admin route did not count refusals before this change.
  • The Prometheus counters on /metrics still reset on restart, as Prometheus counters do. The route no longer reads process memory, so the in-process counters from feat(node): add authenticated node health route #591 are removed.
  • With no counter wired (a standalone vault), the two counts are omitted, not reported as zero.

Stacking

This branch is stacked on feat/138-authenticated-health-detail (#591). The #591 review fixes touch the same files (metrics.rs, the health route, the ruleset handler, the smoke test). Merge #591 into this branch and resolve in favour of this branch's design, which removes the in-process counters. Retarget to main once #591 merges.

Verification

  • just check: exit 0, 1726 passed, 3 skipped.
  • dpp-vault integration suite (integration-tests, shared Postgres): 599 passed, 12 skipped.
  • dpp-node smoke suite: 36 of 36 passed.
  • pg_node_failure (new, DAL): 1 of 1 passed, run before the last change, which touches only the vault handler.

@LKSNDRTMLKV LKSNDRTMLKV added this to the v0.16.0 milestone Nov 5, 2026
@LKSNDRTMLKV LKSNDRTMLKV added review-ready Opt this PR into a CodeRabbit review type/decision Needs a call; analysis alone will not close it urgency/backlog Owed, unscheduled surface/node-runtime Moves how a node boots, is configured, authenticates or is operated; no compliance rule depends epic/operations Running a node: persistence correctness, monitoring, install, update labels Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

epic/operations Running a node: persistence correctness, monitoring, install, update review-ready Opt this PR into a CodeRabbit review surface/node-runtime Moves how a node boots, is configured, authenticates or is operated; no compliance rule depends type/decision Needs a call; analysis alone will not close it urgency/backlog Owed, unscheduled

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant