Skip to content

docs(ops): add retention-gap recovery runbook (#159) - #180

Closed
Hustler490 wants to merge 1 commit into
wraith-protocol:developfrom
Hustler490:docs/159-retention-gap-runbook
Closed

Hustler490 wants to merge 1 commit into
wraith-protocol:developfrom
Hustler490:docs/159-retention-gap-runbook

Conversation

@Hustler490

Copy link
Copy Markdown

Closes #159.

What

Adds guides/ops/retention-gap-recovery.mdx - a step-by-step operator runbook for recovering a scanner that went offline past the Soroban RPC retention window - and links it from the monitoring guide and the sidebar.

Runbook contents

  • Detection + bounding: commands to measure gap_ledgers/gap_hours and classify inside vs. outside the ~17,280-ledger (~24 h) window.
  • Checkpoint handling: export and protect lastProcessedLedger; explicit warning against jumping the checkpoint forward.
  • Archive replay: choosing an archive provider and replaying the missed range, with a "must be complete" verification before advancing.
  • Dual-provider recovery: primary/fallback config and the failover behaviour.
  • Safe rescan + de-duplication: bounded rescan, de-dup audit, reconciliation - with a warning against disabling de-dup.
  • Operator evidence: the exact values/outputs to attach to the incident ticket.
  • Alert thresholds: IndexerScanStopped, IndexerCheckpointStale, IndexerRetentionRisk, WatcherEventDropSpike, SorobanRetentionWindowExceeded.

Links

  • Added to the monitoring guide's Watcher Drop Spike playbook (step 3) and its Cross-References.
  • Added to the Operations group in docs.json.

Acceptance criteria

  • Documents detection, checkpoint handling, archive replay, and dual-provider recovery.
  • Includes safe rescan and deduplication steps.
  • Adds alert thresholds and operator evidence to collect.
  • Linked from the monitoring guide.

@drips-wave

drips-wave Bot commented Sep 29, 2026

Copy link
Copy Markdown

@Hustler490 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@truthixify

Copy link
Copy Markdown
Contributor

The runbook is clear, but it presents commands that do not exist in these repos, including checkpoint export, rescan status, rpc check, and audit duplicates. Use real SDK or operator commands, or label pseudocode clearly. Also match the SDK event identity instead of claiming deduplication by (contract_id, ledger, event_index).

@truthixify

Copy link
Copy Markdown
Contributor

Closing this duplicate because #181 is the updated replacement for the same issue. Please continue fixes there.

@truthixify truthixify closed this Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Wave 9] Add a retention-gap recovery runbook for operators

2 participants