Skip to content

feat(listener): add scheduled database cleanup job for expired temporary records - #910

Merged
Abd-Standard merged 2 commits into
Core-Foundry:mainfrom
najeebullahii:feat/845-db-cleanup-job
Oct 1, 2026
Merged

Abd-Standard merged 2 commits into
Core-Foundry:mainfrom
najeebullahii:feat/845-db-cleanup-job

Conversation

@najeebullahii

Copy link
Copy Markdown
Contributor

Closes #845.

Changes:

  • New scheduled DatabaseCleanupJob (new service file, registered via the existing WorkerManager with graceful shutdown) purging expired temporary data in bounded 1,000-row batched transactions: idempotency keys past expires_at or status EXPIRED, aged processed_events, aged dead-letter and execution-log rows, elapsed rate_limit_events windows, and DEACTIVATED backpressure records.
  • Configurable: CLEANUP_ENABLED (default true), CLEANUP_INTERVAL_MS (default 3600000, min 60000), CLEANUP_RETENTION_DAYS (default 30, min 1) validated through the existing config schema; pre-existing per-table retention env vars remain optional overrides; listener/.env.example updated.
  • Active records never removed: future-dated PROCESSED idempotency keys, execution-log rows tied to PENDING/PROCESSING notifications, ACTIVATED backpressure records, in-window rate-limit events; live cursors, templates and immutable template-audit history explicitly skipped with logged reasons.
  • Ownership boundaries preserved: ArchiveService remains sole owner of scheduled-notification archival and age-based, status-agnostic archive purging (which covers EXPIRED rows from in-flight feat(listener): expire scheduled notifications past a configured deadline #893 once merged); NotificationMetricsRunner owns metrics snapshots. The job logs delegated/skipped owners instead of running competing purges.
  • Failure isolation: a per-table failure logs, preserves partial deleted counts, skips that table, and the run continues; per-run log carries a correlation id, per-table counts, skipped/failed tables, retention, interval and duration.
  • Tests: 7/7 seeded-SQLite job suite (survival/deletion matrix, batching, failure isolation, shutdown timer cleanliness, enabled/disabled modes); config suite 46/46; archive age/status-agnostic purge assertion. Operator docs section added to NOTIFICATION_LIFECYCLE.md.

Testing: focused suites green (53/53); Prettier and git diff --check clean; no migration added; no package/lockfile changes.

Pre-existing baseline (verified against a stashed clean tree, untouched by this PR): six failures in legacy cleanup/archive tests (two EventRegistry malformed-SCVal fixtures, one legacy timer-call count, three archive API response-shape assertions); full npm test red on main (~55 failing suites: request-id.ts syntax error, Stellar XDR test setup, config secret validation); npm run lint fails on index.ts/request-id.ts/security-headers.ts/discord-notification.ts — index.ts contains a pre-existing malformed startup block which this PR wires around minimally without rewriting; lockfile out of sync with package.json (this PR modifies neither).

…ary records. Closes Core-Foundry#845. Configurable CLEANUP_ENABLED/CLEANUP_INTERVAL_MS/CLEANUP_RETENTION_DAYS with validated schema; bounded batched per-table purges of expired idempotency keys, aged processed_events/dead-letter/execution-log/rate-limit rows and DEACTIVATED backpressure; active records (PENDING/PROCESSING-linked logs, future-dated keys, ACTIVATED records, live cursors/templates) never removed; per-table failure isolation with partial counts; per-run logged counts with correlation id; graceful shutdown; operator docs. Archive/metrics ownership delegated, not duplicated.
@drips-wave

drips-wave Bot commented Sep 30, 2026

Copy link
Copy Markdown

@najeebullahii Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@Abd-Standard
Abd-Standard merged commit b0570df into Core-Foundry:main Oct 1, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Database Cleanup Job

2 participants