Cap confirmation retries, persist reconciliation history, and alert on credit-usage spikes

[zainabbaba31-source]

Summary

Closes four issues assigned to this account, covering payments, credits and wallets.

Cap the confirmation retries (#1801)
payment-confirmation.service.ts is single-shot per invocation, so the real unbounded retry surface is the reconciliation loop that re-polls every AWAITING_CONFIRMATION payment every five minutes. This adds two independent hard caps per payment:

• PAYMENT_CONFIRMATION_MAX_ATTEMPTS (default 20) — total scheduled confirmation polls.
• PAYMENT_CONFIRMATION_MAX_PROVIDER_ERROR_STREAK (default 10) — consecutive provider-unreachable polls.

Reaching either cap escalates the payment to MANUAL_REVIEW with a reason naming the cap and the counts, logs an ALERT: line, and excludes it from later passes without another provider call — including a row that was already past a cap when the process started, so a pre-cap deployment cannot keep hammering a degraded provider. The provider-error-streak cap is a deliberate, documented exception to the existing rule that a provider outage must not mass-flag payments: one bad run is evidence of a possible broad outage, whereas a long bounded streak on a single payment is evidence that this payment needs a human. The escalation counters are surfaced in getMetrics().

Persist reconciliation runs (#1800)
Each pass is now recorded in a new reconciliation_runs table (entity, migration 1790003000000, registered in PaymentsModule) with the summary counters, start/finish/duration, outcome, and a details context. A run that throws is marked failed with the error rather than the original failure being masked, and a failed history write is logged and swallowed so it can never crash the cron. Exposed at GET /payments/admin/reconciliation-runs, newest first, with the limit clamped to 1–500 (default 50).

Credit-usage spike alert (#1799)
After a usage event is durably recorded and charged, the service compares the tenant's current rolling window total against the average of comparable prior windows. It alerts only when all three signals agree: enough history (CREDITS_USAGE_SPIKE_MIN_SAMPLES, default 3), a meaningful absolute total (CREDITS_USAGE_SPIKE_MIN_AMOUNT, default 10000 minor units, so tiny tenants do not alert on rounding), and at least CREDITS_USAGE_SPIKE_MULTIPLIER (default 5.0) times the baseline. On a spike it logs an ALERT: line, counts managehub_credit_usage_spikes_total on /metrics, and emails SUPPORT_EMAIL using the existing nodemailer/Handlebars pattern. Detection runs strictly after both durable writes and every failure path is swallowed, so an alert problem can never roll back a charge or make a caller retry.

Wallet balance isolation (#1798)
The balance read previously ran a bare SUM aggregate with no transaction and no lock. It now runs inside a transaction that first takes the same pessimistic_write row lock fundCustodialWallet takes, with the aggregate executed through that same transaction manager, so the read serialises against the app's own writers. The Javadoc states the confirmed isolation level (PostgreSQL's default READ COMMITTED, per-connection overridable), exactly what the lock does guarantee, and what it does not cover — writers outside this service that append to wallet_ledger_entries without taking the same lock. Verified that fundCustodialWallet is currently the only ledger writer; there is no settlement/credit wallet-debit path today. A test asserts the read is transaction- and lock-scoped.

Configuration

New optional variables, documented in backend/.env.example:
PAYMENT_CONFIRMATION_MAX_ATTEMPTS, PAYMENT_CONFIRMATION_MAX_PROVIDER_ERROR_STREAK, CREDITS_USAGE_SPIKE_WINDOW_MINUTES, CREDITS_USAGE_SPIKE_MULTIPLIER, CREDITS_USAGE_SPIKE_MIN_AMOUNT, CREDITS_USAGE_SPIKE_MIN_SAMPLES.

Testing

Per the task constraints, no install, build, lint, or test run was performed — the maintainer runs those manually. No new dependencies were added, so package-lock.json is unaffected. Specs were extended in reconciliation.service.spec.ts, metered-usage.service.spec.ts and wallets.service.spec.ts, plus a new dto/reconciliation-run-response.dto.spec.ts.

Reviewer note

MetricsService is provided only in AppModule and is neither exported nor global, while SettlementService and ReconciliationService inject it from their own modules. That looks like a pre-existing dependency-injection problem on main that this PR inherits (rather than introduces) — worth confirming when the backend is next started.

Closing issues

Closes #1801
Closes #1800
Closes #1799
Closes #1798

[vercel]

@zainabbaba31-source is attempting to deploy a commit to the naijabuz's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@zainabbaba31-source Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits