Skip to content

feat(backend): real-time stream anomaly detection & high-velocity drain sentinel (#1469) - #1571

Open
Aj-Kayvee wants to merge 6 commits into
LabsCrypt:mainfrom
Aj-Kayvee:feature/stream-anomaly-sentinel-1469
Open

Aj-Kayvee wants to merge 6 commits into
LabsCrypt:mainfrom
Aj-Kayvee:feature/stream-anomaly-sentinel-1469

Conversation

@Aj-Kayvee

Copy link
Copy Markdown
Contributor

Summary

Implements #1469 — a real-time Stream Anomaly Detection & High-Velocity Drain Sentinel.

Today the indexer records every withdrawal and stream creation only after the fact, so a compromised payroll wallet can drain many streams before anyone notices. This PR turns those indexed events into a live risk signal: a new SentinelService maintains sliding-window velocity trackers, evaluates a set of heuristics on every indexed event, grades the resulting incidents by severity, fans alerts out to on-call channels, and — for CRITICAL incidents — attaches an HMAC-signed set_emergency_pause(true) proposal so multisig signers can act immediately instead of hand-authoring the payload.

Closes #1469

What changed

File Change
backend/src/lib/redis.ts SlidingWindowTracker built on Redis sorted sets (ZADD / ZREMRANGEBYSCORE) with a bounded in-memory fallback. Exposes getRedisClient().
backend/src/services/sentinel.service.ts New. The monitoring engine: velocity trackers, heuristic rules, severity grading, cooldown dedup, incident history, alert adapter, circuit-breaker proposer.
backend/src/workers/soroban-event-worker.ts Feeds tokens_withdrawn and stream_created events into the sentinel (best-effort — analysis failures never quarantine a valid event).
backend/src/controllers/admin.controller.ts listSentinelAlertsHandler, acknowledgeSentinelAlertHandler.
backend/src/routes/v1/admin.routes.ts GET /v1/admin/sentinel/alerts, POST /v1/admin/sentinel/alerts/:id/acknowledge.
backend/src/lib/metrics.ts flowfi_sentinel_incidents_total, flowfi_sentinel_alerts_dispatched_total, flowfi_sentinel_threat_score.
backend/src/config/swagger.ts SentinelIncident schema.
backend/.env.example Documented SENTINEL_* configuration.
backend/tests/sentinel-service.test.ts New. 25 unit tests.
backend/tests/sentinel.admin.test.ts New. 4 API tests.

Sliding-window velocity trackers

RedisSlidingWindowTracker stores one sample per event in a sorted set, scored by timestamp, and answers two questions in a single range read:

  • count — how many events are inside the window,
  • sum — the volume, recovered from the weight encoded in each sorted-set member.

Samples age out with ZREMRANGEBYSCORE on the retention horizon at insert time. Reads are deliberately read-only: a narrow snapshot() must not prune the longer history a baseline snapshot() depends on. When REDIS_URL is unset (single instance, tests) the same interface is served by a bounded in-memory implementation, so detection degrades rather than disappearing.

Heuristics & severity

Rule Trigger
VELOCITY_SPIKE Per-wallet volume in the rolling window ≥ multiplier × the trailing 24h baseline (default 4× = a 300% increase).
TOKEN_VELOCITY_SPIKE Same, aggregated across all wallets for one token — catches distributed drains that individually stay under the bar.
MULTI_STREAM_DRAIN One address withdrawing from more than SENTINEL_MULTI_STREAM_MAX distinct streams inside the ledger window.
HIGH_VALUE_DRAIN A single withdrawal above the USD notional threshold, or one that drains ≥ SENTINEL_BALANCE_DRAIN_PCT of the stream's deposit (unit-free, so it works without a price feed).
STREAM_CREATION_SPIKE Burst of new streams for a single token.
ZERO_RUNWAY_FLOOD More than SENTINEL_ZERO_RUNWAY_FLOOD_THRESHOLD near-zero-runway streams created in a burst.

Severity is LOW / MEDIUM / HIGH / CRITICAL, derived from how far past the threshold the observation is. Repeats of the same rule + subject inside SENTINEL_ALERT_COOLDOWN_MS are suppressed so one attack cannot generate an alert storm.

Alerting & circuit breaker

A unified adapter fans each incident out to Slack, Discord and PagerDuty (any subset configured), with a rich contextual payload — rule, severity, address, token, stream, ledger, evidence, and the threat score. Delivery failures are recorded (flowfi_sentinel_alerts_dispatched_total{channel,outcome}) and logged, but never thrown, so a stale webhook cannot stop the next incident from being detected.

CRITICAL incidents additionally carry circuitBreaker: an HMAC-SHA256-signed set_emergency_pause(true) proposal (payloadHash + signature + reason + incidentId). Signing proves provenance to multisig tooling and cannot be used to move funds.

Admin API

GET  /v1/admin/sentinel/alerts?severity=CRITICAL&address=G...&limit=50
POST /v1/admin/sentinel/alerts/:id/acknowledge

The list endpoint returns the aggregate 0–100 threat score (with a per-severity breakdown), flagged addresses ranked by accumulated risk, and the incident history, newest first. Both routes sit behind the existing requireAdmin + admin rate limiter.

Configuration

All thresholds are env-driven with sane defaults — see the new SENTINEL_* block in backend/.env.example. SENTINEL_ENABLED=false disables the pipeline entirely.

Testing

  • backend/tests/sentinel-service.test.ts — tracker contracts (Redis path via a fake sorted-set client + in-memory), every heuristic, all four severity bands, alert fan-out with a mocked fetch, cooldown suppression, threat score / flagged addresses, and an end-to-end simulated coordinated drain (15 withdrawals across 15 streams in one minute) asserting both VELOCITY_SPIKE and MULTI_STREAM_DRAIN fire.
  • backend/tests/sentinel.admin.test.ts — route-level tests for the dashboard feed, severity filter, limit, and acknowledge/404.
Test Files  61 passed (61)
     Tests  631 passed | 16 skipped (647)

npx tsc --noEmit is clean for backend/.

Acceptance criteria

  • Redis sliding-window velocity tracker calculates withdrawal rate per address and per token.
  • Threshold breaches categorize severity (LOW / MEDIUM / HIGH / CRITICAL).
  • Webhook alerts are dispatched to configured endpoints with rich contextual incident payloads.
  • Admin API exposes active threat scores and flagged transaction lists.
  • Test suite simulates rapid drain attacks and verifies timely alert triggering.

🤖 Generated with Codebuff

Benjamin O. Ajayi and others added 3 commits September 28, 2026 15:10
Institutional payroll and regulated token distributions need audit-ready
proof that participating wallets were screened before funds move. This adds
an opt-in compliance layer that screens stream creation, top-up and
withdrawal requests against OFAC / sanctions data, caches results with a
configurable TTL (Redis when available, bounded in-process cache otherwise)
and records a structured audit trail for blocked interactions. It also
exposes a SEP-0009 KYC attestation endpoint for organizations.

Enforcement is disabled by default (COMPLIANCE_ENFORCEMENT_ENABLED=false)
so local testing and self-hosted deployments stay fully permissive, and a
fail-open / fail-closed mode controls behaviour when a provider is down.

Closes LabsCrypt#1470

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Reconcile the compliance branch with the latest upstream (allowance-based
streams, dispute escrow, rate modification, token-price UI).

Conflict resolution:
- contracts: de-duplicate StreamClosedEvent and remove_stream, unify the fee
  helpers on the (net, fee, treasury) collect_fee + transfer_fee split,
  renumber the merged error variants (upstream 28-33 kept stable, LabsCrypt#1468 variants
  appended as 34-36), set STREAM_FIELD_COUNT to 17 for the merged Stream struct,
  and replace the re-entrant fee transfer test (Soroban forbids contract
  re-entry) with an equivalent balance/persistence check.
- frontend: stop passing `undefined` explicitly to toast.success and update the
  stream-details test for the new token-price hook and background fetches.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Introduce a SentinelService that watches stream lifecycle events as the
indexer processes them and raises severity-graded incidents for
high-velocity drains, rapid multi-stream withdrawals, high-value and
near-total balance drains, per-token withdrawal spikes, stream-creation
spikes, and near-zero-runway floods.

- lib/redis: sliding-window tracker over Redis sorted sets
  (ZADD / ZREMRANGEBYSCORE) with a bounded in-memory fallback so
  detection still works on single-instance deployments and in tests.
- services/sentinel.service: heuristics with configurable thresholds,
  alert-cooldown deduplication, a bounded incident history, an aggregate
  0-100 threat score, and a unified alert adapter (Slack, Discord,
  PagerDuty). CRITICAL incidents also carry an HMAC-signed
  set_emergency_pause(true) proposal for multisig signers.
- workers/soroban-event-worker: feed withdrawals and stream creations
  into the sentinel; analysis is best-effort and never quarantines a
  valid on-chain event.
- admin API: GET /v1/admin/sentinel/alerts exposes the threat score,
  flagged addresses and incident history; POST
  .../alerts/:id/acknowledge marks an incident reviewed.
- tests: 29 new cases covering the trackers, every heuristic and
  severity band, alert fan-out, and a simulated coordinated drain.

Closes LabsCrypt#1469

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@Aj-Kayvee
Aj-Kayvee force-pushed the feature/stream-anomaly-sentinel-1469 branch from 32eb365 to b21b4d1 Compare September 28, 2026 15:38
Aj-Kayvee and others added 3 commits October 5, 2026 16:08
Merging main into this branch unioned both sides of the contract files, so
`lib.rs` kept the superseded `try_invoke` allowance block (leaving an unclosed
`match`), `errors.rs` redeclared variants 28–33, and `test.rs` duplicated a fee
test, the fuzz initializer's dispute fields, and a comment. Soroban Contracts CI
aborted at `cargo fmt` and never reached the tests.

This branch's feature is backend-only, so keep main's already-reconciled,
green contract tree instead of the stacked-branch churn. `cargo fmt`/`clippy`
and all 230 contract tests pass again.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Backend] Real-Time Stream Anomaly Detection & High-Velocity Drain Sentinel Service

1 participant