🛡️ HookGuard

Webhook engine with HMAC signatures, idempotency, retries, replay protection, and tests.

HookGuard is a small, self-contained reference implementation of the webhook receiver every integration with Stripe, GitHub, or a payment/subscription provider eventually needs to build. It is deliberately scoped to run with minimal infrastructure — a single PostgreSQL database, no message broker, no paid services — while implementing the same correctness guarantees a production ingestion endpoint needs.

✨ Features

• HMAC signature verification — timing-safe comparison, secret bound per-provider so one provider's secret can't validate another provider's traffic.
• Timestamp-bounded replay protection — an in-memory guard rejects a byte-identical request (same signature) seen again within the tolerance window.
• Idempotent event storage — every event is keyed by the provider's own event ID; a legitimate redelivery is acknowledged without being reprocessed.
• Retry-with-backoff pipeline — a failing handler is retried with linear backoff up to WEBHOOK_MAX_DELIVERY_ATTEMPTS, then parked in dead_letter instead of retried forever.
• Replay CLI — list, redrive one, or redrive every dead-lettered event through the exact same processing pipeline the webhook route uses.
• Fixed-window rate limiting — a minimal per-IP limiter in front of the webhook route; stale windows are swept on every request so memory stays bounded, and the IP it keys on is only ever as trustworthy as TRUST_PROXY_HOPS says the deployment is.
• Crash recovery for in-flight events — on startup, before accepting traffic, HookGuard resumes any event left in processing by a previous crash instead of leaving it stuck forever.
• Runtime schema validation — every inbound payload is parsed through a Zod discriminated union before it's trusted.
• PostgreSQL behind an interface — EventStore is the only thing the rest of the codebase depends on; all SQL is parameterized, never string-built.
• Structured logging — Pino with header/secret redaction built in.
• Strict TypeScript, zero-warning ESLint, enforced coverage thresholds — see the Testing and Code quality and security sections below.

🧰 Tech stack

Technology	Role
Express 5	HTTP server and routing
TypeScript (strict)	Type safety
PostgreSQL + pg	Event store (idempotency, retries, dead-letter)
Zod	Runtime validation (webhook payloads, env vars)
Pino / pino-http	Structured logging with secret redaction
Vitest + Supertest	Unit and end-to-end HTTP tests
ESLint, Prettier, Husky, commitlint	Code quality and commit-message gates
GitHub Actions	CI: commit-lint, quality, tests, build, audit

⚠️ Problem statement

A webhook endpoint sits on the public internet and accepts POST requests claiming to be from a trusted provider. Naively implemented, that endpoint is vulnerable in ways that are easy to miss until they cause an incident:

• 🎭 Anyone can POST to the URL. Without a signature check, an attacker can forge events (fake payment.succeeded events, for example) and drive business logic directly.
• 🔁 A captured request can be replayed. Even a validly signed request, if intercepted, can be resent later to trigger the same effect twice — refunding a payment twice, reactivating a cancelled subscription, etc.
• 📨 Providers redeliver on timeout. If your endpoint is slow or the response is lost in transit, the provider will retry the same event, often several times. If processing isn't idempotent, redelivery duplicates side effects (double-charging a ledger, sending a duplicate email).
• 💥 Downstream failures shouldn't become data loss. If your database is briefly down when an event arrives, that event needs a path to be retried and, if it keeps failing, parked somewhere a human can inspect it — not silently dropped or endlessly retried against a request that will never succeed.

HookGuard addresses each of these with a small, auditable amount of code: HMAC verification with a timestamp-bounded replay cache, an idempotency store keyed by the provider's event ID, and a retry-with-backoff pipeline that terminates in a dead-letter state with a CLI to redrive it.

🏗️ Architecture

sequenceDiagram
    participant P as Provider
    participant HG as HookGuard
    participant RG as Replay guard (in-memory)
    participant DB as Event store (PostgreSQL)
    participant H as Handler

    P->>HG: POST /webhooks/:provider<br/>(signed body + timestamp)
    HG->>HG: verify HMAC signature (timing-safe)
    alt signature invalid
        HG-->>P: 401 invalid_signature
    end
    HG->>HG: check timestamp within tolerance
    alt timestamp stale
        HG-->>P: 400 stale_timestamp
    end
    HG->>RG: register(provider, timestamp, signature)
    alt already seen
        RG-->>HG: replay detected
        HG-->>P: 409 replay_detected
    end
    HG->>HG: validate JSON against event schema
    alt schema invalid
        HG-->>P: 422 invalid_payload
    end
    HG->>DB: insertPending(event.id)
    alt event.id already stored
        DB-->>HG: duplicate
        HG-->>P: 200 duplicate (ignored, not reprocessed)
    end
    loop up to WEBHOOK_MAX_DELIVERY_ATTEMPTS
        HG->>H: handle(event)
        alt handler succeeds
            H-->>HG: ok
            HG->>DB: markProcessed
        else handler throws
            H-->>HG: error
            HG->>DB: markFailed (failed | dead_letter)
            HG->>HG: backoff before next attempt
        end
    end
    HG-->>P: 200 { status, eventId, attempts }

Loading

If every attempt fails, the event lands in dead_letter. It stays there until an operator runs the replay CLI (typically after fixing the bug or confirming a downstream dependency has recovered), which resets its attempt count and runs it back through the exact same pipeline.

An event can also be observed in processing — set right before handle(event) runs — if the process crashes mid-attempt. Nothing else ever revisits that event (a legitimate redelivery of the same event ID is short-circuited as a duplicate before it would re-enter this pipeline), so on every startup, before the server accepts traffic, reclaimStuckEvents lists everything still in processing and resumes it through this same pipeline.

🚨 Failure modes

Scenario	Where it's caught	Response
Missing signature/timestamp header	verifySignature	🔒 400 invalid_signature
Signature doesn't match body	verifySignature (timing-safe)	🔒 401 invalid_signature
Body tampered with after signing	verifySignature	🔒 401 invalid_signature
Timestamp older than tolerance window	isTimestampFresh	⏱️ 400 stale_timestamp
Identical request replayed (same sig + ts)	InMemoryReplayGuard	🔁 409 replay_detected
Body is not valid JSON	webhook route	🚫 422 invalid_json
Body doesn't match the event schema	webhookEventSchema (Zod)	🚫 422 invalid_payload
Same event ID delivered again (legit redelivery)	EventStore.insertPending	♻️ 200, status duplicate, no reprocessing
Unknown :provider in the URL	webhook route	❓ 404 unknown_provider
Too many requests from one source	createRateLimiter	🚦 429 rate_limited
Handler throws, attempts remain	processWithRetries	🔄 retried with linear backoff
Handler throws on the final attempt	processWithRetries	🪦 event moved to dead_letter, still 200 to the provider
Unexpected internal error	Express error handler	💥 500 internal_error (never leaks detail)

Note that a downstream processing failure still returns 200 to the provider: retrying a deterministic bug by asking the provider to redeliver accomplishes nothing, which is exactly why dead-letter storage exists as a separate concern from the HTTP contract.

📁 Project structure

src/
  config/env.ts            Validates process.env with Zod; generic errors, never echoes secrets.
  logging/logger.ts         Pino logger with header/secret redaction.
  webhooks/
    schemas.ts              Zod discriminated union for the three demo event types.
    signature.ts             HMAC-SHA256 signing/verification, timing-safe, plus timestamp check.
    replay-guard.ts           In-memory cache blocking a byte-identical delivery seen before.
    handlers.ts               Demo business logic; idempotent by construction.
  db/
    event-store.ts            EventStore interface implemented by PostgresEventStore.
    postgres-event-store.ts   PostgreSQL implementation via the `pg` driver.
  queue/
    processing.ts             Single attempt + retry-with-backoff loop to a terminal state, plus
                               reclaimStuckEvents (crash recovery for events left in `processing`).
    dead-letter.ts             Listing and redriving dead-lettered events.
  http/
    app.ts                     Express app wiring: logging, rate limiting, routes, error handler.
    webhook-router.ts           POST /webhooks/:provider — the pipeline in the sequence diagram.
    rate-limit.ts                Minimal fixed-window limiter keyed by source IP, self-evicting.
  replay/cli.ts                CLI entry point for --list / --id / --all.
  index.ts                     Process entry point: config, store, crash-recovery reclaim, server,
                                graceful shutdown.
scripts/send-fixture.ts     Signs and sends a fixture file for manual/demo use.
fixtures/                   Sample payment.succeeded / payment.failed / subscription.updated events.
tests/                      Vitest suite covering every module above.

🔑 Environment variables

Variable	Purpose	Default
NODE_ENV	development | test | production	development
PORT	HTTP port the server listens on	3000
WEBHOOK_SIGNING_SECRETS	Comma-separated provider_id:secret pairs	(required, no default)
WEBHOOK_TIMESTAMP_TOLERANCE_SECONDS	Max age of a signed request's timestamp before it's rejected	300
WEBHOOK_MAX_DELIVERY_ATTEMPTS	Attempts before an event moves to dead_letter	5
WEBHOOK_RATE_LIMIT_MAX	Requests allowed per source IP within the rate-limit window	100
WEBHOOK_RATE_LIMIT_WINDOW_MS	Rate-limit window, in milliseconds	60000
DATABASE_URL	PostgreSQL connection string	postgres://postgres:postgres@localhost:5432/hookguard
TRUST_PROXY_HOPS	Reverse-proxy hops to trust for req.ip (rate limiting)	0

src/config/env.ts validates every one of these at startup via Zod and throws one generic error if any is missing or malformed, so a bad value never leaks into a raw error message. .env.example documents all of them — copy it to .env and fill in real values.

💻 Local development

Requires Node.js 24.x, npm 11.x (see .nvmrc), and a running PostgreSQL server.

npm ci
cp .env.example .env
# Create the database if it doesn't exist yet, then edit DATABASE_URL in .env to match:
createdb hookguard
npm run dev

HookGuard creates its own events table on startup — no separate migration step is needed.

The server listens on PORT (default 3000) and validates two demo provider secrets out of the box (see .env.example). 📬 Send it a signed sample event:

npm run send:fixture -- payment.succeeded
npm run send:fixture -- payment.failed
npm run send:fixture -- subscription.updated

scripts/send-fixture.ts signs the named file from fixtures/ with the same HMAC scheme the server verifies, so it's a realistic stand-in for a provider's webhook sender.

⚙️ Available commands

Command	Purpose
npm run dev	Start the dev server (tsx watch, loads .env).
npm start	Run the compiled dist/ build (loads .env).
npm run build	Compile src/ to dist/.
npm run replay -- --list	Show everything parked in dead_letter.
npm run replay -- --id	Redrive one dead-lettered event by ID.
npm run replay -- --all	Redrive every dead-lettered event.
npm run send:fixture	Sign and send a fixture file to a running server.
npm run quality	Format check + lint + typecheck.
npm run lint	ESLint (zero warnings allowed).
npm run format / format:check	Prettier write / check.
npm run typecheck	tsc --noEmit.
npm test	Vitest with coverage (needs DATABASE_URL reachable).
npm run test:watch	Vitest in watch mode.
npm run security:audit	npm audit --audit-level=high.

🔁 Replaying dead-lettered events

npm run replay -- --list                 # show everything parked in dead_letter
npm run replay -- --id evt_abc123        # redrive one event
npm run replay -- --all                  # redrive every dead-lettered event

Redriving resets the event's attempt counter and status to received, then runs it through the exact same processWithRetries pipeline the webhook route uses — there is only one retry code path in the codebase, not a separate "replay" implementation to keep in sync.

🧪 Testing

npm test          # vitest, with enforced coverage thresholds (needs DATABASE_URL reachable)

Each test file connects to the PostgreSQL instance from DATABASE_URL but works in its own randomly-named schema, so the suite is safe to run in parallel without a separate database per test file — and safe to point at the same database you use for npm run dev. Coverage thresholds are enforced in vitest.config.mts (80% branches, 85% functions/lines/statements).

The suite covers every module end-to-end through the real Express app (via supertest), plus focused unit tests:

• tests/signature.test.ts — valid signature accepted; wrong secret, tampered body, missing headers, malformed v1= prefix, and non-numeric timestamp all rejected; unknown provider 404s.
• tests/timestamp.test.ts — a stale timestamp is rejected; one just inside the tolerance window is accepted.
• tests/idempotency.test.ts — a second delivery of the same event ID (fresh signature/timestamp) is ignored as duplicate instead of reprocessed.
• tests/validation.test.ts — an unknown event type, a payload missing required fields, and malformed JSON are all rejected with the right error code.
• tests/dead-letter.test.ts — a handler that fails on every attempt moves its event to dead_letter with the error recorded.
• tests/replay.test.ts — the replay pipeline reprocesses a dead-lettered event successfully once the underlying handler is fixed.
• tests/reclaim.test.ts — an event left in processing (simulating a crash mid-attempt) is resumed to processed by reclaimStuckEvents; a no-op when nothing is stuck.
• tests/rate-limit.test.ts — requests under the limit pass; the request that exceeds it gets 429 with a Retry-After header.
• tests/config.test.ts — valid env parses correctly; defaults apply when optional vars are omitted; malformed config throws one generic error that never echoes the invalid value.
• tests/handlers.test.ts — the demo handler logs each known event type and throws on simulateFailure or a custom override.
• tests/app.test.ts — the health check responds, and an unexpected error (closed store) surfaces as a generic 500 instead of leaking detail.

🔒 Code quality and security

• 🔐 Signatures are compared with crypto.timingSafeEqual; secrets are never logged (Pino redaction covers the signature header and any field literally named secret/signingSecrets).
• 🙈 Environment validation failures raise one generic error and never include the invalid value, so a misconfigured secret can't leak into logs or error responses.
• 🧷 All SQL is parameterized through pg's query placeholders — no string-built queries.
• 📦 The webhook body is capped at 256 KB (express.raw({ limit: '256kb' })) before anything else touches it.
• 🚦 A fixed-window rate limiter sits in front of the webhook route to blunt naive floods; expired per-IP windows are swept on every request instead of accumulating for the life of the process.
• 🕵️ trust proxy defaults to 0 (TRUST_PROXY_HOPS), so req.ip — the rate limiter's key — is the raw socket address unless an operator explicitly declares how many proxy hops sit in front of HookGuard and are trusted to set X-Forwarded-For themselves; otherwise the header is attacker-controlled and the limiter is trivially bypassed.
• 🛡️ npm run security:audit fails the build on any high-severity advisory; CI runs it on every push.

A Husky pre-commit hook runs Gitleaks secret scanning against the staged diff, lint-staged (ESLint --fix + Prettier on staged files), the full quality check, and a production build — before any commit is allowed through. commit-msg enforces Conventional Commits via commitlint. The same gates run in CI (.github/workflows/ci.yml): a commitlint job on every pull request, and a quality job (install → format check → lint → typecheck → test with coverage against a PostgreSQL service container → build → npm audit) on every push — so a bypassed or missing local hook still gets caught before anything merges.

🧠 Design decisions

• 🔑 HMAC scheme. The signed string is ${provider}.${timestamp}.${rawBody}, HMAC-SHA256'd with a per-provider secret, sent as X-HookGuard-Signature: v1=<hex> alongside X-HookGuard-Timestamp. Binding the provider ID into the signed payload stops a secret issued for one provider from validating traffic on another provider's path. Comparison uses crypto.timingSafeEqual, never ===, to avoid a timing side-channel on the signature.
• 🔀 Replay protection vs. idempotency are two different mechanisms. The replay guard blocks a byte-identical HTTP request (same signature) from being processed twice within the timestamp tolerance window — this is a transport-layer defense against a captured request being resent. Idempotency is a business-layer concern: the same logical event (by id), even if legitimately redelivered with a fresh timestamp and signature, must not be processed twice. Both are needed; neither substitutes for the other.
• 🐘 PostgreSQL behind an interface. EventStore is the only thing the webhook route, the processor, and the replay CLI depend on. PostgresEventStore is the only implementation today, using the pg driver directly (no ORM). Swapping in a different store means writing one class against the same interface — nothing else in the codebase changes.
• 📨 Retries return 200. A webhook endpoint acknowledges receipt, not successful processing. Returning a 5xx to make a provider retry a deterministic handler bug just adds noise; the dead-letter state is HookGuard's own mechanism for "this needs a human," decoupled from the HTTP contract with the provider.
• 🧪 data.simulateFailure on the demo fixtures exists purely so the retry and dead-letter paths are testable deterministically instead of depending on flaky randomness — it has no effect in the signature/schema/idempotency layers, only inside the demo handler.
• 🩹 Crash recovery reuses the same retry pipeline, not a separate mechanism. An event stuck in processing is, by construction, evidence of a prior crash — nothing else can produce that state and nothing else revisits it. Rather than a bespoke recovery path, reclaimStuckEvents just finds those events and feeds them back into processWithRetries, the same function the webhook route and the replay CLI already use.

🐛 Troubleshooting

• ECONNREFUSED / server won't start — PostgreSQL isn't reachable at DATABASE_URL. Confirm the server is running and the database exists (createdb hookguard).
• password authentication failed for user "postgres" — the credentials in .env don't match your local Postgres role; update DATABASE_URL to the correct user/password.
• npm test fails or hangs on every file — the suite needs a live PostgreSQL instance at DATABASE_URL; there's no in-memory fallback. Each test file creates and drops its own schema, so it's safe to point it at the same database you use for npm run dev.
• A real request gets 401 invalid_signature — the raw body must reach HookGuard byte-for-byte; a proxy or body-parser that re-serializes JSON before signature verification will break the signature. Also double-check the provider/secret pairing in WEBHOOK_SIGNING_SECRETS.
• An event won't come back from dead_letter after replay — the replay CLI runs the handler against the payload exactly as it was stored; if that payload itself triggers the failure (e.g. a demo fixture with simulateFailure: true), fix the underlying issue (or the payload) before redriving.
• Husky pre-commit fails with "gitleaks ... is required" — install Gitleaks 8.30.x and make sure it's on PATH.

🚀 Future improvements

• Shared (e.g. Redis-backed) replay guard and rate limiter, so HookGuard can run as more than one instance without losing that in-memory state.
• Versioned SQL migrations (e.g. node-pg-migrate) once the schema needs to evolve beyond CREATE TABLE IF NOT EXISTS.
• Structured metrics/observability (attempt counts, dead-letter rate) alongside the existing Pino logs.
• Additional provider event schemas beyond the three demo types.
• Authentication on the replay CLI (or any future admin surface) once it's exposed beyond a local operator's terminal.

📄 License

MIT — see LICENSE.