Add auth metrics and global limits

README.md

@@ -63,6 +63,24 @@ disagree, the spec is the thing to fix.
 Still reading across both networks, and ignoring the parameter:
 `/candles/:assetA/:assetB`, `/price/twap/*`, and `/price/vwap/*`.
 
+### API key authentication errors
+
+API-key authentication failures return HTTP 401 with a machine-readable `code`:
+
+| Code | Meaning |
+|---|---|
+| `missing` | No Authorization header was supplied. |
+| `malformed` | The Authorization header is not a valid Bearer header. |
+| `unknown` | The key was not recognized; never-issued and unknown keys have the same response. |
+| `revoked` | The recognized key has been revoked. |
+| `expired` | The recognized key has passed its expiry time. |
+
+Unknown keys always receive the same response, regardless of whether the key
+ever existed. Admin-token failures use `admin_unauthorized`.
+
+Admin key creation accepts an optional future ISO date-time in `expiresAt`.
+Omitting it creates a key without an expiry.
+
 ```bash
 curl "https://api.example.com/price/XLM/USDC?network=mainnet"
 ```
@@ -108,13 +126,17 @@ The GraphQL schema is exported as a canonical Schema Definition Language (SDL) a
 
 Prometheus metrics are exposed on `GET /metrics` (public, no API key required).
 
-Alongside the ingestion and database metrics, Lens exports the three HTTP
-signals needed to answer "is the API healthy":
+Alongside the ingestion and database metrics, Lens exports HTTP request,
+latency, authentication and rate-limit signals:
 
 | Metric | Type | Labels |
 |---|---|---|
 | `http_requests_total` | Counter | `method`, `route`, `status_class` |
 | `http_request_duration_seconds` | Histogram | `method`, `route` |
+| `api_key_auth_failures_total` | Counter | `reason` |
+| `admin_auth_failures_total` | Counter | — |
+| `rate_limit_rejections_total` | Counter | `scope` |
+| `global_shed_requests_total` | Counter | — |
 
 `route` is the matched route **template** (`/price/:assetA/:assetB`), never the
 resolved URL, so the number of time series stays bounded no matter how many
@@ -515,6 +537,7 @@ curl -s http://localhost:3002/price/XLM/USDC | jq '{price, ammPrice, lastUpdated
 | `NODE_ENV` | Environment mode (`development`, `test`, `production`) | `development` | No |
 | `PORT` | API server port | `3002` | No |
 | `HOST` | API server host | `0.0.0.0` | No |
+| `GLOBAL_RATE_LIMIT_MAX` | Maximum accepted non-health requests per rolling minute per instance | `1000` | No |
 | `DATABASE_URL` | PostgreSQL connection string | - | **Yes** |
 | `REDIS_URL` | Redis connection string | - | **Yes** |
 | `PRICE_CACHE_TTL` | Cache duration for price data (seconds) | `10` | No |

docs/http-metrics.md

@@ -7,10 +7,10 @@ Lens exports Prometheus metrics for the HTTP layer on the existing public
 is different: they are labelled by `network` so the per-network ingester loops
 cannot overwrite each other's series.
 
-These three signals — request rate, error rate and p95 latency — are what answer
-"is the API healthy". The pre-existing `price_requests_total` counts price calls
-specifically and cannot answer that question; it is unchanged and still means
-what it always meant.
+Request rate, error rate, latency, authentication failures and load shedding
+help answer "is the API healthy". The pre-existing `price_requests_total`
+counts price calls specifically and cannot answer that question; it is
+unchanged and still means what it always meant.
 
 The same cardinality rule applies to every metric family: labels must come from
 fixed buckets or server-defined values. Never put raw request paths, asset
@@ -37,6 +37,53 @@ Buckets: `0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10`.
 
 Observed for every request that produces a response, successful or not.
 
+### `api_key_auth_failures_total`
+
+Counter. Label: `reason`, with the bounded values `missing`, `malformed`,
+`unknown`, `revoked` and `expired`. The `reason` value matches the
+machine-readable `code` in the corresponding API-key 401 response.
+
+### `admin_auth_failures_total`
+
+Counter. No labels. This tracks rejected authentication attempts on admin
+routes independently of API-key failures.
+
+### Rate-limit counters
+
+`rate_limit_rejections_total` is a counter labelled by the bounded `scope`
+values `key` and `ip`; it counts requests rejected by the per-key or fallback
+IP limiter. Those requests return HTTP 429 and code `PER_KEY_RATE_LIMITED` or
+`IP_RATE_LIMITED`.
+
+`global_shed_requests_total` is an unlabeled counter for requests shed by the
+per-instance global ceiling. Those requests return HTTP 503 with code
+`GLOBAL_RATE_LIMITED`.
+
+Authentication metrics never include keys, hashes, tokens, key prefixes or
+caller addresses as labels.
+
+## Global request ceiling
+
+`GLOBAL_RATE_LIMIT_MAX` configures the maximum accepted non-health requests
+per rolling 60-second window on each instance; it defaults to `1000` and must
+be a positive integer. The global check runs after the per-key/IP limiter, so
+per-key 429 responses remain distinct and are not counted against the global
+ceiling.
+
+At 80% of the ceiling, rounded up, the service starts shedding expensive
+route templates (GraphQL, price, route, pools, candles, history, screener,
+benchmark, oracle, compare, basket, venues, spreads, volumes, book, verify
+and settle) to preserve room for other routes. Once the ceiling is full, it
+sheds all non-health routes.
+`/status`, `/live` and `/ready` are always exempt. Shed requests include a
+`Retry-After` header.
+
+The ceiling is **per instance**, not deployment-wide. With `N` equally
+configured instances the approximate aggregate ceiling is
+`GLOBAL_RATE_LIMIT_MAX x N`; choose the per-instance value based on instance
+capacity and expected autoscaling so scaling out does not imply a higher
+deployment-wide traffic budget than intended.
+
 ## Why the labels look like this
 
 Cardinality is the whole design constraint. A histogram labelled by `req.url`

prisma/schema.prisma

@@ -263,6 +263,7 @@ model ApiKey {
   ratePerMin       Int       @default(60) @map("rate_per_min")
   ratePerDay       Int       @default(10000) @map("rate_per_day")
   revokedAt        DateTime? @map("revoked_at")
+  expiresAt        DateTime? @map("expires_at")
   createdAt        DateTime  @default(now()) @map("created_at")
   monthlyQuotaCents Int?     @default(10000) @map("monthly_quota_cents")
   dailyQuotaCents  Int?      @default(500) @map("daily_quota_cents")

src/__tests__/auth.test.ts

@@ -22,6 +22,7 @@ import {
   lookupApiKey,
 } from '../api/auth'
 import { registerAdminRoutes, generateApiKey } from '../api/admin'
+import { register, api_key_auth_failures_total, admin_auth_failures_total } from '../metrics'
 
 const mockFindUnique = prisma.apiKey.findUnique as unknown as ReturnType<typeof vi.fn>
 const mockCreate = prisma.apiKey.create as unknown as ReturnType<typeof vi.fn>
@@ -31,6 +32,11 @@ function sha256(s: string): string {
   return createHash('sha256').update(s, 'utf8').digest('hex')
 }
 
+async function apiKeyAuthFailureCount(reason: string): Promise<number> {
+  const metric = await api_key_auth_failures_total.get()
+  return metric.values.find((value) => value.labels.reason === reason)?.value ?? 0
+}
+
 /** Builds an app with auth enabled + one protected and one public route. */
 async function buildAuthedApp() {
   const app = Fastify()
@@ -90,18 +96,47 @@ describe('lookupApiKey', () => {
     })
     expect(await lookupApiKey('lens_plain')).toBeNull()
   })
+
+  it('returns null for an expired key', async () => {
+    mockFindUnique.mockResolvedValue({
+      id: 'key-1',
+      label: 'acme',
+      ratePerMin: 60,
+      ratePerDay: 1000,
+      revokedAt: null,
+      expiresAt: new Date(Date.now() - 1000),
+    })
+    expect(await lookupApiKey('lens_plain')).toBeNull()
+  })
 })
 
 describe('API key auth hook', () => {
   it('returns 401 when no key is provided', async () => {
+    const before = await apiKeyAuthFailureCount('missing')
     const app = await buildAuthedApp()
     const res = await app.inject({ method: 'GET', url: '/price/test' })
     expect(res.statusCode).toBe(401)
     expect(res.json()).toMatchObject({ error: 'Unauthorized' })
+    expect(res.json().code).toBe('missing')
+    expect(await apiKeyAuthFailureCount('missing')).toBe(before + 1)
     expect(mockFindUnique).not.toHaveBeenCalled()
   })
 
+  it('counts malformed authorization separately from missing credentials', async () => {
+    const before = await apiKeyAuthFailureCount('malformed')
+    const app = await buildAuthedApp()
+    const res = await app.inject({
+      method: 'GET',
+      url: '/price/test',
+      headers: { authorization: 'Basic value' },
+    })
+    expect(res.statusCode).toBe(401)
+    expect(res.json()).toMatchObject({ code: 'malformed' })
+    expect(await apiKeyAuthFailureCount('malformed')).toBe(before + 1)
+  })
+
   it('returns 401 for an invalid key', async () => {
+    const before = await apiKeyAuthFailureCount('unknown')
     mockFindUnique.mockResolvedValue(null)
     const app = await buildAuthedApp()
     const res = await app.inject({
@@ -110,9 +145,13 @@ describe('API key auth hook', () => {
       headers: { authorization: 'Bearer wrong' },
     })
     expect(res.statusCode).toBe(401)
+    expect(res.json().code).toBe('unknown')
+    expect(res.json().message).toBe('Unknown API key.')
+    expect(await apiKeyAuthFailureCount('unknown')).toBe(before + 1)
   })
 
   it('returns 401 for a revoked key', async () => {
+    const before = await apiKeyAuthFailureCount('revoked')
     mockFindUnique.mockResolvedValue({
       id: 'key-1', label: 'acme', ratePerMin: 60, ratePerDay: 1000, revokedAt: new Date(),
     })
@@ -123,6 +162,31 @@ describe('API key auth hook', () => {
       headers: { authorization: 'Bearer revoked' },
     })
     expect(res.statusCode).toBe(401)
+    expect(res.json().code).toBe('revoked')
+    expect(res.json().message).toBe('Revoked API key. Request a replacement key.')
+    expect(await apiKeyAuthFailureCount('revoked')).toBe(before + 1)
+  })
+
+  it('returns a distinct code and message for an expired key', async () => {
+    const before = await apiKeyAuthFailureCount('expired')
+    mockFindUnique.mockResolvedValue({
+      id: 'key-1',
+      label: 'acme',
+      ratePerMin: 60,
+      ratePerDay: 1000,
+      revokedAt: null,
+      expiresAt: new Date(Date.now() - 1000),
+    })
+    const app = await buildAuthedApp()
+    const res = await app.inject({
+      method: 'GET',
+      url: '/price/test',
+      headers: { authorization: 'Bearer test-expired-api-key' },
+    })
+    expect(res.statusCode).toBe(401)
+    expect(res.json().code).toBe('expired')
+    expect(res.json().message).toBe('Expired API key. Request a replacement key.')
+    expect(await apiKeyAuthFailureCount('expired')).toBe(before + 1)
   })
 
   it('allows a valid key through and attaches metadata', async () => {
@@ -207,12 +271,14 @@ describe('admin endpoints', () => {
   })
 
   it('rejects minting without the admin token', async () => {
+    const before = (await admin_auth_failures_total.get()).values[0]?.value ?? 0
     const app = await buildAdminApp()
     const res = await app.inject({
       method: 'POST', url: '/admin/keys', payload: { label: 'x' },
     })
     expect(res.statusCode).toBe(401)
     expect(mockCreate).not.toHaveBeenCalled()
+    expect((await admin_auth_failures_total.get()).values[0]?.value).toBe(before + 1)
   })
 
   it('mints a key and returns the plaintext once (stored as hash only)', async () => {
@@ -280,4 +346,10 @@ describe('admin endpoints', () => {
     })
     expect(res.statusCode).toBe(404)
   })
+
+  it('registers authentication counters in the Prometheus registry', async () => {
+    const output = await register.metrics()
+    expect(output).toContain('api_key_auth_failures_total')
+    expect(output).toContain('admin_auth_failures_total')
+  })
 })

src/__tests__/globalRateLimit.test.ts

@@ -0,0 +1,68 @@
+import { afterEach, describe, expect, it } from 'vitest'
+import Fastify from 'fastify'
+import rateLimit from '@fastify/rate-limit'
+import { registerGlobalRateLimit } from '../middleware/globalRateLimit'
+import { global_shed_requests_total } from '../metrics'
+
+async function buildApp(globalMax: number, perKeyMax = 10) {
+  const app = Fastify()
+  await app.register(rateLimit, {
+    max: perKeyMax,
+    timeWindow: '1 minute',
+    keyGenerator: () => 'test-key',
+    allowList: (req) => ['/status', '/live', '/ready'].includes(req.url),
+  })
+  registerGlobalRateLimit(app, globalMax)
+  app.get('/work', async () => ({ ok: true }))
+  app.get('/cheap', async () => ({ ok: true }))
+  app.get('/price/test', async () => ({ ok: true }))
+  app.get('/status', async () => ({ ok: true }))
+  app.get('/live', async () => ({ ok: true }))
+  app.get('/ready', async () => ({ ok: true }))
+  await app.ready()
+  return app
+}
+
+describe('global request ceiling', () => {
+  let app: Awaited<ReturnType<typeof buildApp>> | undefined
+
+  afterEach(async () => {
+    await app?.close()
+    app = undefined
+  })
+
+  it('returns a distinct global response while the caller remains within its per-key quota', async () => {
+    app = await buildApp(1, 10)
+    const before = (await global_shed_requests_total.get()).values[0]?.value ?? 0
+
+    expect((await app.inject({ method: 'GET', url: '/work' })).statusCode).toBe(200)
+    const shed = await app.inject({ method: 'GET', url: '/work' })
+
+    expect(shed.statusCode).toBe(503)
+    expect(shed.json()).toMatchObject({ code: 'GLOBAL_RATE_LIMITED' })
+    expect(shed.headers['retry-after']).toBeDefined()
+    expect((await global_shed_requests_total.get()).values[0]?.value).toBe(before + 1)
+  })
+
+  it('sheds expensive routes before cheap routes as the ceiling approaches', async () => {
+    app = await buildApp(5)
+
+    for (let i = 0; i < 4; i++) {
+      expect((await app.inject({ method: 'GET', url: '/cheap' })).statusCode).toBe(200)
+    }
+    expect((await app.inject({ method: 'GET', url: '/price/test' })).json()).toMatchObject({
+      code: 'GLOBAL_RATE_LIMITED',
+    })
+    expect((await app.inject({ method: 'GET', url: '/cheap' })).statusCode).toBe(200)
+    expect((await app.inject({ method: 'GET', url: '/cheap' })).statusCode).toBe(503)
+  })
+
+  it('never sheds health endpoints', async () => {
+    app = await buildApp(1)
+    expect((await app.inject({ method: 'GET', url: '/work' })).statusCode).toBe(200)
+
+    for (const path of ['/status', '/live', '/ready']) {
+      expect((await app.inject({ method: 'GET', url: path })).statusCode).toBe(200)
+    }
+  })
+})