Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
fdd66c4
test(people): define liveness and readiness contract
seonghobae Aug 22, 2026
831de23
feat(people): add liveness and owned-db readiness probes
seonghobae Aug 22, 2026
7b2e193
feat(people): export operability probe contracts
seonghobae Aug 22, 2026
331989f
docs(people): explain probe failure domains
seonghobae Aug 22, 2026
33d0033
docs(operability): trace People probe semantics
seonghobae Aug 22, 2026
0f1b755
docs(operability): record primary probe references
seonghobae Aug 22, 2026
ccd736d
test(operability): require canonical probe runbook
seonghobae Aug 22, 2026
ec49aec
docs(operability): bind People probes to operator runbook
seonghobae Aug 22, 2026
dd862fe
fix(provenance): seal operability runbook evidence
seonghobae Aug 22, 2026
7a86081
test(operability): accept mapping readiness rows
seonghobae Aug 22, 2026
f27420b
fix(operability): decouple readiness from cursor row shape
seonghobae Aug 22, 2026
82a1f43
test(operability): make readiness result row-shape neutral
seonghobae Aug 22, 2026
82c8ef9
test(operability): pin driver-independent readiness traceability
seonghobae Aug 28, 2026
7847f3a
docs(operability): align readiness traceability with row factories
seonghobae Aug 28, 2026
303133e
test(operability): expose readiness event-loop blocking
seonghobae Aug 28, 2026
ac7b947
fix(operability): offload readiness probe from event loop
seonghobae Aug 28, 2026
471d558
test(operability): model blocking readiness concurrency
seonghobae Aug 28, 2026
7d75e68
docs(operability): trace readiness concurrency boundary
seonghobae Aug 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/OPERABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@
- High-impact command audit append success: 99.99% within accepted maintenance windows.
- Integration adapter error visibility: every failed outbound command produces an operator-safe event.

### People API liveness and readiness

- `GET /health` is process-liveness evidence only. It must not call PostgreSQL, Keyverse, Naruon, or another external dependency; a healthy process returns `200` with the bounded body `{"status":"ok"}`. Orchestrators may use this route for restart decisions without turning a transient database outage into a restart loop.
- `GET /ready` is traffic-eligibility evidence for the People API's owned PostgreSQL dependency. The current adapter opens a read-only transaction, issues only `SET TRANSACTION READ ONLY` and `SELECT 1`, and does not query HR application tables or require tenant context. A successful check returns `200` with `{"status":"ready"}`.
- A readiness dependency failure returns bounded HTTP `503` evidence with `error=not_ready` and a retry-oriented next action. The response must not disclose database exception text, credentials, connection strings, SQL details, tenant identifiers, or HR values.
- Readiness deliberately does not probe read-only dedicated-writer dependencies such as Keyverse or Naruon. Their outages retain their separately governed fail-closed/degraded-mode semantics and must not be converted into an Orgmetra process-liveness failure.
- These routes are an executable operability contract, not a deployment or certification claim. Production orchestration still requires reviewed startup/liveness/readiness timing, network policy, resource limits, telemetry, SLO alerting, and recovery evidence on the released artifact.

## Degraded modes

### Keyverse unavailable
Expand Down
15 changes: 15 additions & 0 deletions docs/doctoring/people-api-operability-references.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# People API operability references

## Scope

These references support the active People API liveness/readiness design. They are design evidence only; they do not claim Kubernetes conformance, certification, production deployment, or release acceptance.

## APA 7 references

Kubernetes Authors. (2026). *Liveness, readiness, and startup probes*. Kubernetes. https://kubernetes.io/docs/concepts/workloads/pods/probes/

Kubernetes Authors. (2026). *Configure liveness, readiness and startup probes*. Kubernetes. https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

## Design use

The Kubernetes documentation distinguishes liveness, which can trigger container restart, from readiness, which removes an unavailable Pod from service traffic. It also cautions that incorrectly coupling liveness to transient dependencies can create cascading failures and describes the pattern where readiness additionally verifies required backend services. Orgmetra applies that distinction by keeping `/health` dependency-free and binding `/ready` only to the People API's owned PostgreSQL availability check.
26 changes: 26 additions & 0 deletions docs/traceability/people-api-operability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# People API operability traceability

## Status

This document describes the active PR that introduces executable People API probe semantics. Protected `develop@9e3e4847510e1e612b48474ba42b177b8ed824df` does not yet contain these routes, and this document does not claim production deployment or release readiness.

## Requirement-to-evidence map

| Requirement | Active implementation | Executable evidence | Failure rule |
|---|---|---|---|
| Process liveness must not depend on PostgreSQL availability | `PeopleOperabilityAsgiApp` `GET /health` | `test_health_is_live_without_touching_owned_dependencies` | PostgreSQL failure cannot make liveness fail or trigger a dependency call. |
| Traffic readiness must reflect the service-owned PostgreSQL dependency | `PeopleOperabilityAsgiApp` `GET /ready` + `PostgresReadinessProbe` | `test_ready_checks_owned_dependency_and_reports_success`, `test_postgres_probe_requires_callable_factory_and_a_result_row`, `test_postgres_probe_accepts_mapping_row_factory` | Readiness is 200 only after the reviewed read-only probe produces a row regardless of DB-API row factory. |
| Synchronous readiness work must not monopolize the ASGI event loop | `PeopleOperabilityAsgiApp` offloads `ReadinessProbe.check_ready` with `asyncio.to_thread` | `test_ready_keeps_event_loop_live_during_sync_probe` | The connection factory, cursor, transaction, and `SELECT 1` execute together on the worker thread while the ASGI loop remains schedulable. |
| Backend failure details must not cross the HTTP boundary | bounded `503 not_ready` response | `test_ready_normalizes_dependency_failure_without_leaking_details` | Any owned-dependency exception becomes a stable 503 with a next operator action and no backend message. |
| Probe routes must not become a hidden HR-data or identity integration path | no tenant context, credentials, HR table SQL, or foreign-service calls | source contract plus route tests | `/health` and `/ready` expose status only; Keyverse and other dedicated-writer services remain outside this boundary. |
| Unsupported transport shapes must stay bounded | exact GET-only `/health` and `/ready` surface | `test_unknown_route_and_wrong_method_are_bounded_transport_errors`, `test_non_http_scope_is_rejected_as_a_programming_error` | Unknown route is 404, wrong method is 405 with `Allow: GET`, non-HTTP ASGI is rejected. |

## Operational interpretation

Kubernetes distinguishes liveness from readiness: liveness is used to decide when a container should be restarted, while readiness determines whether it should receive traffic. The official Kubernetes probe guidance specifically warns that an incorrect liveness dependency can cause cascading restarts and notes that a strict backend dependency can be checked by readiness while liveness continues to reflect the application itself. Orgmetra therefore keeps PostgreSQL out of `/health` and checks it only in `/ready`.

The PostgreSQL readiness query deliberately proves infrastructure availability only. It does not prove tenant authorization, HR-data correctness, migration compatibility, downstream integrations, or release acceptance. The constant `SELECT 1` contract requires a result row but does not prescribe whether a DB-API driver represents that row as a tuple, mapping, or another row-factory-owned shape. Because the DB-API probe is synchronous, the ASGI adapter offloads the complete probe call rather than moving a live connection or cursor between threads; this preserves connection-factory/context-manager ownership while preventing a slow readiness dependency from monopolizing unrelated event-loop work. Those concerns remain separate evidence gates.

## Ownership boundary

This slice writes only Orgmetra. PostgreSQL is an Orgmetra-owned runtime dependency. Keyverse, Naruon, contextual-orchestrator, and all other dedicated-writer CWL repositories are not queried or mutated by the probe and are not claimed healthy by its result.
2 changes: 1 addition & 1 deletion manifest.json

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions services/people-api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ The service exposes a governed hire-to-employment read contract. `read_worker_pe

`PeopleAsgiApp` exposes that governed read use case as a dependency-light ASGI route: `GET /v1/tenants/{tenant_record_id}/people/{person_record_id}?effective_on=YYYY-MM-DD&purpose=people_read&fields=...`. It validates the exact route and query shape before authentication, accepts exactly one ASCII Bearer credential, delegates authentication and purpose-bound authorization to injected contracts, and never reads protected worker values after a denied authorization decision. Successful responses contain only authorized fields; all HTTP responses use `Cache-Control: no-store` and `Vary: Authorization`. Authentication, authorization, missing-record, integrity-conflict, and unexpected-backend failures are mapped to stable non-disclosing responses with a useful next action, and bearer tokens are never returned in response text.

`PeopleOperabilityAsgiApp` is a separate PII-free probe surface for deployment composition. `GET /health` reports process liveness and never calls PostgreSQL, while `GET /ready` invokes only the injected Orgmetra-owned readiness dependency and returns `503 not_ready` with a bounded operator action when that dependency fails. `PostgresReadinessProbe` implements the production owned-database check as a read-only transaction containing only `SELECT 1`; it does not set tenant context, read HR tables, accept credentials, or reach into Keyverse/Naruon/other dedicated-writer services. Deployment routing and network policy should expose these endpoints only where orchestration requires them. Metrics, Kubernetes manifests, startup-probe timing, and release certification remain separate release work.

The People API quality workflow is part of this contract and must run for pull requests to every supported protected/default integration branch, including `develop`. Its service tests enforce 100% owned statement and branch coverage and include regression coverage for the workflow dispatch boundary and HTTP security/transport behavior.

`HireAcceptanceAsgiApp` exposes confirmed-hire materialization as `POST /v1/tenants/{tenant_record_id}/candidate-worker-conversions?purpose=candidate_hire`. Authentication and tenant binding occur before request-body parsing, so an unauthenticated or foreign-tenant caller cannot use body parsing or command construction as an oracle. Authenticated requests then pass the validated `Idempotency-Key`, content-type, JSON/schema, authorization, and governed command checks under a 64 KiB cumulative request-body limit, at most 1024 ASGI request frames, and 128 nested JSON containers below the top-level command object. `PostgresHireAcceptancePort` acquires a transaction-scoped lock for the tenant/route/key before it persists Person, Employment, `candidate_worker_conversion_record`, governed audit/outbox evidence, and `people_mutation_idempotency_record` in one tenant-bound transaction. An exact retry returns the first committed person/employment/conversion identities without repeating necessary PII, audit, or outbox writes; reusing the key for changed command semantics fails closed. The legacy `candidate_worker_link` write path is not used.
Expand Down
10 changes: 9 additions & 1 deletion services/people-api/src/orgmetra_people_api/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Request-edge, governed read, confirmed-hire, and People mutation contracts."""
"""Request-edge, governed read, confirmed-hire, mutation, and operability contracts."""

from orgmetra_people_api.auth import (
AuthenticatedPrincipal,
Expand Down Expand Up @@ -32,6 +32,11 @@
create_employment_record,
create_position_record,
)
from orgmetra_people_api.operability import (
PeopleOperabilityAsgiApp,
PostgresReadinessProbe,
ReadinessProbe,
)
from orgmetra_people_api.people import (
AuthorizedWorkerPeopleView,
PeopleReadPort,
Expand Down Expand Up @@ -59,6 +64,7 @@
"PeopleMutationIntegrityError",
"PeopleMutationNotFound",
"PeopleMutationPort",
"PeopleOperabilityAsgiApp",
"PeopleReadPort",
"PeopleRecordIntegrityError",
"PeopleRecordNotFound",
Expand All @@ -67,6 +73,8 @@
"PostgresHireAcceptancePort",
"PostgresPeopleMutationPort",
"PostgresPeopleReadPort",
"PostgresReadinessProbe",
"ReadinessProbe",
"AssignmentMutationCommand",
"AssignmentMutationResult",
"EmploymentMutationCommand",
Expand Down
145 changes: 145 additions & 0 deletions services/people-api/src/orgmetra_people_api/operability.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
"""PII-free liveness and owned-dependency readiness surfaces for People API.

Liveness proves only that the People API process can serve HTTP. Readiness is
stricter: it calls an injected probe for dependencies owned by this service and
returns unavailable until those dependencies can serve work. Identity providers
and other dedicated-writer CWL services are intentionally outside this probe so
the People API never reaches into their private implementation boundaries.
"""

from __future__ import annotations

import asyncio
from contextlib import AbstractContextManager
from dataclasses import dataclass
import json
from typing import Any, Awaitable, Callable, Mapping, Protocol, runtime_checkable

AsgiReceive = Callable[[], Awaitable[dict[str, object]]]
AsgiSend = Callable[[dict[str, object]], Awaitable[None]]
PostgresConnectionFactory = Callable[[], AbstractContextManager[Any]]

_READ_ONLY_SQL = "SET TRANSACTION READ ONLY"
_READINESS_SQL = "SELECT 1"


@runtime_checkable
class ReadinessProbe(Protocol):
"""Check only dependencies whose availability is owned by the People API."""

def check_ready(self) -> None:
"""Return normally when owned dependencies are ready; otherwise raise."""


@dataclass(frozen=True, slots=True)
class PostgresReadinessProbe:
"""Verify the Orgmetra PostgreSQL dependency without reading HR business data.

Deployment code supplies the same kind of DB-API connection factory used by
the People persistence adapters. The probe enters a read-only transaction
and executes only ``SELECT 1``; it does not set tenant context or touch any
application table because readiness is infrastructure evidence, not an HR
data query.
"""

connection_factory: PostgresConnectionFactory

def __post_init__(self) -> None:
"""Reject an unusable connection factory before serving readiness traffic."""
if not callable(self.connection_factory):
raise TypeError("connection_factory must be callable")

def check_ready(self) -> None:
"""Raise when PostgreSQL cannot complete the reviewed read-only probe."""
with self.connection_factory() as connection:
with connection.cursor() as cursor:
cursor.execute(_READ_ONLY_SQL)
cursor.execute(_READINESS_SQL)
row = cursor.fetchone()
if row is None:
raise RuntimeError("owned PostgreSQL readiness query returned no result")


@dataclass(frozen=True, slots=True)
class PeopleOperabilityAsgiApp:
"""Expose dependency-light `/health` and `/ready` endpoints for orchestration.

``/health`` never invokes dependencies and therefore remains suitable for a
liveness probe that should not create restart loops during a database outage.
``/ready`` offloads the supplied synchronous owned-dependency probe from the
ASGI event loop and returns HTTP 503 on any dependency failure. Neither route
accepts credentials, reads HR data, returns dependency details, or claims
that foreign CWL services are healthy.
"""
Comment thread
seonghobae marked this conversation as resolved.

readiness_probe: ReadinessProbe

def __post_init__(self) -> None:
"""Require a concrete readiness contract before the app can be served."""
if not isinstance(self.readiness_probe, ReadinessProbe):
raise TypeError("readiness_probe must implement ReadinessProbe")

async def __call__(self, scope: Mapping[str, object], receive: AsgiReceive, send: AsgiSend) -> None:
"""Serve one PII-free operability request with bounded failure responses."""
del receive
if scope.get("type") != "http":
raise ValueError("PeopleOperabilityAsgiApp accepts only HTTP ASGI scopes")

if scope.get("method") != "GET":
await _send_json(
send,
status=405,
payload={
"error": "method_not_allowed",
"message": "Use GET for People API health and readiness probes.",
},
extra_headers=((b"allow", b"GET"),),
)
return
Comment thread
seonghobae marked this conversation as resolved.

path = scope.get("path")
if path == "/health":
await _send_json(send, status=200, payload={"status": "ok"})
return
if path == "/ready":
try:
await asyncio.to_thread(self.readiness_probe.check_ready)
except Exception: # noqa: BLE001 - readiness must normalize dependency details.
await _send_json(
send,
status=503,
payload={
"error": "not_ready",
"message": "Retry after an Orgmetra operator restores the owned People API dependency.",
},
)
return
await _send_json(send, status=200, payload={"status": "ready"})
return
Comment thread
seonghobae marked this conversation as resolved.

await _send_json(
send,
status=404,
payload={
"error": "route_not_found",
"message": "Use /health for liveness or /ready for owned-dependency readiness.",
},
)


async def _send_json(
send: AsgiSend,
*,
status: int,
payload: Mapping[str, object],
extra_headers: tuple[tuple[bytes, bytes], ...] = (),
) -> None:
"""Emit deterministic non-cacheable JSON without HR or dependency details."""
body = json.dumps(payload, ensure_ascii=True, separators=(",", ":"), sort_keys=True).encode("utf-8")
headers = (
(b"content-type", b"application/json"),
(b"cache-control", b"no-store"),
*extra_headers,
)
await send({"type": "http.response.start", "status": status, "headers": list(headers)})
await send({"type": "http.response.body", "body": body, "more_body": False})
65 changes: 65 additions & 0 deletions services/people-api/tests/test_operability_concurrency.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
"""Concurrency regressions for the People API operability surface."""

from __future__ import annotations

import asyncio
import threading
import unittest

from orgmetra_people_api.operability import PeopleOperabilityAsgiApp


class BlockingReadinessProbe:
"""Model synchronous DB-API work that needs the ASGI event loop to stay live."""

def __init__(self) -> None:
"""Create one release signal and observable probe outcome."""
self.calls = 0
self.release_signal = threading.Event()
self.released_during_check = False

def release(self) -> None:
"""Release the simulated dependency wait from the event-loop callback."""
self.release_signal.set()

def check_ready(self) -> None:
"""Wait briefly for loop progress and fail if the loop is monopolized."""
self.calls += 1
self.released_during_check = self.release_signal.wait(timeout=0.5)
if not self.released_during_check:
raise RuntimeError("ASGI event loop could not progress during readiness work")


class PeopleOperabilityConcurrencyTests(unittest.IsolatedAsyncioTestCase):
"""Keep synchronous owned-dependency work off the ASGI event loop."""

async def test_ready_keeps_event_loop_live_during_sync_probe(self) -> None:
"""A blocking readiness check must execute away from the ASGI event loop."""
probe = BlockingReadinessProbe()
app = PeopleOperabilityAsgiApp(probe)
loop = asyncio.get_running_loop()
messages: list[dict[str, object]] = []

async def receive() -> dict[str, object]:
"""Return an empty request frame; readiness never consumes a body."""
return {"type": "http.request", "body": b"", "more_body": False}

async def send(message: dict[str, object]) -> None:
"""Capture the readiness response."""
messages.append(message)

# The callback can release the simulated DB wait only when the ASGI loop
# remains schedulable while synchronous readiness work is in progress.
loop.call_soon(probe.release)
await app({"type": "http", "method": "GET", "path": "/ready"}, receive, send)

self.assertTrue(
probe.released_during_check,
"readiness must keep the ASGI event loop live during synchronous dependency work",
)
self.assertEqual(probe.calls, 1)
self.assertEqual(messages[0]["status"], 200)


if __name__ == "__main__":
unittest.main()
Loading
Loading