-
Notifications
You must be signed in to change notification settings - Fork 0
feat(operability): add People API health and readiness probes #74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
seonghobae
wants to merge
18
commits into
develop
Choose a base branch
from
feat/people-operability-probes
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
fdd66c4
test(people): define liveness and readiness contract
seonghobae 831de23
feat(people): add liveness and owned-db readiness probes
seonghobae 7b2e193
feat(people): export operability probe contracts
seonghobae 331989f
docs(people): explain probe failure domains
seonghobae 33d0033
docs(operability): trace People probe semantics
seonghobae 0f1b755
docs(operability): record primary probe references
seonghobae ccd736d
test(operability): require canonical probe runbook
seonghobae ec49aec
docs(operability): bind People probes to operator runbook
seonghobae dd862fe
fix(provenance): seal operability runbook evidence
seonghobae 7a86081
test(operability): accept mapping readiness rows
seonghobae f27420b
fix(operability): decouple readiness from cursor row shape
seonghobae 82a1f43
test(operability): make readiness result row-shape neutral
seonghobae 82c8ef9
test(operability): pin driver-independent readiness traceability
seonghobae 7847f3a
docs(operability): align readiness traceability with row factories
seonghobae 303133e
test(operability): expose readiness event-loop blocking
seonghobae ac7b947
fix(operability): offload readiness probe from event loop
seonghobae 471d558
test(operability): model blocking readiness concurrency
seonghobae 7d75e68
docs(operability): trace readiness concurrency boundary
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # People API operability references | ||
|
|
||
| ## Scope | ||
|
|
||
| These references support the active People API liveness/readiness design. They are design evidence only; they do not claim Kubernetes conformance, certification, production deployment, or release acceptance. | ||
|
|
||
| ## APA 7 references | ||
|
|
||
| Kubernetes Authors. (2026). *Liveness, readiness, and startup probes*. Kubernetes. https://kubernetes.io/docs/concepts/workloads/pods/probes/ | ||
|
|
||
| Kubernetes Authors. (2026). *Configure liveness, readiness and startup probes*. Kubernetes. https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ | ||
|
|
||
| ## Design use | ||
|
|
||
| The Kubernetes documentation distinguishes liveness, which can trigger container restart, from readiness, which removes an unavailable Pod from service traffic. It also cautions that incorrectly coupling liveness to transient dependencies can create cascading failures and describes the pattern where readiness additionally verifies required backend services. Orgmetra applies that distinction by keeping `/health` dependency-free and binding `/ready` only to the People API's owned PostgreSQL availability check. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # People API operability traceability | ||
|
|
||
| ## Status | ||
|
|
||
| This document describes the active PR that introduces executable People API probe semantics. Protected `develop@9e3e4847510e1e612b48474ba42b177b8ed824df` does not yet contain these routes, and this document does not claim production deployment or release readiness. | ||
|
|
||
| ## Requirement-to-evidence map | ||
|
|
||
| | Requirement | Active implementation | Executable evidence | Failure rule | | ||
| |---|---|---|---| | ||
| | Process liveness must not depend on PostgreSQL availability | `PeopleOperabilityAsgiApp` `GET /health` | `test_health_is_live_without_touching_owned_dependencies` | PostgreSQL failure cannot make liveness fail or trigger a dependency call. | | ||
| | Traffic readiness must reflect the service-owned PostgreSQL dependency | `PeopleOperabilityAsgiApp` `GET /ready` + `PostgresReadinessProbe` | `test_ready_checks_owned_dependency_and_reports_success`, `test_postgres_probe_requires_callable_factory_and_a_result_row`, `test_postgres_probe_accepts_mapping_row_factory` | Readiness is 200 only after the reviewed read-only probe produces a row regardless of DB-API row factory. | | ||
| | Synchronous readiness work must not monopolize the ASGI event loop | `PeopleOperabilityAsgiApp` offloads `ReadinessProbe.check_ready` with `asyncio.to_thread` | `test_ready_keeps_event_loop_live_during_sync_probe` | The connection factory, cursor, transaction, and `SELECT 1` execute together on the worker thread while the ASGI loop remains schedulable. | | ||
| | Backend failure details must not cross the HTTP boundary | bounded `503 not_ready` response | `test_ready_normalizes_dependency_failure_without_leaking_details` | Any owned-dependency exception becomes a stable 503 with a next operator action and no backend message. | | ||
| | Probe routes must not become a hidden HR-data or identity integration path | no tenant context, credentials, HR table SQL, or foreign-service calls | source contract plus route tests | `/health` and `/ready` expose status only; Keyverse and other dedicated-writer services remain outside this boundary. | | ||
| | Unsupported transport shapes must stay bounded | exact GET-only `/health` and `/ready` surface | `test_unknown_route_and_wrong_method_are_bounded_transport_errors`, `test_non_http_scope_is_rejected_as_a_programming_error` | Unknown route is 404, wrong method is 405 with `Allow: GET`, non-HTTP ASGI is rejected. | | ||
|
|
||
| ## Operational interpretation | ||
|
|
||
| Kubernetes distinguishes liveness from readiness: liveness is used to decide when a container should be restarted, while readiness determines whether it should receive traffic. The official Kubernetes probe guidance specifically warns that an incorrect liveness dependency can cause cascading restarts and notes that a strict backend dependency can be checked by readiness while liveness continues to reflect the application itself. Orgmetra therefore keeps PostgreSQL out of `/health` and checks it only in `/ready`. | ||
|
|
||
| The PostgreSQL readiness query deliberately proves infrastructure availability only. It does not prove tenant authorization, HR-data correctness, migration compatibility, downstream integrations, or release acceptance. The constant `SELECT 1` contract requires a result row but does not prescribe whether a DB-API driver represents that row as a tuple, mapping, or another row-factory-owned shape. Because the DB-API probe is synchronous, the ASGI adapter offloads the complete probe call rather than moving a live connection or cursor between threads; this preserves connection-factory/context-manager ownership while preventing a slow readiness dependency from monopolizing unrelated event-loop work. Those concerns remain separate evidence gates. | ||
|
|
||
| ## Ownership boundary | ||
|
|
||
| This slice writes only Orgmetra. PostgreSQL is an Orgmetra-owned runtime dependency. Keyverse, Naruon, contextual-orchestrator, and all other dedicated-writer CWL repositories are not queried or mutated by the probe and are not claimed healthy by its result. |
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
145 changes: 145 additions & 0 deletions
145
services/people-api/src/orgmetra_people_api/operability.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| """PII-free liveness and owned-dependency readiness surfaces for People API. | ||
|
|
||
| Liveness proves only that the People API process can serve HTTP. Readiness is | ||
| stricter: it calls an injected probe for dependencies owned by this service and | ||
| returns unavailable until those dependencies can serve work. Identity providers | ||
| and other dedicated-writer CWL services are intentionally outside this probe so | ||
| the People API never reaches into their private implementation boundaries. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import asyncio | ||
| from contextlib import AbstractContextManager | ||
| from dataclasses import dataclass | ||
| import json | ||
| from typing import Any, Awaitable, Callable, Mapping, Protocol, runtime_checkable | ||
|
|
||
| AsgiReceive = Callable[[], Awaitable[dict[str, object]]] | ||
| AsgiSend = Callable[[dict[str, object]], Awaitable[None]] | ||
| PostgresConnectionFactory = Callable[[], AbstractContextManager[Any]] | ||
|
|
||
| _READ_ONLY_SQL = "SET TRANSACTION READ ONLY" | ||
| _READINESS_SQL = "SELECT 1" | ||
|
|
||
|
|
||
| @runtime_checkable | ||
| class ReadinessProbe(Protocol): | ||
| """Check only dependencies whose availability is owned by the People API.""" | ||
|
|
||
| def check_ready(self) -> None: | ||
| """Return normally when owned dependencies are ready; otherwise raise.""" | ||
|
|
||
|
|
||
| @dataclass(frozen=True, slots=True) | ||
| class PostgresReadinessProbe: | ||
| """Verify the Orgmetra PostgreSQL dependency without reading HR business data. | ||
|
|
||
| Deployment code supplies the same kind of DB-API connection factory used by | ||
| the People persistence adapters. The probe enters a read-only transaction | ||
| and executes only ``SELECT 1``; it does not set tenant context or touch any | ||
| application table because readiness is infrastructure evidence, not an HR | ||
| data query. | ||
| """ | ||
|
|
||
| connection_factory: PostgresConnectionFactory | ||
|
|
||
| def __post_init__(self) -> None: | ||
| """Reject an unusable connection factory before serving readiness traffic.""" | ||
| if not callable(self.connection_factory): | ||
| raise TypeError("connection_factory must be callable") | ||
|
|
||
| def check_ready(self) -> None: | ||
| """Raise when PostgreSQL cannot complete the reviewed read-only probe.""" | ||
| with self.connection_factory() as connection: | ||
| with connection.cursor() as cursor: | ||
| cursor.execute(_READ_ONLY_SQL) | ||
| cursor.execute(_READINESS_SQL) | ||
| row = cursor.fetchone() | ||
| if row is None: | ||
| raise RuntimeError("owned PostgreSQL readiness query returned no result") | ||
|
|
||
|
|
||
| @dataclass(frozen=True, slots=True) | ||
| class PeopleOperabilityAsgiApp: | ||
| """Expose dependency-light `/health` and `/ready` endpoints for orchestration. | ||
|
|
||
| ``/health`` never invokes dependencies and therefore remains suitable for a | ||
| liveness probe that should not create restart loops during a database outage. | ||
| ``/ready`` offloads the supplied synchronous owned-dependency probe from the | ||
| ASGI event loop and returns HTTP 503 on any dependency failure. Neither route | ||
| accepts credentials, reads HR data, returns dependency details, or claims | ||
| that foreign CWL services are healthy. | ||
| """ | ||
|
|
||
| readiness_probe: ReadinessProbe | ||
|
|
||
| def __post_init__(self) -> None: | ||
| """Require a concrete readiness contract before the app can be served.""" | ||
| if not isinstance(self.readiness_probe, ReadinessProbe): | ||
| raise TypeError("readiness_probe must implement ReadinessProbe") | ||
|
|
||
| async def __call__(self, scope: Mapping[str, object], receive: AsgiReceive, send: AsgiSend) -> None: | ||
| """Serve one PII-free operability request with bounded failure responses.""" | ||
| del receive | ||
| if scope.get("type") != "http": | ||
| raise ValueError("PeopleOperabilityAsgiApp accepts only HTTP ASGI scopes") | ||
|
|
||
| if scope.get("method") != "GET": | ||
| await _send_json( | ||
| send, | ||
| status=405, | ||
| payload={ | ||
| "error": "method_not_allowed", | ||
| "message": "Use GET for People API health and readiness probes.", | ||
| }, | ||
| extra_headers=((b"allow", b"GET"),), | ||
| ) | ||
| return | ||
|
seonghobae marked this conversation as resolved.
|
||
|
|
||
| path = scope.get("path") | ||
| if path == "/health": | ||
| await _send_json(send, status=200, payload={"status": "ok"}) | ||
| return | ||
| if path == "/ready": | ||
| try: | ||
| await asyncio.to_thread(self.readiness_probe.check_ready) | ||
| except Exception: # noqa: BLE001 - readiness must normalize dependency details. | ||
| await _send_json( | ||
| send, | ||
| status=503, | ||
| payload={ | ||
| "error": "not_ready", | ||
| "message": "Retry after an Orgmetra operator restores the owned People API dependency.", | ||
| }, | ||
| ) | ||
| return | ||
| await _send_json(send, status=200, payload={"status": "ready"}) | ||
| return | ||
|
seonghobae marked this conversation as resolved.
|
||
|
|
||
| await _send_json( | ||
| send, | ||
| status=404, | ||
| payload={ | ||
| "error": "route_not_found", | ||
| "message": "Use /health for liveness or /ready for owned-dependency readiness.", | ||
| }, | ||
| ) | ||
|
|
||
|
|
||
| async def _send_json( | ||
| send: AsgiSend, | ||
| *, | ||
| status: int, | ||
| payload: Mapping[str, object], | ||
| extra_headers: tuple[tuple[bytes, bytes], ...] = (), | ||
| ) -> None: | ||
| """Emit deterministic non-cacheable JSON without HR or dependency details.""" | ||
| body = json.dumps(payload, ensure_ascii=True, separators=(",", ":"), sort_keys=True).encode("utf-8") | ||
| headers = ( | ||
| (b"content-type", b"application/json"), | ||
| (b"cache-control", b"no-store"), | ||
| *extra_headers, | ||
| ) | ||
| await send({"type": "http.response.start", "status": status, "headers": list(headers)}) | ||
| await send({"type": "http.response.body", "body": body, "more_body": False}) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| """Concurrency regressions for the People API operability surface.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import asyncio | ||
| import threading | ||
| import unittest | ||
|
|
||
| from orgmetra_people_api.operability import PeopleOperabilityAsgiApp | ||
|
|
||
|
|
||
| class BlockingReadinessProbe: | ||
| """Model synchronous DB-API work that needs the ASGI event loop to stay live.""" | ||
|
|
||
| def __init__(self) -> None: | ||
| """Create one release signal and observable probe outcome.""" | ||
| self.calls = 0 | ||
| self.release_signal = threading.Event() | ||
| self.released_during_check = False | ||
|
|
||
| def release(self) -> None: | ||
| """Release the simulated dependency wait from the event-loop callback.""" | ||
| self.release_signal.set() | ||
|
|
||
| def check_ready(self) -> None: | ||
| """Wait briefly for loop progress and fail if the loop is monopolized.""" | ||
| self.calls += 1 | ||
| self.released_during_check = self.release_signal.wait(timeout=0.5) | ||
| if not self.released_during_check: | ||
| raise RuntimeError("ASGI event loop could not progress during readiness work") | ||
|
|
||
|
|
||
| class PeopleOperabilityConcurrencyTests(unittest.IsolatedAsyncioTestCase): | ||
| """Keep synchronous owned-dependency work off the ASGI event loop.""" | ||
|
|
||
| async def test_ready_keeps_event_loop_live_during_sync_probe(self) -> None: | ||
| """A blocking readiness check must execute away from the ASGI event loop.""" | ||
| probe = BlockingReadinessProbe() | ||
| app = PeopleOperabilityAsgiApp(probe) | ||
| loop = asyncio.get_running_loop() | ||
| messages: list[dict[str, object]] = [] | ||
|
|
||
| async def receive() -> dict[str, object]: | ||
| """Return an empty request frame; readiness never consumes a body.""" | ||
| return {"type": "http.request", "body": b"", "more_body": False} | ||
|
|
||
| async def send(message: dict[str, object]) -> None: | ||
| """Capture the readiness response.""" | ||
| messages.append(message) | ||
|
|
||
| # The callback can release the simulated DB wait only when the ASGI loop | ||
| # remains schedulable while synchronous readiness work is in progress. | ||
| loop.call_soon(probe.release) | ||
| await app({"type": "http", "method": "GET", "path": "/ready"}, receive, send) | ||
|
|
||
| self.assertTrue( | ||
| probe.released_during_check, | ||
| "readiness must keep the ASGI event loop live during synchronous dependency work", | ||
| ) | ||
| self.assertEqual(probe.calls, 1) | ||
| self.assertEqual(messages[0]["status"], 200) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| unittest.main() |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.