docs(observability): describe current OTEL fleet contract

.gds/repository.yaml

@@ -24,12 +24,14 @@ classification:
 product:
   purpose: >-
     Open-source GARM and Incus control-plane components for isolated,
-    disposable GitHub Actions runners.
+    disposable GitHub Actions runners with durable OTEL-native observability.
   capabilities:
     - "Observation-only JobAssigned and durable complete-identity admission"
     - "One job per ephemeral Incus container"
     - "Fail-closed GitHub, provider and instance reconciliation"
     - "Bounded diagnostic WAL with verified remote confirmation"
+    - "OpenTelemetry collection, OTLP transport and OpenObserve asset contracts"
+    - "Bounded host-signal and compliance metrics without duplicate raw logs"
     - "Synthetic configuration examples with no estate data"
   entrypoints:
     - change: "Commands"

CHANGELOG.md

@@ -2,6 +2,20 @@
 
 ## Unreleased
 
+- Keep OpenTelemetry as the sole collection standard, OTLP/HTTP as the
+  transport, and OpenObserve as the only fleet telemetry backend. Classified
+  host warnings are cumulative metrics rather than duplicate raw log records.
+- Added nine managed dashboards and twenty-five managed alerts covering host
+  compliance, classified host signals, diagnostics, lifecycle, capacity,
+  provider reliability, fairness and telemetry durability.
+- Export package/reboot, running-kernel and SRSO state from all fleet hosts
+  through the existing observer and OTEL pipelines.
+- Treat queued and assigned authoritative rehydration gaps as raw diagnostic
+  context; persistent repository/workflow identity paging begins only after a
+  running intent exceeds its own state-entry grace.
+- Bound OpenObserve v0.92 notification silence to ten minutes for pages and
+  fifteen minutes for tickets because the backend pauses outcome evaluation
+  during silence. Sustained range-query hold windows remain unchanged.
 - Encode sustained PromQL holds with range subqueries and keep OpenObserve's
   trigger threshold at one series; v0.92 defines that threshold as coverage,
   not as a consecutive-evaluation counter.
@@ -11,8 +25,6 @@
   listing. Exact-instance operations retain full state, while deleting
   containers no longer trigger recursive cgroup and network stat reads.
 
-## Unreleased
-
 - Keep diagnostic export within one bounded run across transient private-route
   convergence using repeated bounded HEAD rounds, require sustained failure
   before paging, and include the route reconciler timer in platform health.

README.md

@@ -36,6 +36,14 @@ whole-pipeline performance standard implemented by this engine.
 - Host-local CPU, memory and I/O PSI is converted into a hysteretic, expiring
   Incus member signal; missing, stale or closed signals remove capacity rather
   than becoming provider failures or hidden overcommit.
+- OpenTelemetry is the only collection standard, OTLP/HTTP is the transport,
+  and OpenObserve is the only telemetry store. PromQL appears only as
+  OpenObserve's metric-query language; it does not imply a Prometheus server.
+  High-volume kernel/LVM notices are exported as bounded cumulative metrics,
+  while package, reboot, kernel and SRSO state remains visible for every host.
+- Raw missing correlation remains visible during queued and assigned capacity
+  phases. Persistent identity alerts begin only for a running intent after its
+  own convergence grace, so capacity delay is not mislabeled as identity loss.
 - Public repository CI runs only on standard GitHub-hosted runners.
 - Private Linux jobs can use the [trust-scoped package cache](docs/package-cache.md)
   over their existing one-job RustFS identity; compiler and package caches

docs/drakkars-product-contract.md

@@ -180,6 +180,18 @@ Signals share repository, commit, workflow, run, job, attempt, priority,
 technology, package manager, intent, reservation, worker, host, image and
 artifact identities where their trust boundary permits them.
 
+OpenTelemetry is the only collection standard and OTLP/HTTP is the transport.
+OpenObserve is the only durable telemetry backend; its PromQL support is a
+query language rather than a separate Prometheus deployment. Known high-volume
+host warnings are classified into bounded cumulative metrics and are not
+duplicated into the application-log stream. Host package/reboot state, running
+kernel identity and kernel-reported vulnerability status use the same path.
+
+Pre-execution identity is intentionally sparse. Queued and assigned intents
+retain raw missing/unbound counters, while persistent correlation alerts begin
+only after an intent reaches running and exceeds its state-entry convergence
+grace. Capacity delay remains owned by lifecycle and capacity signals.
+
 The system must explain queue delay, critical path, retry reason, constrained
 resource, cache benefit, missing technology, failure ownership and telemetry
 coverage. Raw output is bounded and redacted. A telemetry-store outage delays

docs/observability-dashboards.md

@@ -29,10 +29,13 @@ The reconciler owns only dashboards carrying the versioned `managed-by:gds`
 description marker. It preserves manually created dashboards, uses OpenObserve
 hash preconditions for updates, and requires a clean post-apply read-back.
 
-The six dashboards cover:
+The nine managed dashboards cover:
 
 - capacity and pressure;
 - correlation integrity;
+- diagnostic storage retention;
+- host package, reboot, kernel and SRSO compliance;
+- classified host signals without duplicate raw-log storage;
 - lifecycle phase latency;
 - provider reliability;
 - priority/class fairness;
@@ -44,6 +47,6 @@ human decision surfaces used by alert response and CD health verification; a
 private deployment must not silently edit them.
 
 Dashboards do not prove alerts deliver. `config/observability-rules.yaml` owns
-the separately tested alert contract, and a real backend lifecycle record must
+the twenty-five-rule separately tested alert contract, and a real backend lifecycle record must
 prove Collector buffering, recovered records and explicit loss counters before
 the observability product is accepted.

docs/runbooks/fleet-alerts.md

@@ -3,6 +3,11 @@
 Every fleet alert is symptom-oriented. Confirm the exact metric and correlated
 queue intent, GitHub job, provider lease and Incus instance before recovery.
 
+OpenTelemetry collects and transforms every fleet signal, OTLP/HTTP transports
+it, and OpenObserve stores, queries, dashboards and alerts it. PromQL in this
+contract is OpenObserve's query syntax; no Prometheus server, agent,
+remote-write path or second metric store is part of the product.
+
 `config/observability-rules.yaml` is the canonical rule source. Each rule names
 one real OpenObserve metric stream for API ownership and keeps its PromQL
 expression separate from the comparison operator and threshold. Render the
@@ -39,12 +44,21 @@ gha-fleet reconcile-openobserve-alerts \
   journals and diagnostic bundles, identify the oldest exact identity, and use
   only the bounded recovery operation that matches authoritative evidence.
 - GitHub correlation pages: inspect only the age-qualified metrics. Raw
-  unbound/missing counters describe normal pre-`JobAssigned` transitions and
-  are diagnostic context, not a recovery trigger.
+  unbound/missing counters describe normal queued and assigned capacity phases
+  and are diagnostic context, not a recovery trigger. Persistent identity
+  paging begins only after the intent is running and its state-entry grace has
+  elapsed.
 - Collector queue pages: preserve the queue directory, restore the private
   OpenObserve route/backend, and verify queue drain plus exact record recovery.
 - OOM or pressure pages: close admission; never stop an already running worker
   merely to make utilization look healthy.
+- Host-signal tickets: use `gha_fleet_host_signal_events` cumulative deltas.
+  LVM activation and overlay `xino=off` are workload-volume context; audit
+  suppression and workqueue-hog alerts act only on their bounded burst budget.
+  Raw host-signal logs are intentionally not duplicated into the fleet stream.
+- Compliance alerts: require complete coverage from every declared host before
+  trusting package, reboot, kernel or SRSO state. A kernel-reported hardware or
+  microcode boundary is not a software rollout failure.
 - Provider terminal-circuit pages: preserve the retry journal, prove the
   provider/config repair, stop GARM only with zero running leases, dry-run and
   apply the exact `recover-provider-retry` CAS operation, restart GARM, and
@@ -56,6 +70,11 @@ gha-fleet reconcile-openobserve-alerts \
 - Slow-burn tickets: inspect class/tenant percentiles and capacity evidence;
   do not page an operator for a trend without an immediate action.
 
+OpenObserve v0.92 pauses outcome evaluation during notification silence.
+Managed pages therefore use ten-minute silence and tickets use fifteen-minute
+silence; do not interpret an older `firing` outcome until that bounded window
+has elapsed. Silence never replaces the rule's sustained range-query hold.
+
 Recovery means the triggering expression is false for its configured hold
 window, health is green, and no orphan, missing, uncovered or telemetry-loss
 counter remains non-zero.