diff --git a/.gds/repository.yaml b/.gds/repository.yaml index 212c6c3..c3a3b85 100644 --- a/.gds/repository.yaml +++ b/.gds/repository.yaml @@ -24,12 +24,14 @@ classification: product: purpose: >- Open-source GARM and Incus control-plane components for isolated, - disposable GitHub Actions runners. + disposable GitHub Actions runners with durable OTEL-native observability. capabilities: - "Observation-only JobAssigned and durable complete-identity admission" - "One job per ephemeral Incus container" - "Fail-closed GitHub, provider and instance reconciliation" - "Bounded diagnostic WAL with verified remote confirmation" + - "OpenTelemetry collection, OTLP transport and OpenObserve asset contracts" + - "Bounded host-signal and compliance metrics without duplicate raw logs" - "Synthetic configuration examples with no estate data" entrypoints: - change: "Commands" diff --git a/CHANGELOG.md b/CHANGELOG.md index 22bc58a..3a16f3e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,20 @@ ## Unreleased +- Keep OpenTelemetry as the sole collection standard, OTLP/HTTP as the + transport, and OpenObserve as the only fleet telemetry backend. Classified + host warnings are cumulative metrics rather than duplicate raw log records. +- Added nine managed dashboards and twenty-five managed alerts covering host + compliance, classified host signals, diagnostics, lifecycle, capacity, + provider reliability, fairness and telemetry durability. +- Export package/reboot, running-kernel and SRSO state from all fleet hosts + through the existing observer and OTEL pipelines. +- Treat queued and assigned authoritative rehydration gaps as raw diagnostic + context; persistent repository/workflow identity paging begins only after a + running intent exceeds its own state-entry grace. +- Bound OpenObserve v0.92 notification silence to ten minutes for pages and + fifteen minutes for tickets because the backend pauses outcome evaluation + during silence. Sustained range-query hold windows remain unchanged. - Encode sustained PromQL holds with range subqueries and keep OpenObserve's trigger threshold at one series; v0.92 defines that threshold as coverage, not as a consecutive-evaluation counter. @@ -11,8 +25,6 @@ listing. Exact-instance operations retain full state, while deleting containers no longer trigger recursive cgroup and network stat reads. -## Unreleased - - Keep diagnostic export within one bounded run across transient private-route convergence using repeated bounded HEAD rounds, require sustained failure before paging, and include the route reconciler timer in platform health. diff --git a/README.md b/README.md index 2298d54..3cdb9f7 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,14 @@ whole-pipeline performance standard implemented by this engine. - Host-local CPU, memory and I/O PSI is converted into a hysteretic, expiring Incus member signal; missing, stale or closed signals remove capacity rather than becoming provider failures or hidden overcommit. +- OpenTelemetry is the only collection standard, OTLP/HTTP is the transport, + and OpenObserve is the only telemetry store. PromQL appears only as + OpenObserve's metric-query language; it does not imply a Prometheus server. + High-volume kernel/LVM notices are exported as bounded cumulative metrics, + while package, reboot, kernel and SRSO state remains visible for every host. +- Raw missing correlation remains visible during queued and assigned capacity + phases. Persistent identity alerts begin only for a running intent after its + own convergence grace, so capacity delay is not mislabeled as identity loss. - Public repository CI runs only on standard GitHub-hosted runners. - Private Linux jobs can use the [trust-scoped package cache](docs/package-cache.md) over their existing one-job RustFS identity; compiler and package caches diff --git a/docs/drakkars-product-contract.md b/docs/drakkars-product-contract.md index e5bc569..2e729af 100644 --- a/docs/drakkars-product-contract.md +++ b/docs/drakkars-product-contract.md @@ -180,6 +180,18 @@ Signals share repository, commit, workflow, run, job, attempt, priority, technology, package manager, intent, reservation, worker, host, image and artifact identities where their trust boundary permits them. +OpenTelemetry is the only collection standard and OTLP/HTTP is the transport. +OpenObserve is the only durable telemetry backend; its PromQL support is a +query language rather than a separate Prometheus deployment. Known high-volume +host warnings are classified into bounded cumulative metrics and are not +duplicated into the application-log stream. Host package/reboot state, running +kernel identity and kernel-reported vulnerability status use the same path. + +Pre-execution identity is intentionally sparse. Queued and assigned intents +retain raw missing/unbound counters, while persistent correlation alerts begin +only after an intent reaches running and exceeds its state-entry convergence +grace. Capacity delay remains owned by lifecycle and capacity signals. + The system must explain queue delay, critical path, retry reason, constrained resource, cache benefit, missing technology, failure ownership and telemetry coverage. Raw output is bounded and redacted. A telemetry-store outage delays diff --git a/docs/observability-dashboards.md b/docs/observability-dashboards.md index 796b041..0c6f076 100644 --- a/docs/observability-dashboards.md +++ b/docs/observability-dashboards.md @@ -29,10 +29,13 @@ The reconciler owns only dashboards carrying the versioned `managed-by:gds` description marker. It preserves manually created dashboards, uses OpenObserve hash preconditions for updates, and requires a clean post-apply read-back. -The six dashboards cover: +The nine managed dashboards cover: - capacity and pressure; - correlation integrity; +- diagnostic storage retention; +- host package, reboot, kernel and SRSO compliance; +- classified host signals without duplicate raw-log storage; - lifecycle phase latency; - provider reliability; - priority/class fairness; @@ -44,6 +47,6 @@ human decision surfaces used by alert response and CD health verification; a private deployment must not silently edit them. Dashboards do not prove alerts deliver. `config/observability-rules.yaml` owns -the separately tested alert contract, and a real backend lifecycle record must +the twenty-five-rule separately tested alert contract, and a real backend lifecycle record must prove Collector buffering, recovered records and explicit loss counters before the observability product is accepted. diff --git a/docs/runbooks/fleet-alerts.md b/docs/runbooks/fleet-alerts.md index 7d1b056..0826552 100644 --- a/docs/runbooks/fleet-alerts.md +++ b/docs/runbooks/fleet-alerts.md @@ -3,6 +3,11 @@ Every fleet alert is symptom-oriented. Confirm the exact metric and correlated queue intent, GitHub job, provider lease and Incus instance before recovery. +OpenTelemetry collects and transforms every fleet signal, OTLP/HTTP transports +it, and OpenObserve stores, queries, dashboards and alerts it. PromQL in this +contract is OpenObserve's query syntax; no Prometheus server, agent, +remote-write path or second metric store is part of the product. + `config/observability-rules.yaml` is the canonical rule source. Each rule names one real OpenObserve metric stream for API ownership and keeps its PromQL expression separate from the comparison operator and threshold. Render the @@ -39,12 +44,21 @@ gha-fleet reconcile-openobserve-alerts \ journals and diagnostic bundles, identify the oldest exact identity, and use only the bounded recovery operation that matches authoritative evidence. - GitHub correlation pages: inspect only the age-qualified metrics. Raw - unbound/missing counters describe normal pre-`JobAssigned` transitions and - are diagnostic context, not a recovery trigger. + unbound/missing counters describe normal queued and assigned capacity phases + and are diagnostic context, not a recovery trigger. Persistent identity + paging begins only after the intent is running and its state-entry grace has + elapsed. - Collector queue pages: preserve the queue directory, restore the private OpenObserve route/backend, and verify queue drain plus exact record recovery. - OOM or pressure pages: close admission; never stop an already running worker merely to make utilization look healthy. +- Host-signal tickets: use `gha_fleet_host_signal_events` cumulative deltas. + LVM activation and overlay `xino=off` are workload-volume context; audit + suppression and workqueue-hog alerts act only on their bounded burst budget. + Raw host-signal logs are intentionally not duplicated into the fleet stream. +- Compliance alerts: require complete coverage from every declared host before + trusting package, reboot, kernel or SRSO state. A kernel-reported hardware or + microcode boundary is not a software rollout failure. - Provider terminal-circuit pages: preserve the retry journal, prove the provider/config repair, stop GARM only with zero running leases, dry-run and apply the exact `recover-provider-retry` CAS operation, restart GARM, and @@ -56,6 +70,11 @@ gha-fleet reconcile-openobserve-alerts \ - Slow-burn tickets: inspect class/tenant percentiles and capacity evidence; do not page an operator for a trend without an immediate action. +OpenObserve v0.92 pauses outcome evaluation during notification silence. +Managed pages therefore use ten-minute silence and tickets use fifteen-minute +silence; do not interpret an older `firing` outcome until that bounded window +has elapsed. Silence never replaces the rule's sustained range-query hold. + Recovery means the triggering expression is false for its configured hold window, health is green, and no orphan, missing, uncovered or telemetry-loss counter remains non-zero.