From 64db362394ed745abf683b5eb84d13084220e42f Mon Sep 17 00:00:00 2001 From: rldyourmnd Date: Wed, 26 Aug 2026 22:58:59 +0500 Subject: [PATCH] docs(skills): teach Drakkars lifecycle recovery --- .../gds-drakkars-audit-fleet/SKILL.md | 22 ++++++++++++++++++- .../gds-drakkars-audit-workflow/SKILL.md | 8 ++++++- skills/canonical/gds-drakkars-orient/SKILL.md | 4 +++- .../gds-drakkars-triage-run/SKILL.md | 22 ++++++++++++++++++- 4 files changed, 52 insertions(+), 4 deletions(-) diff --git a/skills/canonical/gds-drakkars-audit-fleet/SKILL.md b/skills/canonical/gds-drakkars-audit-fleet/SKILL.md index 94aa988..3a55e1f 100644 --- a/skills/canonical/gds-drakkars-audit-fleet/SKILL.md +++ b/skills/canonical/gds-drakkars-audit-fleet/SKILL.md @@ -28,6 +28,10 @@ Explain fleet health and bottlenecks from correlated, time-bounded evidence. 1. Resolve GDS context and the private estate source without copying its facts into public outputs. 2. Establish the audit window and inventory expected pools, capacity, priority, and technologies. 3. Correlate GitHub queue/start/end events with scheduler, provider, host, runner, and telemetry records using stable run, job, intent, and instance identifiers. + Before attributing queued jobs to fleet capacity, check GitHub Actions + service status and prove whether the exact `workflow_job` reached the + control-plane database. A GitHub job with no corresponding row is inbound + event-delivery delay, not a missing runner or exhausted pool. 4. Measure end-to-end latency, queue time, provisioning, setup, execution, teardown, utilization, failure, retry, and orphan rates by pool and priority. Under contention, verify that one repository uses no more than 75 percent of slot, measured CPU, and measured memory capacity; without a competing @@ -48,6 +52,20 @@ Explain fleet health and bottlenecks from correlated, time-bounded evidence. provider spans expose `incus_member`. Do not use the provider process resource host as the compute placement member. 8. Separate confirmed faults, saturation, waste, and `NOT_PROVEN` gaps. +9. Inspect durable lifecycle recovery rather than process health alone: + terminal job tombstones, overdue non-terminal provider retries, + assigned intents with an exact workflow-job row but no instance, scheduler + recovery startup grace/cooldown/active attempt, and vanished-runner recovery + transactions. An incomplete bounded attempt is evidence of partial progress, + not success and not permission for an immediate duplicate restart. + +## Runner scale sets + +For Runner Scale Set V2, `runs-on` can target `RunnerScaleSetName`. An empty +classic-label array from the self-hosted runner REST endpoint does not by itself +prove an unserviceable runner. Correlate the requested target with the runner's +scale-set id/name and the scale-set listener before diagnosing label drift or +proposing classic-label mutation. ## Safety @@ -63,7 +81,9 @@ Stop before restart, retry, cancellation, deployment, resize, or configuration w ## Verification -Cross-check GitHub, runtime journals, provider inventory, hosts, and observability freshness; mark gaps `NOT_PROVEN`. +Cross-check GitHub service status, exact workflow-job delivery, runtime +journals, recovery state, provider inventory, runner scale-set identity, hosts, +and observability freshness; mark gaps `NOT_PROVEN`. ## References diff --git a/skills/canonical/gds-drakkars-audit-workflow/SKILL.md b/skills/canonical/gds-drakkars-audit-workflow/SKILL.md index 0090f09..7acfb63 100644 --- a/skills/canonical/gds-drakkars-audit-workflow/SKILL.md +++ b/skills/canonical/gds-drakkars-audit-workflow/SKILL.md @@ -31,6 +31,10 @@ Audit one project's CI/CD behavior without reducing verification scope or mutati use the declared private fleet, including private repositories in free organizations; keep macOS and Windows hosted unless private capacity is explicitly declared. + For estate-wide claims, inspect active default-branch workflow content rather + than search hits from archived branches or historical evidence. Separate + blocking hosted/pin drift from an explicitly tracked repository-authority or + product-plan gap. 4. Build the dependency graph and critical path. Find accidental serialization, duplicated setup and security placements, oversized matrices, unsafe cache keys, and concurrency groups that can discard queued or running evidence. @@ -47,7 +51,9 @@ Stop before editing YAML, changing rulesets, dispatching workflows, or changing ## Verification -Prove each finding from workflow source, manifests, policy, and required-check state. +Prove each finding from default-branch workflow source, manifests, policy, +required-check state and, when scope spans private repositories, a complete +active-repository inventory. ## References diff --git a/skills/canonical/gds-drakkars-orient/SKILL.md b/skills/canonical/gds-drakkars-orient/SKILL.md index b47fa35..091da2c 100644 --- a/skills/canonical/gds-drakkars-orient/SKILL.md +++ b/skills/canonical/gds-drakkars-orient/SKILL.md @@ -30,7 +30,9 @@ Resolve the current CI/fleet context and its authority boundaries without mutati 3. Treat portable engines, reusable workflows, schemas, and generic skills as public product concerns. 4. Treat organizations, repository identities, priorities, hosts, networks, credentials, and runtime evidence as private estate concerns. 5. Mark unfetched provider or telemetry facts `NOT_PROVEN`. -6. Recommend the narrowest next skill and Git boundary. +6. Treat Runner Scale Set V2 names as routing targets. Do not equate an empty + classic self-hosted-label response with a missing scale-set name or listener. +7. Recommend the narrowest next skill and Git boundary. ## Stop conditions diff --git a/skills/canonical/gds-drakkars-triage-run/SKILL.md b/skills/canonical/gds-drakkars-triage-run/SKILL.md index 02fcb11..ab1f307 100644 --- a/skills/canonical/gds-drakkars-triage-run/SKILL.md +++ b/skills/canonical/gds-drakkars-triage-run/SKILL.md @@ -33,11 +33,29 @@ Trace one run from event receipt to terminal result without changing state. `github_workflow_ref`, `github_commit_sha`, `runner_name`, `instance_name`). Join traces by `runner_name`: `queue.*` supplies `queue_job_uuid`, and provider create/delete supplies `incus_member`. + If GitHub shows the job but the exact numeric job id is absent from the GARM + workflow-job store, stop the fleet trace at inbound delivery and check the + current GitHub Actions incident/queue state. Do not infer capacity or runner + label failure from a job the fleet never received. 3. Locate the first abnormal stage; distinguish queue pressure, missing capability, infrastructure failure, deterministic test failure, flaky failure, timeout, and observability loss. 4. Compare with adjacent successful runs using the same workflow and toolchain. 5. If a transient infrastructure error is proven, recommend a bounded rerun; do not execute it here. 6. Preserve running jobs and label unsupported claims `NOT_PROVEN`. +## Recovery state + +- For a delayed `JobAssigned`, check terminal tombstones before treating a + completed job as live demand. +- For an assigned job without an instance, inspect the scheduler recovery + attempt, startup grace and cooldown; healthy sibling progress is not proof + that the exact identity advanced. +- For an `in_progress` job whose runner id disappeared, inspect the durable + vanished-runner transaction and authoritative `run_attempt`. A force-cancel + followed by one full rerun is one recovery lifecycle, not two independent + mutations. +- Distinguish a failed bounded recovery with progressed identities from a + complete recovery and from a restart storm. + ## Direct-JIT identity Do not require `github_runner_request_id` from a running direct-JIT job. It is @@ -60,4 +78,6 @@ Use stable identifiers and distinguish observation, inference, and `NOT_PROVEN`. ## References -Use current GitHub job events, queue intents, provider leases, runner logs, and traces. +Use current GitHub service status and job events, queue intents, terminal +tombstones, scheduler/vanished recovery state, provider leases, runner logs, +and traces.