[skills] Add aws-eks-operations-review skill - #77
Open
shyamkulkarni wants to merge 7 commits into
Open
Conversation
Add a DevOps Agent skill that grades an Amazon EKS cluster against best practices across nine pillars — Operations, Resilience, Security, Scalability, Performance, Observability, Networking, Cost, Control Plane — plus AWS API and Cluster Insights rows, returning evidence-backed PASS/FAIL/N/A scorecards with prioritized remediations. SKILL.md drives an S0-S8 state machine and loads references just-in-time to bound context. The references tree carries the runtime contracts (check manifest, discovery manifest, gates, guards, thresholds, QA checklist, report contract), nine pillar definitions, 52 remediation shards, control-plane Logs Insights query shards, and three decision trees. Only eval definitions and fixtures are included. Generated eval run output under evals/functional, evals/best-practices, and evals/structure is reproducible via evals/run-functional-serial.py, which prunes those same paths when building a workspace, so committing it would add ~8 MB of derived artifacts. The eval helper scripts are excluded by the skills/.gitignore extension allowlist, which permits only the file types DevOps Agent skill uploads accept.
CONTRIBUTING requires each skill's frontmatter to carry a metadata block with author and version, and each skill README to state a non-production disclaimer. Add both to aws-eks-operations-review. Version 1.9.3 matches the top entry in the skill's CHANGELOG.
Add a custom agent that orchestrates the aws-eks-operations-review skill and publishes the review as multiple artifacts — one Summary plus one per graded pillar — rather than a single cumulative artifact. A full review spans 49 discovery areas and roughly 288 graded rows. Assembling that into one artifact means many sequential render calls whose payload grows each time, which is what produces render stalls and half-written reports. Splitting by pillar keeps most artifacts to one or two calls, makes each finished pillar a standalone deliverable, and removes cross-pillar accumulation risk. The system prompt carries the artifact split design, per-artifact render chunking with a character ceiling, a monotonicity guard that treats lost element accumulation as a failed call regardless of reported success, a runtime budget with a degradation ladder that declares skipped scope rather than omitting it silently, the Remediation Depth Contract defining full and compact finding forms, evidence rules requiring an observed result for every verdict, the QA coverage gate, and the artifact element schemas. The README documents the skill, the 21 tools, and the three memory stores to assign, and notes that tools and memory stores are Chat-only. It also covers third-party MCP servers for environments whose telemetry lives outside CloudWatch, including the prompt change required alongside assigning those tools.
A render call was rejected with "Invalid \escape" because remediation prose escaped JMESPath backtick literals as \` — a backslash before a backtick is not a valid JSON escape, so the payload failed to parse and nothing was written. The prompt specified what findings must contain but never how to encode them. The Depth Contract asks for concrete CLI snippets in remediation steps, which is exactly where backticks, quotes, and nested JSON accumulate, so this failure mode was likely rather than incidental. Add an encoding section to Artifacts: only JSON's own escapes are valid, backticks are never escaped, and JMESPath single-quoted raw strings are preferred over backtick literals. Require a parse self-check before each call. Also classify an encoding failure separately from a stall in 4e. Batch halving, finding demotion, and sibling artifacts all address payload size and cannot fix invalid escaping — the correct response is to fix the escape and resend the same content unchanged.
Without an EKS access entry for the Agent Space IAM role, use_kubectl cannot reach the cluster: all 49 discovery areas return n/a with a permission error and the review completes almost entirely unassessed. Neither README covered this, so the most likely first-run failure had no documented cause. Add the setup steps and a link to the DevOps Agent user guide to both READMEs, and state the policy choice this review needs. The documented default, AmazonAIOpsAssistantPolicy, suits incident investigation, but a full review walks the whole cluster object graph — RBAC, admission webhooks, CRDs, StorageClasses, PDBs, NetworkPolicies, quotas, ServiceAccounts — and any object kind the policy misses is recorded N/A for lack of access rather than graded. Recommend AmazonEKSAdminViewPolicy with access scope Cluster, and note that namespace-scoped access hides cluster-scoped objects. Also flag that AmazonEKSAdminViewPolicy grants read access to all objects including Secrets. The skill never fetches Secret values and grades them on existence and metadata only, so the grant is broader than the agent uses and should be an explicit decision, with AmazonAIOpsAssistantPolicy as the narrower fallback.
Drop the sample-code disclaimer from both the skill and custom agent READMEs. For the custom agent this matches convention: no other custom agent in the repo carries a disclaimer. For the skill it diverges from CONTRIBUTING item 7, which asks for a non-production note in a skill's README, and from the 14 of 22 existing skills that carry one.
CONTRIBUTING item 7 requires a non-production note in a skill's README, so restore it. The custom agent README stays without one, matching the other custom agents in the repo, none of which carry a disclaimer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
skills/aws-eks-operations-review, a read-only DevOps Agent skill that grades an Amazon EKS cluster against best practices and returns evidence-backed scorecards with prioritized remediations.Coverage is nine pillars — Operations, Resilience, Security, Scalability, Performance, Observability, Networking, Cost, Control Plane — plus AWS API and Cluster Insights rows. Every row is graded PASS / FAIL / N/A against an observed result; a row that cannot be assessed stays in the report as N/A with the exact reason rather than being dropped or guessed.
SKILL.mddrives an S0–S8 state machine and loads references just-in-time to bound context. Remediation shards load only after a FAIL verdict exists, and decision trees only after their matching signal.Contents
references/runtime/— discovery manifest, inventory schema, router, cluster gates, grading guards, check manifest, telemetry thresholds, common-check crosswalk, QA gate, report contractreferences/pillars/— nine canonical pillar definitionsreferences/remediations/— FAIL-only dispatch index plus 51 shardsreferences/control-plane-health/— metric sources, four Logs Insights query shards, thresholds, playbooksreferences/decision-trees/— pending pods, OOMKilled, API latency / 429references/docs/— operator and background material, explicitly not runtime authoritySafety posture
All access is read-only.
use_kubectlis limited toget,describe,logs,version,config current-context,cluster-info,top, andget --raw. Kubernetes Secret values are never fetched. Remediations are proposals for human approval, never mutations. Customer-account reads go through audited read-only agent access rather than local CLI or boto3 credentials.Not included
Generated eval run output under
evals/functional,evals/best-practices, andevals/structureis omitted. It is derived output —evals/run-functional-serial.pywrites into those paths and prunes the same three when building a workspace — so committing it would add roughly 8 MB of reproducible artifacts. Eval definitions and fixtures (evals.json,eval_queries.json,evals/files/) are included.Type of change
Testing
Validated read-only against a live EKS cluster (
retail-store-demo, us-east-1) in a non-production account, producing a full review across all nine pillars plus AWS API and Cluster Insights rows, with the QA coverage gate passing before the report was rendered.Structure and consistency suites live in
evals/and are documented inevals/TESTING.md. Per that file, model-matrix and live-validation results should not be claimed until the suites are actually re-run, so this PR states only the live review above.Repo conventions verified before submitting:
SKILL.mdfrontmatter parses as YAML and carries the requiredmetadatablock withauthorandversion; the skill README leads with the non-production disclaimer; the skill directory hasSKILL.md,README.md, andCHANGELOG.md. All internal reference links resolve, and no reference file is orphaned.License confirmation