Add aws-backup-coverage-review skill - #71
Open
vediyappan-kk wants to merge 13 commits into
Open
Conversation
Adds a read-only skill that reports AWS Backup coverage and posture across all enabled Regions of an account. AWS Backup Audit Manager's coverage control requires AWS Config resource recording plus a framework and a report plan that has already run, so in accounts without that setup an operator cannot ask what is not being backed up. This skill computes the answer live from read-only APIs and uses AWS Config only as an optimization. Coverage is modelled as five states rather than a boolean: Protected, Stale, SelectedNotProtected, Unprotected, and OptInBlocked. The last three all render as healthy in the AWS Backup console, which is what makes them worth a skill. 21 fixed numbered checks across 5 dimensions, with a mandatory 21-row Check Coverage Matrix so no check can be silently dropped. Thresholds match the Backup Audit Manager control defaults so output is comparable with Audit Manager. Permission gaps are excluded from the coverage denominator and cap the rating rather than being scored as coverage gaps. Read-only throughout, with an explicit API allowlist and hard denials on any Put/Delete/Create/Update/Start operation. CloudTrail is deliberately unused. Also adds the llms.txt entry and an EnableAwsBackupCoverageReview parameter, condition, read-only inline policy, and SkillPolicySummary line in cloudformation/devops-agent-skill-policies.yaml.
Addresses TFC domain review feedback on observability. Adds two checks to the Coverage integrity dimension, taking the total to 23: - 5.4 verifies an Audit Manager report plan is scheduled in each Region that has backup activity. Report plans are per Region, so a single plan can look like account-wide reporting when it is not. - 5.5 verifies an Audit Manager framework is configured where protected resources exist. A report plan alone reports job activity without evaluating control compliance, so the two are complementary. The finding notes the AWS Config dependency where the recorder is inactive. Both consume ListReportPlans and ListFrameworks, which the data collection phase already gathered but no check previously used. Checks 5.1 to 5.3 ask whether protection is real; 5.4 and 5.5 ask whether a decline in it would be noticed. Coverage is a point-in-time state, and without scheduled reporting or evaluated controls a regression surfaces only when someone next runs a review by hand. Also updates the check count across all assertions, adds remediation text and three AWS documentation references, and corrects the check-inventory eval case which asserted the previous count.
Re-ran Agent Skill Eval against the current content so the committed evidence matches the code. Root passed: true, Overall Grade A (0.91) — audit 98/100, trigger 1.00, functional outcome and style 1.00, PARETO_BETTER (quality +0.51, cost -62.5%). Also broadens one assertion in the membership-vs-protection case. It previously accepted only the literal strings "not protection" / "is not protection" / "not the same", and failed a correct answer that said "membership alone is never protection". The assertion still requires the negation, it just no longer depends on one phrasing of it.
Live testing showed the report being replaced by a conversational summary in roughly half of runs, correlating with the host delegating the account sweep to a research subagent — which does not reliably load the skill's reference files. Instruction inside the skill did not fix this across four attempts. Follows the pattern every report-producing skill in this repository already uses: report output is specified in a paired custom agent's system prompt rather than in the skill alone. Adds custom-agents/aws-backup-coverage-review/ with the standard Goal/Approach/Constraints/Output structure, restating the report structure and the 23-row check matrix so the artifact is produced even when references/ is not loaded. Also adds a remediation boundary to the skill. In testing, after the review the agent proposed and then attempted DeleteBackupSelection and CreateBackupSelection; only IAM refused it. The skill documented hard denials on those calls but said nothing about what to do when a user asks for a fix. It now returns the exact change and resource identifiers for a human to apply, and declines to attempt the call rather than relying on IAM to stop it. Reinforces per-Region enumeration completeness after a run reported "I did not get to check FSx, Redshift, Timestream, Storage Gateway, or EKS" — a type that was never queried is indistinguishable in the report from a type with no resources, and the second reads as full coverage.
Tools are now selectable directly on the Create agent form, so the separate Edit -> Chat step to add use_aws is no longer needed. Keeps the chat flow as a documented fallback for Agent Spaces on an older release, since the form there has no Tools selector.
A live run reported 33 CloudFormation stacks against 56 actual. It probed the 18 Regions with no backup activity using ListBackupPlans and ListProtectedResources only, found them empty, and stopped — missing 22 stacks spread across those Regions by StackSets and CDK bootstrap. AWS Backup APIs returning nothing means AWS Backup is not configured there, which is the finding rather than a reason to stop looking. A Region with no backup plans and 40 unprotected resources is the case this review exists to surface. A Region may now be dropped only after the Phase 3 inventory calls have run and returned zero for every type, and the Scope table must state how each Region was established as empty. Calls out the bulk types this trips on, since StackSet instances, CDK bootstrap stacks, and replication buckets are commonly spread across every enabled Region regardless of where backups are configured.
…inferred A live run reported S3 as opted in for us-east-1 and us-west-2 and opted out for us-east-2. The actual ResourceTypeOptInPreference is the exact inverse: S3 is false in us-east-1 and us-west-2 and true in us-east-2. The same run inverted CloudFormation for us-east-2, and presented the change as a correction to an earlier run that had been right. This is a worse class of error than a miscount: it sends the operator to change opt-in in the wrong Region, and it carried false confidence because it was framed as a fix. The likely cause is inferring opt-in from the absence of a matching selection rather than reading the boolean. Check 1.1 now requires the literal boolean from DescribeRegionSettings.ResourceTypeOptInPreference for the specific Region, quoted in the observed column, and explicitly forbids inferring it from a missing selection, from AdvancedBackupSettings, or from resources being unprotected. Where the boolean cannot be quoted the Region is marked Unconfirmed rather than asserting a direction, and a re-run may not contradict an earlier value without citing the response that justifies it.
Evaluated with the internal DevOps Agent skill-eval tool. Structure passed 12/12, but best-practices failed BP-12 (templates and outputs must not be in the body) in all three iterations and BP-03 (detailed reference materials must not be in the body) in two, scoring 79 to 86 percent. Both findings pointed at the same thing. The report skeleton was inlined into SKILL.md earlier as insurance against references/ failing to load, but the paired custom agent now carries the report structure and that is what actually made the report render reliably, so the skeleton was redundant. Removing it also brings SKILL.md from 5,364 to 4,837 tokens, under the 5,000 guidance, which clears the last finding from the public eval tool as well. The error-handling table moves to references/data-collection.md, next to the API allowlist whose failures it classifies. The Known API Quirks table stays in the body deliberately: it prevents a silent failure where reading the wrong ListBackupSelections response key makes every resource look unprotected, and a footgun in a reference file that may not load is the wrong trade. Recorded as a reasoned deviation rather than claiming full compliance. Best-practices now passes 2 of 3 iterations at 100, 100 and 93 percent with 94 percent consistency. The single remaining failure is judge variance on BP-16, which cites the 10-step Execution Flow as evidence against a step-by-step guidance check that the other two iterations pass. Also corrects the coverage model wording from five states to six, which had not been updated when OrphanedRecoveryPoint was added.
… README The skill README already states that each Agent Space has its own IAM role and that the policy must be applied per space, but the custom agent README did not. That is where it matters more, since creating a custom agent usually means creating a space, and a new space starts with none of the skill's added permissions even when another space in the same account already has them. Adds the requirement, the consequence (GetSupportedResourceTypes and the Storage Gateway and DSQL inventory calls return AccessDenied and appear as unverifiable rather than as findings), and the list-associations command that resolves a space's role.
A live run reported Storage Gateway as unenumerable because storagegateway:ListVolumes was unavailable, and excluded the type from the coverage denominator. The test account has two active gateways, but both are FILE_S3 file gateways, which expose file shares rather than volumes. AWS Backup's Storage Gateway resource type covers volume gateway volumes; file gateway data lives in the backing S3 bucket or FSx file system and is already assessed under the S3 or FSx type. So the correct answer for that account was zero eligible Storage Gateway resources, not a blind spot. The skill now enumerates gateways first and only calls ListVolumes for Regions that contain a volume gateway. An unavailable ListVolumes is a genuine ToolingFailure only when a volume gateway exists; where only file gateways are present it notes that their data is assessed under S3 or FSx and names the gateways, rather than reporting a gap. Reporting a blind spot that isn't one overstates the unknown, which for a coverage review is as misleading as understating it. Adds storagegateway:ListGateways, which AIDevOpsAgentAccessPolicy does not grant, to the skill's inline policy in the CloudFormation template and to the README's required action list — the delta is now six actions rather than five.
…nenumerable Every report listed VirtualMachine under the Inventory Completeness Notice as requiring an AWS Backup gateway and a registered hypervisor. That is true of the type but wrong as a conclusion: backup-gateway:ListHypervisors answers whether any hypervisor is registered, the baseline managed policy already allows it, and in the test account it returns zero in every Region. So VirtualMachine has zero eligible resources, and reporting it as a permanent blind spot overstates the unknown — for a coverage review that misleads as much as understating a gap, and it would have done so for every account with no VMware estate. The type is now enumerated: zero hypervisors means zero resources recorded in the by-type table, a registered hypervisor means enumerate with ListVirtualMachines, and only a failure of ListHypervisors itself makes the type genuinely unverifiable. SAP HANA on Amazon EC2 stays NotEnumerated by design, since proving absence needs SSM inventory of the Backint agent, and the reason is stated rather than implying the type was checked. The report notice now also says that a type which was queried and returned nothing belongs in the by-type table with zero eligible resources rather than in the notice. Adds backup-gateway:ListHypervisors and ListVirtualMachines to the API allowlist and the README action list. Both are covered by AIDevOpsAgentAccessPolicy, so the CloudFormation policy is unchanged and the delta remains six actions.
Author
|
Since the last update: added a paired custom agent (report output now renders reliably), moved the report skeleton and error-handling table into references/ per skill-eval BP-12/BP-03, and fixed Storage Gateway and VirtualMachine enumeration so neither is reported as a blind spot when it isn't. Internal skill-eval: structure 12/12, best-practices 93–100% across iterations at 94% consistency. |
Upstream advanced 30 commits, adding the agentcore-observability-setup and eks-upgrade-readiness skills plus SUPPORT.md. Both conflicts were additive collisions in the two files every skill contribution touches: llms.txt and cloudformation/devops-agent-skill-policies.yaml. Resolved by keeping both sides throughout. The template needed care rather than a keep-both: the two new IAM policy resources shared their Roles, PolicyDocument and trailing Resource lines across the conflict boundary, so a naive resolution would have produced one policy with the other's statements. Reconstructed as two complete AWS::IAM::Policy resources, and gave EnableAwsBackupCoverageReview its own AllowedValues and Default rather than letting them attach only to the AgentCore parameter. Verified with aws cloudformation validate-template that both Enable parameters, both Conditions and both Policy resources survive, and that both llms.txt entries are present. Also corrects two stale counts carried in our own lines: the llms.txt entry said 21 fixed checks and the template comment said 43 of 49 actions; both are now 23 checks and 45 of 52 actions.
The evals/structure/ and evals/best-practices/ directories were committed by an overly broad git add. They contain no account or customer data, but they should not be in a public repository: they carry the internal tool's full BP-01 to BP-17 check taxonomy and its judge model identifier, and that tool has not been through AppSec, so publishing its schema is its owner's decision rather than ours. They were also untidy — one of the two best-practices runs is a failing v1 kept alongside a passing v2, and four structure reports differed only by timestamp. The remaining evals/ contents now match what every other skill in the repository has: benchmark.json, eval_queries.json, evals.json, report.json, trigger_report.json and files/. Those come from the public evaluation framework that CONTRIBUTING.md points to. Evaluation results are recorded in the pull request discussion instead, where a summary belongs. Takes the contribution from 31 files to 17.
|
Reviewed as a Backup SME - this is a well-reasoned skill and the output holds up against what I verified in a live account. I verified the skill's output against a test account with known backup plans, vaults, and coverage gaps: it correctly identified unprotected EC2/EBS/Aurora/DynamoDB/DocumentDB/EKS/S3 resources, an orphaned recovery point excluded from both numerator and denominator, per-Region opt-in states, and sub-retention/vault-posture findings. Also, it works well in fetching the audit manager plans and reports to ensure that we have sufficient compliance reporting mechanisms. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
aws-backup-coverage-review, a read-only skill that reports AWS Backup coverage and posture across all enabled Regions of an account.Why. AWS Backup Audit Manager can answer "what isn't backed up?", but its
BACKUP_RESOURCES_PROTECTED_BY_BACKUP_PLANcontrol requires AWS Config resource recording, a framework, and a report plan that has already run. Accounts without that setup cannot answer the question at the moment an operator asks it. This skill computes the answer live from read-only APIs, building its own resource inventory and diffing it againstListProtectedResources, and treats AWS Config as an optimization rather than a prerequisite.The core model is that coverage is not binary. Every eligible resource resolves to one of six states:
ProtectedStaleSelectedNotProtectedUnprotectedOptInBlockedOrphanedRecoveryPointListProtectedResourcesafter the resource was deletedThe middle three all render as healthy in the AWS Backup console, which is what makes them worth a skill. Testing surfaced a real instance of this: a backup selection that named a resource by literal ARN after that resource had been deleted, so every scheduled job failed while the plan still showed as active with a recent execution date — and the resources that had replaced it were covered by nothing.
Structure. 23 fixed, numbered checks across 5 dimensions (service enablement, coverage, plan quality, vault posture, coverage integrity), with a mandatory 23-row Check Coverage Matrix so no check can be silently dropped — following the
analytics-opensearch-expertisepattern. Thresholds match the AWS Backup Audit Manager control defaults so output is comparable with Audit Manager.Read-only throughout, with an explicit API allowlist and hard denials on any
Put*/Delete*/Create*/Update*/Start*. CloudTrail is deliberately unused, since the agent tool policy currently classifies that namespace as mutative. Permission gaps are excluded from the coverage denominator and cap the rating rather than being scored as coverage gaps — a blind spot is not a coverage gap.Non-duplication. No existing skill covers the AWS Backup service.
storage-s3-resiliency-expertiseexplicitly excludes it ("Do NOT use for … EFS, FSx, AWS Backup, or Storage Gateway").rds-operation-reviewcovers RDS-nativeBackupRetentionPeriod. Open PR #26 (drift-detection-baseline) has one row checking RDSBackupRetentionPeriodand DynamoDB PITR — that is service-native configuration, not the AWS Backup service, and there is no overlap in APIs or findings.IAM. Verified with
iam:SimulatePrincipalPolicyagainst live agent roles:AIDevOpsAgentAccessPolicyalready allows 45 of the 52 actions used. Only six need adding —backup:GetSupportedResourceTypes,config:SelectResourceConfig,dsql:ListClusters,storagegateway:ListFileShares,storagegateway:ListGateways,storagegateway:ListVolumes— all read-only. Added under a newEnableAwsBackupCoverageReviewparameter incloudformation/devops-agent-skill-policies.yaml. Every mutating AWS Backup action (StartBackupJob,StartRestoreJob,StartCopyJob,DeleteRecoveryPoint,DeleteBackupPlan,PutBackupVaultLockConfiguration,UpdateRegionSettings) was confirmed denied on all roles tested, so the read-only guarantee is enforced by IAM rather than only by the skill's instructions.Also in this PR:
llms.txtentry, and a paired custom agent atcustom-agents/aws-backup-coverage-review/following the pattern used by the repository's other report-producing skills.Type of change
Testing
Agent Skill Eval — root
"passed": true, Overall Grade A (0.91)Note on the functional score: the sub-scores are outcome 1.0, style 1.0, efficiency 1.0, process 0.21.
process_scoreismin(1, mean_tool_calls_with / mean_tool_calls_without), so it drops precisely because the skill reaches the answer in fewer tool calls than the no-skill baseline. Every functional assertion passes.Manual DevOps Agent testing — tested over repeated runs across several Agent Spaces against a test account with real backup plans, vaults, and coverage gaps. Prompts were always phrased as an operator would ("What isn't being backed up in this account?"), never naming the skill. Triggering was consistent in every run.
Early runs surfaced accurate findings but returned them as a conversational summary rather than the report. Adding an Output Contract and a countable self-check to
SKILL.mdimproved this but did not make it reliable — the failures correlated with the host delegating the account sweep to a research subagent, which does not reliably load the skill's reference files.The fix was to follow the pattern every report-producing skill in this repository already uses: a paired custom agent whose system prompt restates the report structure. Since adding
custom-agents/aws-backup-coverage-review/, runs executed through the custom agent have produced the complete report artifact consistently — all required sections, the full 23-row Check Coverage Matrix, and self-consistent arithmetic. The skill remains usable on its own in chat, where it produces the findings but not reliably the full report; that is documented in the skill README.Findings verified independently with direct read-only API calls against the test account. Across runs the skill correctly identified: unprotected EC2, EBS, Aurora, DynamoDB, DocumentDB, EKS, S3 and CloudFormation resources; a backup selection referencing a deleted EC2 instance whose jobs had been failing silently with
InvalidInstanceID.NotFound; per-Region resource type opt-in states; an orphaned recovery point for an EFS file system deleted four years earlier, correctly excluded from both numerator and denominator; sub-35-day retention on three plans; absent cross-Region and cross-account copies; unlocked vaults; missing vault access policies and failure notifications; and absent Audit Manager report plans and frameworks including the AWS Config dependency that blocks framework controls.Two behaviours were added in response to live testing and are worth calling out, because both are safety-relevant:
DeleteBackupSelectionandCreateBackupSelectionwhen asked to fix a finding. IAM refused it, but the skill had said nothing about what to do when a user asks for remediation. It now returns the exact change and resource identifiers for a human to apply and declines to attempt the call, rather than relying on IAM to stop it.Known limitation, also stated in the README. Per-resource states are reliable: a named ARN reported as unprotected is a verified fact, and the findings and remediation are sound. Aggregate counts are less precise. In the most recent verified run the account-wide total was 113 eligible resources, with S3 bucket counts and per-Region attribution exact and CloudFormation within two of actual. The skill therefore presents the coverage percentage as indicative, names the Coverage Matrix as the record of record, and never lets a coverage total influence a severity. I would rather state this plainly than imply a precision the skill does not have.
Also validated:
mkdocs build --strictpasses and both the skill and custom agent catalog pages render with correct metadata, tools and skills extracted;aws cloudformation validate-templatepasses and resolves the new parameter; every AWS documentation URL inreferences/backup-best-practices.mdreturns HTTP 200; the 23 check IDs are identical betweenreferences/coverage-logic.mdand the Check Coverage Matrix inreferences/report-format.md; andiam:SimulatePrincipalPolicyagainst live agent roles confirms all 7 mutating AWS Backup actions remain denied with the six added read actions in place.One platform issue encountered, unrelated to this skill.
backup:ListRestoreTestingPlansis cancelled inside DevOps Agent, reported by the agent as being held by an operator-approval guardrail — which an unattended custom-agent run has no operator to satisfy — in every Region and on every role tested — including roles whereiam:SimulatePrincipalPolicyconfirms the action is allowed and where the API succeeds when called directly outside the agent. Check 5.1 therefore returnsToolingFailurerather than a verdict. Every otherbackup:List*call in the same runs succeeded, so this appears specific rather than namespace-wide. Raising separately.License confirmation