Skip to content

fix(aws-cloudwatchlogs): data protection and account policies - #1700

Merged
NitinKumar004 merged 1 commit into
developmentfrom
fix/cloudwatchlogs-policies-1476
Oct 23, 2026
Merged

NitinKumar004 merged 1 commit into
developmentfrom
fix/cloudwatchlogs-policies-1476

Conversation

@NitinKumar004

Copy link
Copy Markdown
Collaborator

This adds CloudWatch Logs data protection policies and account policies. Until now these six operations returned UnknownOperationException, so aws_cloudwatch_log_data_protection_policy, aws_cloudwatch_log_account_policy and any CLI or SDK code that masks sensitive log data failed against cloudemu.

Data protection (PutDataProtectionPolicy, GetDataProtectionPolicy, DeleteDataProtectionPolicy, on the optional driver.AWSLogging interface; the wire handler only translates)

  • logGroupIdentifier takes a log group name or ARN, with the API's length and pattern rules.
  • The policy document is validated against the AWS syntax:
    • at most 30720 characters;
    • Version must be 2021-06-01 when present;
    • exactly one Audit statement with a FindingsDestination and one Deidentify statement with an empty MaskConfig;
    • both statements must list the same DataIdentifier array;
    • identifiers are managed ARNs (arn:aws:dataprotection::aws:data-identifier/<Name>) or custom identifiers from Configuration.CustomDataIdentifier;
    • custom identifiers: at most 10, names up to 128 characters, regexes up to 200 characters from the documented character set, and no name may match a managed identifier;
    • a CloudWatch Logs findings destination must be an existing log group.
  • Masking happens at ingest, so events ingested before the policy stay as they were. Each sensitive character becomes *, and every egress sees the masked text: GetLogEvents, FilterLogEvents, Logs Insights, metric filters, subscription filters and export tasks.
  • unmask: true on GetLogEvents, FilterLogEvents and GetLogRecord returns the original text, which the provider keeps alongside the masked event.
  • Detected managed identifiers are EmailAddress, IpAddress (IPv4 and IPv6), AwsSecretKey (keyword plus a 40-character key) and CreditCardNumber (Luhn-checked), plus any custom regex. Every regex runs on Go's RE2 engine, which is linear-time, so a hostile custom pattern cannot backtrack.
  • GetDataProtectionPolicy on a group without a policy returns no policyDocument, which Terraform reads as gone. DescribeLogGroups reports dataProtectionStatus: ACTIVATED while a policy is set.
  • Errors:
    • a missing group, or deleting a policy that doesn't exist, returns ResourceNotFoundException;
    • an invalid document or identifier returns InvalidParameterException.

Account policies (PutAccountPolicy, DescribeAccountPolicies, DeleteAccountPolicy)

  • All five policy types are covered: DATA_PROTECTION_POLICY, SUBSCRIPTION_FILTER_POLICY, FIELD_INDEX_POLICY, TRANSFORMER_POLICY and METRIC_EXTRACTION_POLICY. scope is ALL (the default), and policies are keyed by type and name.
  • selectionCriteria is parsed and checked per type:
    • LogGroupName NOT IN [...] for subscription filters;
    • LogGroupNamePrefix for transformers;
    • LogGroupNamePrefix, or DataSourceName AND DataSourceType, for field indexes;
    • IN / NOT IN on names or prefixes for metric extraction;
    • none for data protection.
  • Quotas:
    • one data protection policy and one subscription filter policy per account;
    • otherwise one unscoped policy plus up to 20 scoped ones (5 for metric extraction);
    • scoped prefixes can't overlap, and a type can have only one NOT IN policy;
    • LimitExceededException over a quota, InvalidParameterException for an overlap.
  • The account data protection policy masks every log group, including ones created later, cumulatively with the group's own policy.
  • The account subscription filter policy delivers matching events in addition to the group's filters, and skips the groups excluded by NOT IN.
  • Field index, transformer and metric extraction documents are validated, stored and returned (CRUD parity), with no data-plane effect.
  • policyName can't start with aws/. DescribeAccountPolicies requires policyType and accepts policyName and accountIdentifiers (another account returns an empty list). Deleting an unknown policy returns ResourceNotFoundException.

Persistence: snapshots keep group policies, account policies and the original text of masked events, and restored policies keep masking.

Not modelled (written to docs/coverage/nongoals/logging.md):

  • other managed identifiers are accepted but not detected;
  • audit findings reports and the LogEventsWithFindings metric;
  • logs:Unmask enforcement;
  • the Insights unmask command;
  • the account policy's 5-minute rollout delay;
  • the data-plane effects of field index, transformer and metric extraction policies.

Tests

  • dataprotection package: every validation rule, each detector (including non-matches such as clock times and Luhn failures), cumulative masking, and a linear-time check on a (a+)+b custom regex.
  • Provider:
    • masking only after a put, unmask on all three reads;
    • group and account policies combining, metric filters and subscriptions receiving masked text;
    • lifecycle and error codes;
    • account policy validation, quotas and overlaps;
    • NOT IN exclusion on account subscription delivery;
    • a snapshot round trip.
  • aws-sdk-go-v2 wire tests: data_protection_sdk_test.go and account_policy_sdk_test.go.

E2E against cloudemu serve, all checks passed

  • AWS CLI:
    • put-data-protection-policy by ARN, then events containing an email and an aws_secret_access_key;
    • get-log-events and filter-log-events showed the values masked, and --unmask showed the originals; an event from before the policy stayed unmasked;
    • updating the policy added IpAddress masking, and after delete get-data-protection-policy returned no document;
    • every error case: ResourceNotFoundException ×3, InvalidParameterException ×5;
    • put-account-policy for DATA_PROTECTION (which masked the group), SUBSCRIPTION_FILTER with NOT IN, FIELD_INDEX and TRANSFORMER;
    • describe-account-policies by type and by name;
    • account policy errors: LimitExceededException ×2, InvalidParameterException ×6;
    • delete-account-policy, a second delete returning ResourceNotFoundException, and no masking afterwards.
  • Terraform (hashicorp/aws 6.66.0) with aws_cloudwatch_log_data_protection_policy and two aws_cloudwatch_log_account_policy resources (DATA_PROTECTION_POLICY and SUBSCRIPTION_FILTER_POLICY):
    • apply, then plan showed no changes;
    • an in-place update changed the identifiers on both policies and the subscription pattern (3 changed, no replacement), then plan showed no changes;
    • masking was verified after both applies;
    • destroy, then describe-account-policies was empty.

Part of #1476

@NitinKumar004
NitinKumar004 merged commit 7f0d741 into development Oct 23, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant