K10 Log Bundle Anonymizer

A Python utility that anonymizes sensitive infrastructure data from Kasten K10 support log bundles, making them safe to share externally (e.g. with vendor support) while preserving log structure and debuggability.

Why

K10's built-in log collector (k10tools) does a good job masking secret values, but the resulting bundle still contains infrastructure fingerprinting data: cluster identity, internal IPs, storage targets, FQDNs, and customer workload namespaces. This script replaces those with consistent, deterministic fake values and produces a reversible mapping table so the original data can be recovered internally if needed.

This is a community tool. Anyone working with K10 support bundles — operators, partners, customers — can use it before sharing logs outside their organization.

What gets anonymized

#	Category	Example (before → after)
1	Cluster identity – UUIDs and short names	a1b2c3d4-e5f6-… → 00000000-…-ef0000000001, my-ocp-cluster → cluster01
2	FQDNs / OpenShift routes – any TLD	kasten.apps.my-ocp-cluster.example.com → redacted-host02.example.internal
3	Object storage – endpoints (incl. regional S3), buckets, S3 access keys	s3.us-east-1.amazonaws.com → s3.anon-storage01.example.com
4	IPv4 addresses – pod IPs, ClusterIPs, node IPs	10.131.2.45 → 198.18.0.21
5	Customer workload namespaces – all lengths	my-app-staging → app-ns-005, dev → app-ns-001
6	Kopia debug logs – S3 URLs, AWS SigV4 Credential=	Credential=ABCD1234…/date/region/s3/… → Credential=REDACTED_ACCESS_KEY_002/…
7	OIDC auth tokens – authorization codes and state params	code=xyzabc123… → code=REDACTED_OIDC_CODE_0013
8	IPv6 addresses – pod, node and service addresses	2001:470:1f0b:abc::42 → 2001:db8::1
9	Bearer credentials – JWTs (service account / OIDC id_tokens) and opaque bearer tokens	Bearer eyJhbGciOi… → Bearer REDACTED_JWT_001
10	Email addresses – OIDC claims emitted by Dex/auth-svc	john.doe@customer-corp.com → user001@example.invalid

Standard OpenShift (openshift-*) and K10 (kasten-io) namespaces are preserved so logs remain useful for troubleshooting. Binary files in the bundle are copied byte-for-byte without modification.

The reversal key (mapping JSON + summary) is written outside the output directory, so the directory you archive and send never contains the means to undo the anonymization. See Output structure.

Requirements

• Python 3.9+
• No external dependencies (stdlib only)
• Optional: pyahocorasick for faster literal matching on large pattern sets (see Aho-Corasick acceleration)

Quick start

# 1. Extract the K10 log bundle
unzip k10logs.zip -d k10logs

# 2. Run the anonymizer
python3 k10_log_anonymizer.py k10logs/k10logs anonymized_logs

# 3. Review the output (the summary is a SIBLING of the output dir)
cat anonymized_logs_summary.txt

Usage

python3 k10_log_anonymizer.py <input_dir> <output_dir> [options]

Option	Description
input_dir	Path to the extracted K10 log bundle directory (the anonymized bundle with --reverse)
output_dir	Destination for the anonymized copy, created if missing (the restored copy with --reverse)
--mapping-in FILE	Reuse an existing mapping JSON for cross-bundle consistency; required with --reverse
--mapping-out FILE	Where to write the mapping JSON. Default: <output_dir>_mapping.json, a sibling of output_dir
--reverse	Restore the original values in an anonymized bundle using --mapping-in
--dry-run	Detect and write the mapping without anonymizing files
--verify	After anonymization, re-scan the output and warn on remaining sensitive patterns; exits 3 if anything is found
-v, --verbose	Verbose output (DEBUG level)
-q, --quiet	Quiet output (WARNING level and above)

Exit codes: 0 success, 1 input directory missing, 2 fatal error (bad arguments, unusable mapping, input/output overlap), 3 --verify found a potential leak.

input_dir and output_dir must not contain one another — the run is refused rather than writing into the source bundle.

--dry-run cannot be combined with --verify (there is no output to check) and --reverse cannot be combined with --dry-run (--reverse has no detect-only mode). Both combinations are rejected rather than silently ignored.

If a reversal key is found inside the input, it is excluded from the output with a warning — it lists original values, so copying it into the shared directory would ship the key with the bundle. Both the v3.3 fixed names (_anonymization_mapping.json, _anonymization_summary.txt) and the v3.4 derived ones (<output_dir>_mapping.json, <output_dir>_summary.txt) are recognized. Because the v3.4 names are arbitrary, a candidate is confirmed by its content before being dropped: a customer file that merely ends in _mapping.json is kept. --verify reports any key that does reach the output, so exit-3 gating catches it.

Common workflows

Validate detection before producing output:

python3 k10_log_anonymizer.py k10logs/ /tmp/preview --dry-run
cat /tmp/preview_summary.txt

Anonymize with post-anonymization verification:

python3 k10_log_anonymizer.py k10logs/ anonymized_logs/ --verify

Process multiple bundles from the same cluster with consistent mapping:

# First bundle establishes the mapping (written to out_day1_mapping.json)
python3 k10_log_anonymizer.py bundle_day1/ out_day1/

# Subsequent bundles reuse it so the same IP, namespace, etc. map identically
python3 k10_log_anonymizer.py bundle_day2/ out_day2/ \
    --mapping-in out_day1_mapping.json

python3 k10_log_anonymizer.py bundle_day3/ out_day3/ \
    --mapping-in out_day2_mapping.json

Read back what an anonymized label meant:

# Support replies "the failure is in app-ns-007" — restore the real names
python3 k10_log_anonymizer.py out_day1/ restored_day1/ \
    --reverse --mapping-in out_day1_mapping.json

--reverse refuses to run on an ambiguous mapping (the same anonymized label pointing at two different originals) rather than guessing. The restored directory holds the original data and must be treated as internal.

Output structure

anonymized_logs_mapping.json      # Reversal key, mode 0600 — NEVER share
anonymized_logs_summary.txt       # Human-readable mapping overview, mode 0600 — NEVER share
anonymized_logs/                  # <-- this is the only thing you send
├── k10_debug_logs.txt            # Anonymized debug logs
└── services/
    ├── auth-svc-xxxxx.txt
    ├── catalog-svc-xxxxx.txt
    ├── crypto-svc-xxxxx.txt
    ├── ...
    └── gateway-xxxxx.txt

The two key files are siblings of the output directory, never inside it, and are created with owner-only permissions (0600). This is deliberate: they list every original value, so zip -r bundle.zip anonymized_logs/ cannot accidentally ship the means to undo the anonymization. Use --mapping-out to place them elsewhere (a warning is logged if you point them back inside the output directory).

Mapping table

The <output_dir>_mapping.json file is the only way to reverse the anonymization (see --reverse). Keep it in a secure, internal location — do not share it alongside the anonymized logs.

Structure:

{
  "cluster_uuid": {
    "a1b2c3d4-e5f6-7890-abcd-ef1234567890": "00000000-0000-0000-abcd-ef0000000001"
  },
  "cluster_name": {
    "my-ocp-cluster": "cluster01"
  },
  "ip_address": {
    "10.131.2.45": "198.18.0.21",
    "172.30.99.10": "198.18.0.4"
  }
}

Performance

Benchmarked on two real-world bundles, measured on v3.3:

Bundle	Size	Files	Patterns	Time	Throughput
Small (S3-compatible storage, 65 patterns)	95 MB	22	65	8.8s	10.9 MB/s
Large (OIDC+Kopia debug, 2919 patterns)	285 MB	44	2919	74s	3.8 MB/s

The throughput difference reflects the OIDC token volume: the larger bundle had 2500+ unique OIDC state tokens and 45+ auth codes, handled by dedicated regex passes. The core mega-regex stays small (~90 literals).

v3.4 cost. The three new detection passes (IPv6, email, JWT/bearer) each scan roughly as much text as the existing IPv4 and FQDN passes, so they add a linear, predictable cost rather than a pathological one. On a 78 MB generated bundle of K10-shaped log lines, throughput dropped from 6.1 MB/s to 4.8 MB/s — about 21% slower for the added coverage. Per-pattern cost on that bundle was ~0.02 s/MB for IPv6 and email, ~0.01 s/MB for JWT, against ~0.03 s/MB for the pre-existing IPv4 pass.

These are one-off measurements on a generated fixture that is not committed to the repo, taken on a single machine — treat them as an order of magnitude, not a benchmark. The real-bundle figures above, and the "~63% of total time" and "46% faster" figures below, were measured on earlier versions and have not been re-taken.

How it works under the hood

The script runs in two phases: detection (scan all files to discover sensitive values) and replacement (substitute them with anonymized equivalents).

File classification happens first: each file is checked in full for NUL bytes or invalid UTF-8. Binary files are copied verbatim to the output and never anonymized; only text files enter the detection pipeline. Whole-file classification matters because a file that looks like text in its first few KB but turns binary later would otherwise be decoded lossily and written back with those bytes destroyed.

Detection accounts for ~63% of total time on text files. Files are read once and cached in memory for reuse during replacement. Detection patterns are split into two groups: "always-scan" patterns (IPs and domains, which have no cheap keyword trigger) and "keyword-gated" patterns that only run on files containing the relevant keyword. The gated set covers cluster_name, namespace, appName, /namespaces/, bucket, endpoint, cluster, /clusters/, accessKeyID, Credential=, code=, oidc-auth-state, eyJ (JWT), earer (bearer) and @ (email).

Replacement uses a multi-pass strategy, each pass optimized for its data type:

1. JWTs — pattern pass matching three to five base64url segments whose first starts with eyJ (base64 of {", the opening of every JWT header), with dict lookup. Three segments is a JWS, five a compact JWE; matching only the first three would leave a JWE's last two segments in the output.
2. Opaque bearer tokens — pattern pass on Bearer <token> for anything that is not a well-formed JWT.
3. Email addresses — pattern pass with dict lookup.
4. IPv6 addresses — loose candidate pattern, then every candidate is validated with the stdlib ipaddress module. Matches are normalized to their canonical compressed form before lookup, so 2001:0470:0::1 and 2001:470::1 resolve to the same anonymized address. Validation is what keeps clock times (12:34:56), MAC addresses and host:port pairs out of the results. The address is replaced by an internal placeholder, resolved after every other pass — see Pass interference.
5. Non-IP literals (domains, endpoints, access keys, namespaces ≥ 4 chars, bucket names, cluster names ≥ 8 chars) — single compiled alternation regex or Aho-Corasick trie, with dict callback.
6. Cluster UUIDs — dedicated regex pass with case-insensitive dict lookup. UUIDs are handled by regex rather than as literals so that mixed-case forms (A1b2C3d4-...) resolve to the same anonymized value as their canonical lowercase form without requiring every casing variant to be pre-enumerated in the literal table.
7. IPv4 addresses — single octet-bounded pattern (each octet 0–255) with dict lookup. Separate from literals because a pattern-based approach is 46% faster than 300+ IP alternations.
8. OIDC codes — dedicated code=xxx regex with dict callback, kept out of the literal table so thousands of unique tokens cannot degrade it.
9. OIDC states — same treatment for state=oidc-auth-state-xxx.
10. Short namespaces (< 4 chars) — contextual regex that only replaces in known patterns (namespace=, appName=, /namespaces/, "namespace"=>), avoiding false positives from substring matches.
11. Short cluster names (< 8 chars) — contextual regex anchored on cluster=, cluster_name=, and /clusters/<name>/. Common cluster names like prod, dev, qa are whole words in ordinary log prose often enough that anchoring, not just boundary matching, is the safer default. Longer names remain in the literal table where collision probability is negligible.

The credential passes run before the literal table so each secret is replaced as one opaque unit — substituting a domain inside an email address first would leave a half-anonymized address behind.

Pass interference

Hex groups inside an IPv6 address are exactly the shape of a short literal — beef, db8, 2001 are all plausible namespace or bucket names — and : is not an identifier character, so the boundary check accepts them. Neither pass order is safe:

• literals before IPv6 corrupts the real address: 2001:beef::42 → 2001:app-ns-001::42;
• IPv6 before literals corrupts the anonymized output: 2001:db8::1 → 2001:anon-bucket-001::1.

Both leave the mapping describing an address that appears nowhere in the output. The IPv6 pass therefore substitutes a NUL-delimited placeholder and the real anonymized address is written back after every other pass. A bundle's text never contains NUL — such files are classified binary and never reach the replacer — so no literal can match across or inside a placeholder, and the ordering dependency disappears rather than being traded for a different one.

Identifier boundaries: the literal pass never substitutes a value that is embedded in a longer identifier. A match is only applied when neither neighbouring character is in [A-Za-z0-9_-], so a namespace named data leaves metadata, dataset and database untouched. . is deliberately not a continuation character, so a namespace is still replaced inside service FQDNs like myns.svc.cluster.local.

Cross-category collision avoidance: a value detected in multiple categories (e.g. an OCP api.* FQDN matched both as a domain and as a storage_endpoint) is claimed by the highest-priority category only. Priority order: domain > storage_endpoint > bucket > access_key > namespace ≥ 4 > cluster_name. This prevents the same string from receiving two different anonymized values across passes.

Optional: Aho-Corasick acceleration

If pyahocorasick is installed (pip install pyahocorasick), the script automatically uses it for the non-IP literal matching pass. This replaces the regex alternation with an O(text_length) trie automaton, which scales better when the number of patterns grows into the hundreds (large multi-cluster environments with many namespaces, domains, and endpoints).

Matches reported by Aho-Corasick are resolved with longest-leftmost semantics: when two patterns overlap (e.g. my-cluster and kasten.apps.my-cluster.example.com both match starting at the same region of text), the longer match wins. Without this resolution step, the trie would emit matches in end-position order and short patterns finishing earlier could displace longer ones, producing partial substitutions like kasten.apps.cluster01.example.com.

Both backends enforce the same identifier boundaries and must produce byte-identical output. Boundary filtering happens before overlap resolution in the Aho-Corasick path, so a rejected match never masks a valid overlapping one. The test suite asserts both backends agree, and checks the active backend against a brute-force reference implementation.

For bundles with fewer than ~50 non-IP patterns, the built-in mega-regex (which is implicitly longest-first thanks to length-descending alternation) is already efficient and Aho-Corasick is not required.

Going further: Rust/PyO3

For extreme throughput requirements (multi-GB bundles, automated pipelines processing dozens of bundles per day), the detection and replacement hot paths can be rewritten in Rust using PyO3. The Rust aho_corasick and regex crates (the same ones behind ripgrep) provide SIMD-accelerated matching that can reach 50–100 MB/s. The Python script would import the compiled Rust module as a drop-in replacement, with automatic fallback to pure Python if the Rust extension is not installed.

Design decisions

Consistency — The same original value always produces the same anonymized value across all log files. If an IP appears in both gateway and executor-svc logs, it maps to the same anonymized IP in both. With --mapping-in, consistency also extends across bundles.

Single-read architecture — Every text file is read once during detection, cached in memory, then reused for replacement. This eliminates the double I/O cost of scanning then re-reading. Binary files are detected up front and bypass the cache entirely.

The reversal key never travels with the bundle — The mapping and summary list every original value, so writing them into the directory the user is about to archive and send would defeat the whole tool. They are written as siblings of output_dir with mode 0600. The output directory contains anonymized logs and nothing else.

Binary preservation — Files containing NUL bytes or invalid UTF-8 anywhere are classified as binary and copied verbatim via shutil.copy2, preserving content and metadata. Text is decoded strictly, never with errors="replace", so a file is either reproduced exactly or copied byte-for-byte — it can never be written back lossily.

Idempotence — Running the anonymizer on its own output is a no-op, for every category. Detection filters out the anonymized output pool itself — app-ns-NNN, anon-bucket-NNN, clusterNN, REDACTED_*, oidc-auth-state-redacted-NNNN, userNNN@example.invalid, redacted-hostNN.example.internal, s3.anon-storageNN.example.com, the 198.18.0.0/16 and 2001:db8::/32 ranges (plus the legacy 198.51.x pool emitted up to v3.3), and the 00000000-0000-0000-abcd-ef... UUID range. This is a correctness property, not a convenience: re-detecting a label makes a second pass reassign it (app-ns-002 → app-ns-001 while a genuinely new namespace takes app-ns-002), which silently turns the mapping into a wrong reversal table. Indices already present in the input are additionally reserved, so a fresh value in a bundle mixing anonymized and new content can never be handed a label the text already uses.

Strict IP validation — The IP regex enforces each octet to be 0–255, reducing false positives from version strings, build numbers, or hash fragments that happen to look like dotted quads.

IP range choice — Anonymized IPv4 addresses are drawn from 198.18.0.0/16 (65 536 labels), which sits inside the 198.18.0.0/15 block reserved for benchmarking by RFC 2544 and listed as special-purpose and not globally routable by RFC 6890. A bundle with more than 65 536 unique addresses stops with an error rather than recycling labels. IPv6 addresses use 2001:db8::/32 (RFC 3849, reserved for documentation). Neither will collide with real infrastructure.

Up to v3.3 the IPv4 pool was documented as 198.51.0.0/24 "RFC 5737 TEST-NET-2". That was wrong twice over: TEST-NET-2 is 198.51.100.0/24, and the counter actually emitted across the whole of 198.51.0.0/16 — mostly real, allocatable address space, so anonymized logs could appear to reference genuine third-party hosts. Output from those versions is still recognized at detection, so upgrading does not break idempotence or --mapping-in chains.

Boundary-aware literals — A literal is only substituted when it is not embedded in a longer identifier. Without this, a namespace named data rewrote metadata into metaapp-ns-001 across the whole bundle: unreadable logs, and a leak of the anonymization scheme into unrelated words.

Multi-format support — K10 logs mix several formats and the replacer handles all of them: standard JSON ("cluster_name":"a1b2c3d4-…"), Fluentd/Ruby ("namespace"=>"my-app"), Prometheus labels (bucket="my-bucket"), freetext (Kopia description strings, URL query params like cl=my-cluster or appName=dev), Kubernetes API paths (/api/v1/namespaces/<ns>/... as seen in audit logs and raw apiserver dumps), Kopia HTTP debug dumps (full S3 request URLs with Authorization: AWS4-HMAC-SHA256 Credential=…), and OIDC redirect URLs (code=xxx&state=oidc-auth-state-xxx).

Safe-list approach — Platform namespaces (openshift-*, kube-*, kasten-io) and K8s/OCP API groups are explicitly preserved. Only customer-specific namespaces get anonymized.

No K8s secrets in scope — The script does not target K8s Secret values (they're already masked by K10's collector as <set to the key ... in secret ...>). It does catch: S3 access key IDs from both accessKeyID= fields and AWS SigV4 Credential= headers in Kopia HTTP debug logs, and OIDC authorization codes and state tokens from gateway redirect URLs.

Verification

The --verify flag re-scans the output directory after anonymization and reports any sensitive patterns that may have slipped through. It checks for:

• Anonymizer reversal keys (mapping or summary files) present in the output at all
• IPv4 addresses outside the anonymized range (198.18.0.0/16, or the legacy 198.51.x pool)
• IPv6 addresses outside 2001:db8::/32
• FQDN patterns that don't end with example.internal and aren't in the safe-list
• S3 URLs that don't point to anon-storage*
• JWTs and Bearer tokens that aren't already redacted
• Email addresses that don't end with @example.invalid
• OIDC code= and state= values that aren't already redacted
• AWS SigV4 Credential= values that aren't already redacted

Findings are capped at 5 examples per category across the whole tree, and the process exits 3 when anything is found, so --verify can gate a pipeline:

python3 k10_log_anonymizer.py k10logs/ out/ --verify || echo "leak detected, do not send"

This is a useful safety net for high-confidence delivery to external recipients. A clean verify pass is not a formal proof of completeness, but it catches the common gaps.

Tests

A unit test suite is included:

python3 -m unittest test_anonymizer.py -v

The suite (124 tests) covers binary file classification, mapping consistency, the longest-match overlap resolution, OIDC handling, AWS SigV4 detection, S3 URL detection with regional hostnames, safe-namespace preservation, and the verification pass. TestRegressionsV33 guards the four v3.3 fixes: short cluster_name word boundary (prod must not corrupt production), namespace visible only in /namespaces/<ns>/ API paths, mixed-case UUID detection, and double-pass idempotence.

The v3.4 classes each guard one of the fixes in that release:

Class	Guards
TestBoundaryAwareLiterals	A namespace or bucket name is never substituted inside a longer word; . stays a boundary so service FQDNs still get replaced
TestLongClusterName	Cluster names of 20+ characters are anonymized; a cluster UUID is not mistaken for a name
TestReversalKeyLocation	Key files land outside the output dir, are 0600, and --mapping-out is honoured
TestJWTAndBearer	JWTs and opaque bearer tokens are redacted, consistently, with no gap between the two passes
TestEmail	Emails are anonymized; Go module paths and @sha256: digests are not mangled
TestIPv6	IPv6 is anonymized; timestamps, MACs and host:port are not; all spellings of one address collapse to one value
TestIdempotenceV34	A second pass detects nothing, and no fresh value ever reuses a label already in the bundle
TestMappingStoreCounter	Counters resume past the highest index in use, not past the entry count
TestVerifyCap	The example cap holds across files
TestExitCodes	0 clean, 3 on verify findings, 2 on fatal error
TestDirectoryOverlap	Input/output containment is refused in both directions
TestBinaryClassification	Late NUL bytes and late invalid UTF-8 are preserved byte-for-byte
TestNoSysExitInLibrary	process_bundle raises instead of killing the interpreter
TestReverseMode	Round-trip restores the original exactly; ambiguous mappings are refused
TestLiteralReplacerSemantics	The active backend matches a brute-force reference, and both backends agree
TestVerifyFalsePositives	--verify mirrors detection's safe-list, so svc.cluster.local does not trip exit-3 gating
TestStaleKeyFileInInput	A reversal key in the input never reaches the output, in either naming scheme; an unrelated file with the same suffix is kept; verification flags a key that does reach the output
TestClusterNameDetectionAnchors	Clusters named only at cluster_name= or /clusters/<name>/ are detected, and prose following the key is not captured
TestS3UrlIdempotence	S3 endpoints survive a second pass; bucket names starting with a hex-looking word are still detected
TestClusterUuidIndexReservation	Two distinct clusters never collapse onto one anonymized UUID
TestCounterIndexParsing	Hex- and dotted-quad-shaped labels resume the counter correctly
TestIPv6LiteralInterference	Literals and IPv6 cannot corrupt each other in either direction, no placeholder reaches the output, and the mapping always describes the actual output
TestMultiSegmentJWT	A five-segment JWE is fully redacted; verification flags an uppercase BEARER leak
TestBearerCaseInsensitive	BEARER and BeArEr are redacted; a padded JWT is mapped once
TestIPv4PoolExhaustion	The pool errors instead of recycling labels
TestCliArgumentCombinations	Contradictory flag combinations are rejected
TestAnonShapeInvariants	Every formatter's output is recognized as anonymized and reads back to its own index, for all 14 categories

One test is skipped unless pyahocorasick is installed.

Customization

The script has configuration constants at the top of the file that can be adjusted per environment:

Constant	Purpose
KNOWN_SAFE_NAMESPACES	Namespaces that should never be anonymized
SAFE_NAMESPACE_PREFIXES	Namespace prefixes to preserve (e.g. openshift-)
IP_PASSTHROUGH	IPs to skip (loopback, unspecified)
SAFE_DOMAIN_PATTERNS	FQDNs that are API groups, not infrastructure
SAFE_DOMAIN_SUBSTRINGS	Domain substrings to always preserve (also includes anonymized output suffixes for idempotence)
IPV6_PASSTHROUGH	IPv6 addresses to skip (::, ::1)
ANON_IP_PREFIX	Prefix of the anonymized IPv4 pool; filtered out at detection to guarantee idempotence
LEGACY_ANON_IP_PREFIXES	Pools emitted by earlier versions, still recognized so upgrades stay idempotent
ANON_IPV4_POOL_SIZE	Number of available IPv4 labels; the run stops rather than recycling past it
ANON_KEY_FILENAMES / ANON_KEY_SUFFIXES	Reversal-key names and suffixes; a matching file whose content confirms it is excluded from the output
ANON_IPV6_NETWORK	Anonymized IPv6 pool (2001:db8::/32), likewise filtered at detection
ANON_FORMATTERS	The single source of truth for what each category's anonymized label looks like
ANON_OUTPUT_PATTERNS	Every shape this tool emits; a value matching one is never re-detected
ANON_LABEL_CATEGORIES	The same shapes with their index and owning category, used to reserve indices already present in the input
IDENT_CONT_CLASS / IDENT_CONT_CHARS	Characters that make a literal part of a longer identifier; a match adjacent to one is rejected
CLUSTER_NAME_LITERAL_MIN	Minimum cluster_name length to enter the literal table; shorter names use contextual matching, and names under 3 characters are skipped entirely
BINARY_CHUNK_BYTES	Chunk size used when streaming a file to classify it as text or binary
CACHE_WARN_MB	Soft warning threshold for in-memory text cache size
KEY_FILE_MODE	Permissions for the mapping and summary files (default 0o600)
MAX_RESERVABLE_INDEX	Upper bound on an index read back from an anonymized label

Adding new detection patterns

To anonymize additional categories:

1. Add a new set() in detect_and_cache() with the appropriate regex — if the pattern has a keyword anchor (e.g. vaultAddr), add it to the gated patterns list for efficiency.
2. In build_replacer(), decide the replacement strategy:
   • Literal (few unique values, low collision risk, e.g. long hostnames): add to non_ip_lookup → goes into the mega-regex.
   • Pattern-based (many unique values, e.g. session IDs, or any value where casing must be normalized like UUIDs): add a dedicated regex pass in replace_content() with dict callback.
   • Contextual (short values, high false-positive risk because the value may match unrelated words): add to a separate contextual replacement like short_ns_patterns or short_cluster_patterns. Anchor on explicit key=value patterns and URL paths only.

The literal threshold for cluster names is governed by CLUSTER_NAME_LITERAL_MIN (default 8). Adjust if your environment uses longer or shorter cluster naming conventions that could collide with common log vocabulary.

Limitations

• Pod and container names are not anonymized (they contain K10 component names, not customer data).
• Timestamps are preserved as-is (needed for log correlation).
• K8s resource UUIDs (action IDs, manifest IDs) are not anonymized — only the cluster UUID is. This is intentional: action UUIDs are needed for troubleshooting and are not externally meaningful.
• Secret checksums (e.g. checksum/secret: abc123…) are left in place. They are SHA hashes of the secret content, not the secret itself, and are useful for verifying that all pods share the same secret version.
• Memory cap — the text cache lives in memory. A soft warning fires above 1 GB; for multi-GB bundles, consider splitting the input directory before running.

Coverage gaps (not currently anonymized)

These categories are out of scope for the current version. Review your bundle manually if any apply:

• Non-bearer opaque secrets — tokens that appear without a Bearer prefix or a recognized key name are not detected. Service account tokens are covered when they are JWTs (they normally are) or presented as Bearer <token>; a raw token pasted into a log line is not.
• OIDC claim values other than email (preferred_username, groups, name) emitted by Dex/auth-svc.
• Bare node hostnames without an FQDN suffix (e.g. worker-node-01 standalone, not part of a full domain).
• PVC, PV, and StorageClass names when they encode customer or application identifiers.
• Inline certificates (-----BEGIN CERTIFICATE----- blocks); the CN and SAN fields they carry may be identifying.

Two consequences of the idempotence guarantee are worth knowing:

• A real value that happens to be shaped exactly like anonymizer output (a cluster genuinely named cluster01, a namespace named app-ns-004) is treated as already-anonymized and passes through. This is the deliberate trade-off that makes double-passes safe.
• Index reservation reads labels from their normal anchored positions (namespace=, bucket=, an IP, a hostname). A stray label sitting in free-form prose is not reserved.

Changelog

v3.4 — Security hardening, coverage gaps, reversibility

Security — the reversal key shipped inside the shared directory. _anonymization_mapping.json and _anonymization_summary.txt list every original value — cluster UUIDs, internal IPs, FQDNs, bucket names, S3 access key IDs — and were written into output_dir, the directory the user archives and sends to vendor support. The README already warned "do not share it alongside the anonymized logs" while the tool did exactly that by default. Both files now land as siblings of output_dir with mode 0600, --mapping-out relocates them, and the output directory contains anonymized logs and nothing else.

Bug fix — substring leak on namespaces and buckets. v3.3 fixed this class of bug for short cluster names but left namespaces (≥ 4 chars), buckets, endpoints and access keys in the literal table with no word-boundary check. With a namespace named data, the output turned metadata into metaapp-ns-001, database into app-ns-001base and dataset into app-ns-001set; a namespace named test turned latest into laapp-ns-002. The literal pass is now identifier-boundary aware in both backends: a match is applied only when neither neighbouring character is in [A-Za-z0-9_-]. . remains a boundary so service FQDNs are still replaced.

Bug fix — long cluster names were never anonymized. The cluster_names detection filter was len(val) < 20, intended to reject UUIDs captured by the cluster= keyword. It also silently let every cluster name of 20 characters or more through: cluster=my-production-cluster (21 chars) appeared verbatim in the shared output. The length cap is replaced by an explicit UUID-shape rejection.

Bug fix — idempotence was incomplete, and the gap corrupted the mapping. v3.3 filtered only anonymized hosts, the 198.51.0.0/24 range and the anonymized UUID range. app-ns-NNN, anon-bucket-NNN, clusterNN, REDACTED_* and oidc-auth-state-redacted-NNNN were re-detected on a second pass. A directory holding the already-anonymized namespace="app-ns-002" plus a genuinely new namespace="customer-xyz" produced {"app-ns-002": "app-ns-001", "customer-xyz": "app-ns-002"} — two different originals colliding on one label, so reversing the mapping returned the wrong namespace. Every emitted shape is now filtered at detection, and indices already present in the input are reserved so a fresh value can never be handed a label the bundle already uses.

New — IPv6 anonymization. Detected with a loose candidate pattern validated by the stdlib ipaddress module, mapped into 2001:db8::/32 (RFC 3849). Clock times, MAC addresses and host:port pairs are rejected by validation. Addresses are keyed on their canonical compressed form, so every spelling of one address maps identically.

New — JWT and bearer token redaction. Service account tokens and OIDC id_tokens in Authorization: headers are live credentials and were passing through untouched. JWTs are matched as three base64url segments whose first starts with eyJ; anything else presented as Bearer <token> is redacted as an opaque token, so nothing falls between the two passes.

New — email address anonymization. OIDC claim emails emitted by Dex/auth-svc are PII and were passing through. Go module paths (client-go@v0.26.0) and image digests (app@sha256:…) are deliberately not matched.

New — --reverse. The README described the mapping as "the only way to reverse the anonymization" but nothing implemented reversal. --reverse restores originals from a mapping, and refuses to run on an ambiguous mapping rather than guessing.

Bug fix — --verify always exited 0. A leak found in an automated pipeline was silent. --verify now exits 3 on findings. Its example cap also applied per file, so findings kept accumulating across the tree; it now holds globally.

Bug fix — binary classification only inspected the first 4 KB. A file that looked like text in its first 4 KB but turned binary later was decoded with errors="replace" and written back with those bytes destroyed. Classification now covers the whole file and decoding is strict: a file is either reproduced exactly or copied byte-for-byte.

Bug fix — MappingStore.load_existing resumed from the entry count. A mapping that had been filtered, merged or hand-edited has fewer entries than its highest index, so the counter restarted inside the range already in use and handed an existing label to a different original. The counter now resumes past the highest index actually present.

Bug fix — process_bundle called sys.exit(). A library function killed the interpreter on a missing mapping file. It now raises AnonymizerError, which main() translates into an exit code.

New — input/output overlap guard. Running with output_dir inside input_dir (or the reverse, or both equal) wrote into the source bundle. The run is now refused.

Bug fix — --verify rejected every clean bundle. The FQDN check consulted SAFE_DOMAIN_SUBSTRINGS but not SAFE_DOMAIN_PATTERNS, which detection does honour. cluster.local appears in every K10 bundle, so verification always reported a finding — and once --verify started exiting 3, the documented pipeline gate rejected 100% of clean bundles.

Bug fix — verification had a blind spot, and stale keys leaked. verify_output skipped files named _anonymization_*. That was right in v3.3, when the key lived inside output_dir; now that it does not, the skip was a pure hole. Re-processing a v3.3 output directory — the documented upgrade path — copied its old _anonymization_mapping.json, original values and all, into the new shared directory, and verification never opened it. Every file is now scanned, and a reversal key found in the input is excluded from the output.

Bug fix — clusters named only at cluster_name= or /clusters/<name>/ leaked. The cluster= gated pattern cannot match cluster_name= (the next character is _, not one of =:">), and cluster_name was bound to the UUID pattern only. A cluster referred to by name at either anchor was never detected, so it appeared verbatim in the shared output — the same gap v3.3 closed for namespaces. Both anchors now have detection patterns.

Bug fix — S3 endpoints were re-anonymized on a second pass. The PAT_S3_URL block sits outside the gated loop and so never got the already-anonymized guard, letting s3.anon-storage01.example.com become s3.anon-storage02.example.com. One unguarded detection site is enough to re-map a value everywhere it appears.

Bug fix — two clusters could collapse onto one label. cluster_uuid was missing from ANON_LABEL_CATEGORIES, and its index is hex, not decimal. In a bundle mixing an already-anonymized cluster with a fresh one, both ended up as …-ef0000000001, indistinguishable in the output.

Bug fix — counter resume misread non-decimal labels. Scraping trailing digits reads 2001:db8::a as 8 and 198.18.1.0 as 0, so --mapping-in could hand an in-use label to a new value and make the mapping unreversible. The index is now read back by inverting each category's own formatter.

Bug fix — a hex-named literal corrupted IPv6 addresses. With a namespace named beef, 2001:beef::42 became 2001:app-ns-001::42 and the mapping described an address that appeared nowhere. The IPv6 pass now runs before the literal table.

Bug fix — bucket names starting with a hex-looking word were skipped. The ^[0-9a-f]{8}- filter meant to skip Kopia's UUID-named prefixes also swallowed real names like deadbeef-acme-payroll. It now matches the full UUID shape.

Internal — one source of truth for label shapes. ANON_FORMATTERS, ANON_OUTPUT_PATTERNS and ANON_LABEL_CATEGORIES have to agree, and both places they had silently drifted were mapping-corruption bugs. Every formatter now lives in ANON_FORMATTERS, and TestAnonShapeInvariants asserts the three agree in both directions across all 14 categories, so the next added category cannot forget one.

Bug fix — IPv6 and the literal table corrupted each other. Whichever ran first damaged the other's work, and in both directions the mapping ended up describing an address absent from the output. The IPv6 pass now writes a placeholder that no literal can match, resolved after every other pass.

Bug fix — compact JWE tokens leaked their tail. The JWT pattern matched exactly three segments, so a five-segment JWE had its last two written to the output verbatim. The pattern now spans three to five segments.

Also fixed: bearer headers in any casing were matched by neither detection nor verification — the substring gate could not express case-insensitivity and was narrower than the pattern it guarded; a base64-padded JWT was mapped twice; --dry-run --verify and --reverse --dry-run silently did the wrong thing and are now rejected; MAX_RESERVABLE_INDEX is applied to every category rather than IPv6 alone; the IPv4 pool errors instead of recycling labels past 65 536; overwriting an existing reversal key now warns.

Compatibility — Python 3.9. v3.3 used PEP 604 annotations evaluated at import time, so the module could not even be imported on Python 3.9 — the version shipped by stock macOS and RHEL 8, where support engineers run this ad hoc. from __future__ import annotations restores 3.9 support; the documented requirement drops from 3.10+ to 3.9+.

Pipeline change. replace_content grows from 7 to 11 passes: three credential passes (JWT, bearer, email) run before the literal table so each secret is replaced as one opaque unit, and an IPv6 pass follows IPv4.

New regression tests. The suite grows from 27 to 124 tests, with one class per fix above plus TestLiteralReplacerSemantics, which checks the active literal backend against a brute-force reference implementation and asserts both backends agree.

v3.3 — Anonymization completeness, substring leak fixes, idempotence

v3.2 and v3.3 are released as a single consolidated commit (v3.2 was never pushed). The combined entry covers all changes since v3.1.

Bug fix — substring leak on short cluster names. Cluster names like prod, dev, qa, or test were placed in the literal mega-regex without word boundaries, so they matched inside unrelated words. A cluster named prod corrupted production into cluster01uction and reproduce into recluster01uce. The corruption was silent and partially hid the cluster name fragment inside other words, defeating the anonymization. v3.3 introduces a length threshold CLUSTER_NAME_LITERAL_MIN (default 8): shorter names go through a contextual pattern that only matches at explicit anchors (cluster=, cluster_name=, /clusters/<name>/). Longer names remain in the mega-regex where collision probability is negligible.

Bug fix — namespace invisible in Kubernetes API URL paths. Namespaces that only appeared as /api/v1/namespaces/<ns>/... (kube-apiserver audit logs, raw API traces, oc get --raw dumps) were never detected and therefore never anonymized. A namespace mentioned elsewhere in the bundle was caught and then propagated to URL paths by the literal pass, but a namespace exclusive to URL paths leaked. v3.3 adds a gated detection pattern keyed on /namespaces/ with a lookahead-bounded capture.

Bug fix — UUIDs in mixed case not anonymized. The literal table was pre-populated with only the lowercase and uppercase forms of each detected UUID, missing every other casing variant. A UUID written as A1b2C3d4-E5f6-... (as commonly seen in logs that downcase only some components) passed through unchanged. v3.3 removes UUIDs from the literal table and handles them in a dedicated regex pass with case-insensitive dict lookup, so all casings resolve to the same anonymized value.

Bug fix — pipeline was not idempotent. Running the anonymizer twice on the same input corrupted previously-anonymized hosts: redacted-host01.example.internal became redacted-host01.redacted-host01.example.internal because .internal was matched as an internal TLD on the second pass. Similarly, anonymized IPs in 198.51.0.0/24 were re-detected and re-anonymized. v3.3 filters out the anonymized output pool itself at detection time (not just at the optional verification pass): example.internal, anon-storage*, the 198.51.0.0/24 range, and the 00000000-0000-0000-abcd-ef... UUID range are skipped.

Pipeline change. replace_content grows from 5 to 7 passes:

• New pass 2: UUID regex with case-insensitive dict lookup.
• New pass 7: short cluster_name contextual patterns. Pass ordering is documented inline. Existing pass semantics are unchanged.

New regression tests. The test suite grows from 23 to 27 tests with a dedicated TestRegressionsV33 class:

• test_short_cluster_name_word_boundary — prod must not match inside production/reproduce.
• test_namespace_only_in_url_path — namespace exclusive to /namespaces/<ns>/ is detected.
• test_mixed_case_uuid_replaced — all casings map to the same anonymized UUID.
• test_double_pass_is_stable — second pass on anonymized output is a no-op (output identical, mapping empty).

v3.2 — Correctness fixes and quality-of-life features

Bug fix — binary files were silently corrupted. Previous versions opened every file with errors="replace", which replaces invalid UTF-8 bytes with U+FFFD and writes the result as text. Any binary content (DER certs, archives, dumps) in the bundle was therefore corrupted in the output. v3.2 sniffs the first 4 KB of each file for NUL bytes or invalid UTF-8 and routes binary content through shutil.copy2 so it is preserved byte-for-byte.

Bug fix — overlapping matches in Aho-Corasick mode. When a short literal (e.g. cluster short name my-cluster) and a longer one (kasten.apps.my-cluster.example.com) both matched in the same region, the Aho-Corasick automaton emitted them in end-position order. The first-match-wins greedy pass could pick the shorter one, leaving partial substitutions like kasten.apps.cluster01.example.com. v3.2 collects all matches, sorts them by start position then by length descending, and applies longest-leftmost resolution. The mega-regex fallback was already correct (alternation is length-descending) but is now consistent with the AC path.

Bug fix — S3 URLs with regional hostnames were missed. The detection regex s3[a-zA-Z0-9.]*\.[a-z]{2,} did not allow dashes in the hostname, so s3.us-east-1.amazonaws.com and similar regional endpoints in Kopia HTTP debug logs were never detected. The character class now includes -.

Bug fix — cross-category collision. A value captured by two detection paths (e.g. an OCP api.* FQDN, captured both by the domain regex and by the endpoint= gated pattern) was inserted twice into the literal lookup and could receive an unstable replacement. v3.2 introduces a claimed set with explicit priority: domain > storage_endpoint > bucket > access_key > namespace ≥ 4 > cluster_name.

Bug fix — IP false positives. The IP regex now bounds each octet to 0–255 instead of accepting any 1–3 digit sequence, reducing false positives from version strings and decimal hash fragments.

Bug fix — cluster names too short to replace. Names shorter than 3 characters were added to the mapping but never replaced in the output, producing misleading mapping entries. They are now skipped at detection time.

Bug fix — summary truncation arithmetic. Previous versions showed up to 10 items when entries exceeded 50, with the "X more entries" count referring to 50, not 10. The summary writer now shows up to 50 inline and reports the remaining count based on what was actually shown.

New — --dry-run mode. Runs detection and writes the mapping summary without producing anonymized output files. Useful for validating detection coverage on a sensitive bundle before committing to a full anonymization pass.

New — --verify mode. After anonymization, re-scans the output for sensitive patterns that weren't replaced. A clean verify pass is a useful safety net before sharing externally.

New — --mapping-in for incremental processing. Reuses an existing mapping JSON, guaranteeing that values appearing in multiple bundles (e.g. successive support snapshots from the same cluster) receive identical anonymized labels across runs.

New — structured logging. print() calls replaced with the logging module. -v and -q flags adjust verbosity.

New — basic test suite. test_anonymizer.py covers binary classification, mapping consistency, longest-match resolution, OIDC handling, AWS SigV4, S3 URLs with regional hostnames, safe namespaces, and the verify pass.

Internal — MappingStore.load_existing. Restores counters past the highest existing index so new values cannot collide with previously assigned labels.

v3.1 — Gap fixes from second bundle analysis

Validated against two bundles of different sizes and configurations (different storage providers, authentication modes, and namespace counts).

Gap 1 — FQDN detection generalized to any TLD. The original regex only matched .home, .local, .lab, .corp, .internal, .lan. Environments using other public TLDs were missed. Fixed by generalizing the apps.* pattern to match any TLD, and adding detection for api.xxx OCP API server FQDNs.

Gap 2 — S3 endpoints in Kopia HTTP debug logs. Kopia debug files (kopia_debug_files/*.log) contain full HTTP request dumps with S3 URLs like https://s3.region.provider.net/bucket-name/path. The original script only detected endpoints from endpoint="…" key-value patterns. Added a dedicated URL-based S3 endpoint and bucket detection regex.

Gap 3 — AWS SigV4 Credential= in Authorization headers. Kopia HTTP logs also include Authorization: AWS4-HMAC-SHA256 Credential=ACCESS_KEY/date/region/s3/aws4_request. The access key in this format was not caught by accessKeyID= patterns. Added Credential=([a-zA-Z0-9]{20,})/ detection.

Gap 4 — OIDC authorization codes. OIDC redirect URLs in gateway logs contain authorization codes (code=abc123def456…) that are potentially replayable. Added pattern-based detection and replacement.

Gap 5 — OIDC state tokens. Hundreds to thousands of state=oidc-auth-state-xxx values appear in auth flows. To avoid mega-regex explosion (thousands of literals would kill performance), OIDC codes and states are handled by a dedicated regex pass with dict-callback, not as mega-regex literals. Each token is still individually tracked in the mapping table for reversibility.

Gap 6 — Short namespaces (< 4 chars). Very short namespace names were missed because the word-boundary replacement threshold was set at 4 characters. Short namespaces are now replaced contextually in known patterns: namespace="xxx", "namespace"=>"xxx", /namespaces/xxx/, and appName=xxx. This avoids false positives from substring matches in unrelated text. The OCP cluster name is also auto-extracted from the base domain (e.g. apps.my-cluster.example.com → my-cluster added as cluster_name).

License

Community tool, use it at your own risk.