Skip to content

test: expand trace redaction regression coverage - #34

Merged
utsab345 merged 1 commit into
utsab345:mainfrom
how2how2how2-arch:test/redaction-regression-coverage
Oct 24, 2026
Merged

utsab345 merged 1 commit into
utsab345:mainfrom
how2how2how2-arch:test/redaction-regression-coverage

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Fixes #29

Summary

Expands tests/unit/test_redaction.py to cover the areas #29 names — credential formats, nested structures, exception messages, and values that must survive untouched — and adds the first direct tests for prepare_trace_for_persistence, the function that runs before persistence and previously had none.

No production code is changed, and no stronger guarantee than the README's existing "best-effort" is claimed.

Changes

Exception messages and other shapes the existing tests did not reach

  • a credential in an exception message, and inside a multi-line traceback (the traceback's own text is asserted to survive);
  • Authorization: Bearer …, asserted to redact exactly once (the two overlapping patterns must not double-substitute);
  • a dotenv-style OPENAI_API_KEY=sk-… line;
  • credentials in string items of a list, asserting the per-item finding path (/log/1);
  • ordinary values preserved exactly: int, float, bool, None, empty string, Devanagari text, and a literal [REDACTED];
  • an empty payload produces no findings.

prepare_trace_for_persistence (previously untested)

  • event-level: sensitive key redacted, sibling value preserved, redactions list written, manifest entry carries (event_id, path, reason), and no raw secret in the dumped payload;
  • a credential inside a manifest failure message, recorded against MANIFEST_REDACTION_ID;
  • appending to an event's existing redactions list rather than replacing it;
  • RedactionError raised for a credential in a protected structural field;
  • the input trace is not mutated;
  • stability across a second pass.

Regression pin for formats outside the pattern set — a parametrized test records what is not redacted today, so adding a pattern for one of them is a deliberate change instead of an unnoticed one.

Behaviour observed while writing these tests

Reporting rather than fixing, since a fix would exceed a test-coverage PR:

  1. A bare {"token": …} key is not redacted. SENSITIVE_KEY_NORMALIZED holds accesstoken, sessiontoken, refreshtoken, authtoken and bearertoken, but not token itself, so {"token": "<opaque bearer>"} passes through unless the value also matches a credential pattern. This is the one entry in the pinned list that may be worth changing; the smallest fix is adding token to that set, though it would also redact non-secret uses of the key, so it is a judgement call for a maintainer.
  2. Authorization: Basic … is not covered by the pattern set (only Bearer).
  3. Re-preparing an already prepared trace returns an empty redaction manifest. Trace and manifest dumps are stable across a second pass, but entries are derived from that pass's findings, and an already-redacted payload yields none. The event's own redactions list still records the path, so nothing is lost from a bundle written once, but a second pass over a loaded trace would report no redactions. The test pins this so changing it is deliberate.

Also not covered, and out of scope for "supported credential formats": AWS access keys, Slack (xoxb-), Google (AIza), GitLab (glpat-), and PEM private-key blocks.

Testing

uv run pytest                                698 passed in 25.08s
uv run pytest tests/unit/test_redaction.py    32 passed in 0.34s
uv run ruff check .                          All checks passed!
uv run ruff format --check .                 196 files already formatted
uv run mypy                                  Success: no issues found in 132 source files

All fixtures use synthetic values only. Tests are deterministic and offline — no network access, no real credentials.

Checklist

  • Tests added/updated
  • pytest passes
  • ruff check passes
  • ruff format check passes
  • mypy passes
  • docs updated if necessary — not necessary: no behaviour change, and README's best-effort wording already holds
  • no secrets/sensitive traces included

Cover the gaps named in utsab345#29 that the existing tests did not reach:

- credentials in exception messages, including a multi-line traceback and a
  dotenv-style KEY=value line;
- string items inside lists, with per-item finding paths;
- the Authorization: Bearer header form, asserted to redact exactly once;
- ordinary values (ints, floats, bools, None, empty strings, Devanagari text,
  and a literal [REDACTED]) preserved untouched.

Add direct coverage for prepare_trace_for_persistence, which had none: event
redaction entries and their JSON-pointer paths, a credential inside a
manifest failure message recorded against the manifest sentinel id, appending
to an existing event redactions list, rejection of a credential in a protected
structural field, non-mutation of the input trace, and stability across a
second pass.

Record the formats the current pattern set does not redact as a parametrized
regression pin, so adding a pattern for one of them is a deliberate change
rather than an unnoticed one. README already documents redaction as best
effort, so this claims nothing stronger.

No production code changed.
@utsab345
utsab345 merged commit cca9919 into utsab345:main Oct 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: expand trace redaction regression coverage

2 participants