[akash-kr]

Summary

Adds an opt-in setting, SCRAPER_BOTWALL_LOG_RETENTION_DAYS, that keeps the botwall audit logs to the last N days. The default is 0, which keeps everything exactly as today.

Background

botwall_events.jsonl, botwall_excluded.jsonl and botwall_excluded.log get one line per scrape event or exclusion and are never trimmed. On OwnFeed's server they hold every event since July (2.4 MB, 9,000+ lines). Long-running deployments have to rotate them by hand or let them grow forever.

What changed

• A new setting, read with the existing _int_env and documented in the config comment block, the README Tunables list and the CHANGELOG (Unreleased).
• When it is above 0, _log_event and _log_exclusion trim all three files after appending, at most once an hour per process. A trim keeps:
  • lines whose timestamp is newer than the cutoff (JSON ts, or the first token of a .log line);
  • lines with no readable timestamp.
• The rewrite goes to a temp file in the same folder, keeps the original file's permissions, and swaps in with os.replace. A file with nothing to drop isn't rewritten.
• A new _LOG_LOCK covers the appends and the rewrite, so writes from other threads in the same process aren't lost.
• A trim failure only logs botwall: log trim failed for <path>: ..., and scraping continues.
• botwall_db.json is never touched.

What correct output looks like

With =5, the first write in a process (and the first after each hour) leaves each file with only lines from the last 5 days, plus lines whose timestamp can't be read. The file mode is unchanged, no *.tmp.* files are left behind, and writes within the same hour don't rewrite the files.

Default unchanged

Unset or 0: _maybe_trim_logs returns at once. Log contents, formats, paths, signatures and return types are identical. test_default_unset_keeps_everything pins this.

Tests

tests/test_log_retention.py:

• unset: nothing is trimmed;
• 5: old lines are dropped in all three files, recent and unreadable lines are kept, the file mode is kept, and no temp file is left;
• a second write within the hour doesn't rewrite the file.

Full suite: 117 passed, 6 skipped (main: 114 passed).

Known limit

If another process appends to the same state folder during a rewrite, that line can be lost (noted in a shortcut: comment). A file lock would fix it if several processes ever share one state folder.

After merge

Release a new version. OwnFeed then bumps switchback>= and sets SCRAPER_BOTWALL_LOG_RETENTION_DAYS=5.