Skip to content

Overhaul of structured logging - #628

Open
glaslos wants to merge 8 commits into
mainfrom
struc_log_overhaul
Open

glaslos wants to merge 8 commits into
mainfrom
struc_log_overhaul

Conversation

@glaslos

@glaslos glaslos commented Sep 5, 2026

Copy link
Copy Markdown
Member

Structured attack-event logging (schema_version 1) across JSON, SQLite, syslog, HPFriends, and TAXII sinks
Attack events now include event_time distinct from session start; syslog receives attack JSON explicitly
Breaking: HPFriends/JSON attack payloads use session_id/protocol/flat endpoints instead of id/data_type/remote tuples

@glaslos

glaslos commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

@t3chn0m4g3 could you give this a spin?

glaslos and others added 3 commits September 5, 2026 12:18
Resolve bacnet/ftp conflicts by keeping main's BACnet/IP decode and
FTP command pipeline fixes while using session.log_event.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ethan-thomason

ethan-thomason commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Ran the full suite on this branch: 224 passed, 3 skipped, no failures (Python 3.14). No regressions against main.

Worth flagging that this does more than the title suggests. Replacing datetime.utcnow() with datetime.now(timezone.utc) removes an API that's scheduled for removal, and it makes every logged timestamp timezone-aware rather than a naive value that happens to be UTC. Combined with _to_iso() attaching UTC when tzinfo is missing, the whole path from session start to serialized event is now unambiguous. That matters for anyone correlating events across sensors in different regions, or joining honeypot events against external enrichment data.

One suggestion on the v1 schema. sensorid identifies which sensor produced an event, but nothing in the envelope records what that sensor was running — no template, no version or commit.

Concrete example: I run a few honeypots and recently discovered one had been running five-month-old code. I couldn't tell that from the logs, because nothing in them records the deployed version, and the behavioral differences I was seeing were ambiguous between "different deployment" and "different code." With a single sensor that's tolerable. Across several it's the difference between a comparable dataset and an uninterpretable one.

Adding template and a version/commit field to the envelope would make events self-describing, and it's much cheaper at v1 than as a later migration. It also connects nicely to core/templates.py in #632. Once template metadata is a first-class concept, carrying it into the event is a small step.

Unrelated minor thing: test_ipmi_server.py fails with FileNotFoundError rather than skipping when ipmitool isn't installed. Cost me a few minutes assuming it was the branch.

Slightly tangential, but related to the schema shape: #444 has been open since 2019. A user is confused by "request": null, "response": null on every s7comm event. Looking at their output, those were all NEW_CONNECTION / CONNECTION_LOST events, which correctly have no request or response. Not a bug, but the shape made it look like one, and the thread never landed on that. (The s7comm changes here are a refactor of calls that already passed request/response, so this isn't something the branch introduced or breaks.)

normalize_event still emits request, response, and error unconditionally, so lifecycle events carry three nulls. Worth considering omitting keys that are None? Smaller events, and lifecycle events stop looking broken. The tradeoff is that consumers can no longer rely on key presence, which cuts both ways so it depends whether v1 is meant to be a fixed envelope or a sparse one. Either is defensible, but it's much cheaper to settle now than later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants