Skip to content

Latest commit

 

History

History
 
 

sensor

Folders and files

README.md

Maltrail sensor

The Maltrail sensor: packet capture and trail detection, in Rust. Same maltrail.conf, same trails.csv, same event format — so the Maltrail server and web interface need no changes.

Road to becoming Maltrail's default sensor: docs/ROADMAP.md — gated, with verifiable exit criteria and the open blockers stated plainly.

Needs no root: CAP_NET_RAW (plus CAP_NET_ADMIN for promiscuous mode and PACKET_FANOUT) is enough, and -T validates a deployment before you trust it.

# build
cd sensor && cargo build --release

# run (live capture)
sudo ./sensor/target/release/maltrail-sensor

# run (offline replay)
./sensor/target/release/maltrail-sensor -r capture.pcap -c maltrail.conf --offline

Status

  • Corpus: 42 cases, each declaring the detections it must produce, replayed and asserted by tests/replay.rs. Five pin behaviour that deliberately differs from the retired Python sensor, where that sensor was demonstrably wrong — see docs/COMPATIBILITY.md §2.
  • Tests: unit + integration + shared vectors generated by calling the real Python functions + deterministic fuzzing of every parser, run in both the debug and release profiles. Run sh sensor/tools/check.sh for the current count; the suite grows, so it is not quoted here.
  • Live capture and PACKET_FANOUT verified on real hardware: 4 workers in one kernel group, 20,000 packets distributed 5018/5025/5047/4910 and not duplicated (4-worker total identical to the 1-worker baseline).
  • Measured offline full-sensor replay with a real 1.5M-trail set: ~13.8x lower per-packet cost than sensor.py on the reference box (1.2 µs vs 16.7 µs) — the conservative floor of the 14-37x range across the tested systems (docs/REPORT.md §4a) — and 3.4x faster end to end for a 300k-packet run (startup dominates a short replay). Software packet path scales to ~10 Mpps across 16 workers on an 8-core/16-thread laptop.
  • Where sensor.py still wins: startup (0.20 s warm vs 1.18 s) and peak RSS (63 vs 88 MB), because core/trailsbin.py mmaps a prebuilt binary trail store while the sensor parses the CSV on every start. Python's cold start, which builds that store, takes 7.63 s. Closing this is the top next step in docs/REPORT.md §7.1.

See docs/REPORT.md for the measurements and how to reproduce them, and docs/COMPATIBILITY.md for the (short) list of deliberate differences and unimplemented features.

Documentation

document contents
docs/INSTALL.md prerequisites, build, run, configuration, systemd, migration plan, troubleshooting
docs/ARCHITECTURE.md data flow, threading model, module map, hot-path guarantees, reload, metrics
docs/COMPATIBILITY.md what is fully compatible, deliberate differences, not implemented, edge cases
docs/PORTING_MAP.md Python source area → Rust module, function by function
docs/REPORT.md implementation report: changed files, test/parity/benchmark results, limitations, next steps

Layout

sensor/
  src/
    main.rs          CLI, startup, diagnostics, worker spawn, reload + metrics threads
    config.rs        maltrail.conf parser (port of read_config)
    settings.rs      compiled regexes / matchers
    settings_gen.rs  GENERATED from core/settings.py by tools/gen_settings.py
    pyre.rs          Python `re` compatibility (re.escape, \Z, literal braces, group rules)
    addr.rs          native IP type + Maltrail's exact address rendering
    smallstr.rs      fixed-capacity stack string (allocation-free rendering)
    trails/          batched CSV loader (parallel parse), string + native lookup tables, wildcard regex
    whitelist.rs     WHITELIST / WHITELIST_RANGES
    packet/          DLT/VLAN offsets, offset learner, IP/TCP/UDP/ICMP
    protocols/       dns.rs http.rs tls.rs quic.rs
    heuristics/      scan.rs dns_exhaustion.rs nxdomain.rs
    process.rs       the port of _process_packet() and _check_domain()
    state.rs         per-worker state (caches, burst suppression, accumulators)
    event.rs         event tuple, safe_value(), log-line + repr rendering
    output.rs        log file, LOG_SERVER, CEF, Logstash, condensing, throttling
    ignore.rs        IGNORE_EVENTS + IGNORE_EVENTS_REGEX
    capture/         libpcap handles, PACKET_FANOUT
    worker.rs        run-to-completion capture loop
    metrics.rs       worker-local counters + lock-free aggregation
    throttle.rs      event-log throttling (summarize / legacy / off)
    meta.rs          condensed observable store (meta.sqlite), port of core/meta.py
    trailupdate.rs   drives core/update.py via tools/update_trails.py
    selftest.rs      -T preflight
    stats.rs         Prometheus endpoint (STATS_ADDRESS)
    colorized.rs     port of core/colorized.py (--console output)
    fasthash.rs      FxHash for integer-keyed maps (SipHash kept for wire-derived keys)
    testkit.rs       in-process harness for tests, benches and fuzz targets
  tests/             18 integration suites (detection, replay, trails, vectors, meta,
                     loader_parity, multi_worker_parity, fail_closed, lifecycle, ...)
  tests/corpus/      generated replay corpus (39 pcaps + fixture trails + manifest)
  tests/vectors/     test vectors generated by calling the real Python functions
  benches/hotpath.rs microbench / packet-path / replay / worker-scaling benchmarks
  tools/             gen_settings.py gen_vectors.py gen_ja_vectors.py gen_corpus.py
                     check_trails.py check_version.py update_trails.py dump_trails.py
                     fanout_check.py adversarial_traffic.py check.sh

Everything that must pass

sh sensor/tools/check.sh

which regenerates the Python-derived constants/vectors/corpus, then runs cargo fmt --check, cargo clippy -D warnings, the full test suite in both profiles, and a release build.

Separately, once per deployment (needs privileges):

sudo python3 sensor/tools/fanout_check.py --interface eth0 --workers 4

Benchmarks

cargo bench --bench hotpath                                    # staged microbenchmarks

Numbers are reported in three clearly separated categories — isolated microbenchmarks, full packet path, and pcap replay — plus worker scaling, so a microbenchmark can never be mistaken for sensor throughput.