diff --git a/README.md b/README.md index 1f6050a2..72a284da 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ The library contains the next components: * **Parsers**: parse the logs received from the reader. * **Detectors**: return alerts if anomalies are detected. * **Alert Aggregation**: aggregate the alerts produced by the detectors. -* **Schemas**: standard data classes use in DetectMate. +* **Schemas**: standard data classes used in DetectMate. ``` +--------+ +-----------+ +-------------------+ | Parser | --> | Detector | -> | Alert Aggregation | diff --git a/docs/advanced/overall_architecture.md b/docs/advanced/overall_architecture.md index 44320fff..71e09028 100644 --- a/docs/advanced/overall_architecture.md +++ b/docs/advanced/overall_architecture.md @@ -25,7 +25,7 @@ Each arrow represents a stream of Schema objects. Components are designed to run ## Components architecture -All components inherit from a `CoreComponent` class. This class provides all the essential functionality required for DetectMate to operate (see UML diagram below). Every `Detector` must inherit from `CoreDetector`, every `AlertAggregator` must inherit from `CoreAlertAggregation` and every `Parser` must inherit from `CoreParser` to ensure compatibility with DetectMate. +All components inherit from a `CoreComponent` class. This class provides all the essential functionality required for DetectMate to operate (see UML diagram below). Every `Detector` must inherit from `CoreDetector`, every `AlertAggregator` must inherit from `CoreAlertAggregator` and every `Parser` must inherit from `CoreParser` to ensure compatibility with DetectMate. Each component's arguments must be stored in its corresponding configuration class. These config classes follow the same design pattern as their components and must inherit from `CoreConfig`. @@ -38,5 +38,3 @@ Each Core* base class exposes a small, stable API that implementations must impl ```python --8<-- "docs/examples/others/components_methods.py:read" ``` - -Go back [Index](../index.md) diff --git a/docs/alert_aggregator.md b/docs/alert_aggregator.md index 300d2a89..97fa7636 100644 --- a/docs/alert_aggregator.md +++ b/docs/alert_aggregator.md @@ -1,42 +1,69 @@ -# Components: Alert Aggregation +# Alert Aggregation -Alert aggregation aggregates alerts from detectors. +Alert aggregators combine the alerts produced by detectors into aggregated records. | | Schema | Description | |------------|----------------------------|--------------------| | **Input** | [DetectorSchema](schemas.md) | Alerts from detectors | | **Output** | [AggregateSchema](schemas.md) | Aggregated alerts | -This document explains expected APIs, how to implement a parser, testing tips and common pitfalls. +This document explains the expected API and how to implement an alert aggregator. ## Overview -- Alert aggregation must inherit from `CoreAlertAggregation` and provide a `aggregate_alerts()` implementation. -- `CoreParser.run()` handles lifecycle and calls `aggregate_alerts()` for each input; implement pure alert aggregation logic inside `aggregate_alerts()` where possible. -- Use a typed `Config` class (subclass of `CoreAlertAggregationConfig`) to hold runtime parameters. +- Alert aggregators must inherit from `CoreAlertAggregator` and provide an `aggregate_alerts()` implementation. +- `CoreAlertAggregator.run()` handles the lifecycle and calls `aggregate_alerts()`. Aggregators use a + sliding window (a `WINDOW` [data buffer](auxiliar/input_buffer.md)) of `buffer_size` alerts, so + `aggregate_alerts()` always receives a **list**: the most recent `buffer_size` alerts. It is first + called once the window is full, and then once for every new alert, so consecutive windows overlap. + With the default `buffer_size` of 1, it is called for every alert with a list of one. +- Use a typed `Config` class (subclass of `CoreAlertAggregatorConfig`) to hold runtime parameters. -## CoreParser -- minimal API - -Recommended signatures and behavior: +## CoreAlertAggregator: minimal API ```python -class CoreAlertAggregation: - def aggregate_alerts( +class CoreAlertAggregator(CoreComponent): + def __init__( + self, + name: str = "CoreAlertAggregator", + buffer_mode: BufferMode = BufferMode.WINDOW, + buffer_size: int | None = 1, + config: CoreAlertAggregatorConfig | dict[str, Any] | None = CoreAlertAggregatorConfig(), + ) -> None: + """buffer_mode and buffer_size set which alerts aggregate_alerts() receives per call.""" + + def aggregate_alerts( self, input_: list[DetectorSchema] | DetectorSchema, output_: AggregateSchema, ) -> bool: - return True + """Implement the aggregation here. + - Return True to emit output_ as the aggregated record. + - Return False to emit nothing: process() then returns None. + """ - @override - def train( - self, input_: DetectorSchema | list[DetectorSchema] - ) -> None: - pass + def train(self, input_: DetectorSchema | list[DetectorSchema]) -> None: + """Optional: learn from alerts. Can be a no-op.""" ``` -## Available alert aggregation +## What `run()` fills in for you -- [Basic Concat](alert_aggregators/basic_concatenation.md) +Before calling `aggregate_alerts()`, `CoreAlertAggregator.run()` collects these fields from the +alerts in the window: -Go back to [Index](index.md) +- `detectorIDs`, `detectorTypes`, `alertIDs`: one entry per alert +- `logIDs`, `extractedTimestamps`: the entries of all alerts, combined into one list + +After `aggregate_alerts()` returns `True`, `run()` also sets `outputTimestamp`. So +`aggregate_alerts()` only has to set `description` and `alertsObtain`. + +!!! warning "The alerts' details are not carried over" + `run()` copies only IDs and timestamps. The `description` and `alertsObtain` of the incoming + alerts, which hold the actual alert messages, are **not** copied into the aggregate, and + `AggregateSchema` has no `score` field at all. If your aggregator needs this information, read it + from `input_` in `aggregate_alerts()` and write it into `output_["description"]` or + `output_["alertsObtain"]` yourself. + +## Available alert aggregators + +- [Basic Concat](alert_aggregators/basic_concatenation.md) diff --git a/docs/auxiliar/input_buffer.md b/docs/auxiliar/input_buffer.md index dcd7f17f..09a53397 100644 --- a/docs/auxiliar/input_buffer.md +++ b/docs/auxiliar/input_buffer.md @@ -1,6 +1,6 @@ # Data Buffer -The data buffer is an auxiliar methods that can be use in all the components. It takes the stream data and formated to the specifications given. +The data buffer is an auxiliary method that can be used in all the components. It takes the stream data and formats it to the given specifications. It has different configuration states to configure its behaviour. @@ -31,5 +31,3 @@ Code examples to show the behaviour of the **DataBuffer** class. ```python --8<-- "docs/examples/others/data_buffer.py:example_3" ``` - -Go back [Index](../index.md) diff --git a/docs/auxiliar/persistency.md b/docs/auxiliar/persistency.md index 097236e6..808af5c0 100644 --- a/docs/auxiliar/persistency.md +++ b/docs/auxiliar/persistency.md @@ -31,12 +31,12 @@ Two families ship today: raw rows. Very storage heavy and *not recommended* for production-ready detectors. - **Tracker backends** (`EventStabilityTracker`) keep only derived features (e.g. "this variable has been constant for the last 10k events") that are relevant for the detector. Use these - when you only need a summary or a subset of the log's information, not the raw history -- they cost a fraction of + when you only need a summary or a subset of the log's information, not the raw history: they cost a fraction of the memory. All backends implement the same four-method contract: `add_data`, `get_data`, `dump`, `load`. That contract is what `EventPersistency` and -`PersistencySaver` rely on -- anything you add later only has to follow it. +`PersistencySaver` rely on, so anything you add later only has to follow it. ### 3. Saver lifecycle (`PersistencySaver`) @@ -44,7 +44,7 @@ All backends implement the same four-method contract: `add_data`, `get_data`, state has to be written somewhere. `PersistencySaver` wraps an `EventPersistency` and: -- writes to disk (or any `fsspec` URI) on two triggers -- a wall-clock interval +- writes to disk (or any `fsspec` URI) on two triggers: a wall-clock interval and an event-count threshold; - optionally `auto_load`s previously saved state during construction; - exposes `start()` / `stop()` so the background timer can be torn down @@ -106,7 +106,7 @@ ep[event_id] # alias for get_event_data |---|---| | `persistency.EventStabilityTracker` | You only care about how variables behave over time (`STATIC` / `STABLE` / `UNSTABLE` / `RANDOM`). Cheapest memory footprint. | -All three are re-exported from the top of the package -- `persistency.X` is the +All three are re-exported from the top of the package: `persistency.X` is the canonical import; the deeply nested submodules are an implementation detail. ### Persisting to disk @@ -129,7 +129,7 @@ saver.stop() # final flush, stops the background timer `PersistencySaver.save()` is thread-safe, and `stop()` is idempotent. The two save triggers (`save_interval_seconds` and `events_until_save`) are -independent -- whichever fires first wins. +independent: whichever fires first wins. If writing to storage fails (unwritable path, full disk, lost credentials), `save()` and `stop()` raise `persistency.PersistencySaveError`, and @@ -149,13 +149,13 @@ saver = persistency.PersistencySaver( ``` If `auto_load=True` and no saved state exists, the constructor raises -`persistency.PersistencyLoadError` immediately -- fail-fast rather than +`persistency.PersistencyLoadError` immediately: it fails fast rather than silently starting empty. #### Exporting and importing state on demand -For one-shot transfers -- e.g. moving trained state to a new environment, or -taking a manual snapshot -- use the standalone functions directly: +For one-shot transfers (e.g. moving trained state to a new environment, or +taking a manual snapshot), use the standalone functions directly: ```python from detectmatelibrary.utils import persistency @@ -163,7 +163,7 @@ from detectmatelibrary.utils import persistency # Export to a file URI persistency.save(ep, "./snapshots/trained-state") -# Export to bytes (no disk I/O -- useful when sending state over a network API) +# Export to bytes (no disk I/O; useful when sending state over a network API) data: bytes = persistency.save(ep) # Import from a file URI @@ -187,7 +187,7 @@ when a saver is active. #### Detector-level export and import When working through a detector (the typical path for DetectMateService), use -the methods on the detector object directly -- no need to access +the methods on the detector object directly, with no need to access `EventPersistency` internals: ```python diff --git a/docs/contribution.md b/docs/contribution.md index d517c2c8..64b81925 100644 --- a/docs/contribution.md +++ b/docs/contribution.md @@ -12,7 +12,7 @@ at least the following information in a bug report: 1. Description of the bug. Describe the problem clearly. 2. Steps to reproduce. With the following configuration, go to.., click.., see error -3. Expected behavoir. What should happen? +3. Expected behavior. What should happen? 4. Environment. What was the environment for the test(version, browser, etc..) *Please don't include any private/sensitive information in your issue! For reporting security-related issues, see [SECURITY.md](https://github.com/ait-detectmate/DetectMateLibrary/blob/main/SECURITY.md)* @@ -39,7 +39,7 @@ git clone -b development git@github.com:YOURUSERNAME/DetectMateLibrary.git ### 3. Create a feature branch -Every single workpackage should be developed in it's own feature-branch. Use a name that describes the feature: +Every single workpackage should be developed in its own feature-branch. Use a name that describes the feature: ```bash cd DetectMateLibrary @@ -48,7 +48,7 @@ git checkout -b feature-some_important_work ### 4. Develop your feature and improvements in the feature-branch -Please make sure that you commit only improvements that are related to the workpage you created the feature-branch for. +Please make sure that you commit only improvements that are related to the workpackage you created the feature-branch for. ### 5. Fetch and merge from the upstream @@ -103,13 +103,13 @@ git rebase -i HEAD~2 Delete your local feature-branch after the pull-request was merged into the development branch. -### 8. Update your local main branch +### 8. Update your local development branch Update your local development branch: ```bash git fetch upstream development -git checkout -b development +git checkout development git rebase upstream/development ``` @@ -117,9 +117,9 @@ Additional infos: - [https://www.atlassian.com/git/tutorials/merging-vs-rebasing](https://www.atlassian.com/git/tutorials/merging-vs-rebasing) -### 9. Update your main branch in your github-repository +### 9. Update the development branch in your GitHub repository -Please make sure that you updated your local development branch as described in section 8. above. After that push the changes to your github-repository to keep it up2date: +Please make sure that you updated your local development branch as described in section 8. above. After that push the changes to your github-repository to keep it up to date: ```bash git push diff --git a/docs/detectors.md b/docs/detectors.md index cfabb06f..f3d9e5fe 100644 --- a/docs/detectors.md +++ b/docs/detectors.md @@ -10,7 +10,7 @@ Detectors process structured logs from Parsers and emit alerts when anomalies ar This document describes the minimal API, implementation guidance, a short example detector and a unit test pattern. -## CoreDetector -- minimal API +## CoreDetector: minimal API @@ -18,12 +18,23 @@ This document describes the minimal API, implementation guidance, a short exampl class CoreDetectorConfig(CoreConfig): component_type: str = "detectors" method_type: str = "core_detector" - parser: str = "" + parser: str = "PARSER" auto_config: bool = True + events: EventsConfig | dict[str, Any] = {} + global_instances: dict[str, Any] = {} # written as `global` in YAML class CoreDetector(CoreComponent): + def __init__( + self, + name: str = "CoreDetector", + buffer_mode: BufferMode = BufferMode.NO_BUF, + buffer_size: int | None = None, + config: CoreDetectorConfig | dict[str, Any] | None = CoreDetectorConfig(), + ) -> None: + """buffer_mode and buffer_size set how many logs detect() receives per call.""" + def run( self, input_: List[ParserSchema] | ParserSchema, output_: DetectorSchema ) -> bool: @@ -40,7 +51,7 @@ class CoreDetector(CoreComponent): """Empty, can be define in the detector. It trains the detector""" ``` -## Implementing a detector -- example +## Implementing a detector: example Simple detector that raises an alert when a numeric variable exceeds a threshold. @@ -127,7 +138,14 @@ The detectors are numbered from simplest to most complex, and the sidebar lists Every detector page shows a minimal, working configuration file next to its example. The reference below explains the blocks those files use. -When `auto_config` is set to `False`, the detector expects an explicit `events` or `global` block that specifies exactly which variables to monitor. `events`refers to event-specific variables while `global` refers to variables, that are not bound to events (`header_variables`can but don't have to be event bound): +`auto_config` defaults to `True` for detectors, but the configure phase only runs when +`data_use_configure` is set. A detector with an `events` block and no +`data_use_configure` therefore uses that block even if `auto_config` is left out. The +example configurations set `auto_config: false` anyway, to make this explicit. If +`auto_config` is `True` *and* `data_use_configure` is set, the configure phase replaces the +`events` block with its own selection. + +When `auto_config` is set to `False`, the detector expects an explicit `events` or `global` block that specifies exactly which variables to monitor. `events` refers to event-specific variables while `global` refers to variables, that are not bound to events (`header_variables` can but don't have to be event bound): ```yaml detectors: @@ -144,7 +162,7 @@ detectors: - pos: 0 # location of an unnamed variable from the log message name: var1 # name of variable (arbitrary) header_variables: - - pos: level # location of a named variable (defined in log_format of parser) + - pos: Level # name of a field in the parser's log_format (case-sensitive: -> Level) global: # define global instance for new_value_detector similar to "events" global_instance1: # define instance name header_variables: # same logic as header_variables in "events" @@ -154,7 +172,7 @@ detectors: ### Common parameters (all detectors) -There are some parameters, that **every** detector inhertis from `CoreDetectorConfig`/`CoreConfig`/`BasicConfig`, regardless of what it does. The other parameters, that are **specific** for the respective detector, are explained right at the detectors documentation page, later on. +There are some parameters, that **every** detector inherits from `CoreDetectorConfig`/`CoreConfig`/`BasicConfig`, regardless of what it does. The other parameters, that are **specific** for the respective detector, are explained right at the detectors documentation page, later on. ???+ note "Top level" @@ -237,24 +255,50 @@ The two neural detectors ([DeepLog](detectors/deeplog.md), [LogBERT](detectors/l ### Configuration semantics (preliminary) -**`events` key** -- The integer key is the `EventID` (or `event_id`) to monitor (see the [Template Matcher](parsers/template_matcher.md) docs for how the EventID is assigned. +**`events` key**: the key is the `EventID` (or `event_id`) to monitor: an integer as the parser emits it (see the [Template Matcher](parsers/template_matcher.md) docs for how the EventID is assigned), or a name (see [Named EventIDs and variables](#named-eventids-and-variables) below). **`global` key** - This one has a similar functionality as the `events` key but refers to variables, that are not bound to events (thus can only contain `header_variables`). -**`variables[].pos`** -- The 0-indexed position of the `<*>` wildcard in the matched template, counting from left to right starting at 0. For example, given: +**`variables[].pos`**: the 0-indexed position of the `<*>` wildcard in the matched template, counting from left to right starting at 0. For example, given: ```text pid=<*> uid=<*> auid=<*> ses=<*> msg='op=<*> acct=<*> exe=<*> hostname=<*> addr=<*> terminal=<*> res=<*>' ``` -`pos: 0` captures `pid=`, `pos: 6` captures `exe=`, etc. +`pos: 0` captures the value after `pid=` (for example `10125`), `pos: 6` the value after +`exe=` (for example `"/usr/sbin/cron"`), and so on. + +**`header_variables[].pos`**: a named field from the log format string (e.g., `Type`, `Time`, `Content`) rather than a wildcard position. + +### Named EventIDs and variables -**`header_variables[].pos`** -- A named field from the log format string (e.g., `Type`, `Time`, `Content`) rather than a wildcard position. +Instead of numbers, the `events` block can use names: a name for the `EventID`, and +the name of a wildcard for `variables[].pos`. The names come from the +[Template Matcher](parsers/template_matcher.md)'s template file: + +- **Named variables:** write `` instead of `<*>` in a template. A template uses + either named wildcards or `<*>`, not both. This works in `.txt` and `.csv` template files. +- **Named EventIDs:** use a `.csv` template file with an `EventId` column next to the + `EventTemplate` column: + + ```text + --8<-- "docs/examples/data/named_templates.csv" + ``` + +The parser still emits numbers (`EventID` 1 and position 0 for `` in the second +template), so the names must be translated before the detector runs. Do this once, +after creating both components, with `compile_detector_config()`. **Without this step +the detector monitors nothing**; it only logs a warning that the named `EventID` was +never observed in training. + +```python +--8<-- "docs/examples/detectors/named_events.py:example" +``` ### Auto-configuration (optional) -Detectors can optionally support **auto-configuration** -- a process where the detector automatically discovers which variables are worth monitoring, instead of requiring the user to specify them manually. +Detectors can optionally support **auto-configuration**: a process where the detector automatically discovers which variables are worth monitoring, instead of requiring the user to specify them manually. Auto-configuration is controlled by the `auto_config` flag in the pipeline config (e.g. `config/pipeline_config_default.yaml`): @@ -266,7 +310,7 @@ detectors: params: data_use_configure: 1000 # logs used to pick the variables data_use_training: 1000 # logs used to train on them afterwards - # no "events" block needed -- it will be generated automatically + # no "events" block needed: it is generated automatically ``` @@ -274,9 +318,9 @@ detectors: When auto-configuration is enabled, the detector goes through two extra phases before training: -**Phase 1 -- `configure(input_)`**: The detector ingests events into an `EventPersistency` instance that uses a tracker backend to analyze variable behavior -- for example, whether each variable is stable, random, or still has insufficient data. This instance is typically separate from the one used for training, because the configuration phase needs to observe *all* variables to decide which ones are worth monitoring, while training only tracks the variables that were selected as a result. +**Phase 1, `configure(input_)`**: The detector ingests events into an `EventPersistency` instance that uses a tracker backend to analyze variable behavior, for example whether each variable is stable, random, or still has insufficient data. This instance is typically separate from the one used for training, because the configuration phase needs to observe *all* variables to decide which ones are worth monitoring, while training only tracks the variables that were selected as a result. -**Phase 2 -- `set_configuration()`**: After enough data has been ingested, the detector queries the tracker to select variables that meet its criteria (e.g. only stable variables). It then generates a full `events` configuration from those results and updates its own config. At this point `auto_config` is set to `False` in the generated config, since the configuration is now explicit. +**Phase 2, `set_configuration()`**: After enough data has been ingested, the detector queries the tracker to select variables that meet its criteria (e.g. only stable variables). It then generates a full `events` configuration from those results and updates its own config. At this point `auto_config` is set to `False` in the generated config, since the configuration is now explicit. After these two phases, the detector proceeds with the normal `train()` and `detect()` lifecycle using the generated configuration. @@ -300,9 +344,9 @@ the hyperparameter searches of `DrainParser` and the deep-learning detectors. That distinction is visible in the config. A detector's settings live in two blocks: -* **`auto_config_params`** -- inputs *to* the configure phase. They pick which +* **`auto_config_params`**: inputs *to* the configure phase. They pick which variables the phase selects and are read only while `auto_config` is `True`. -* **`params`** -- operational settings, read during training and detection on +* **`params`**: operational settings, read during training and detection on every run. The configure phase writes its results into the top-level `events` block (and, @@ -325,11 +369,11 @@ axes: | `slope_time` | change centroid vs. `slope_threshold` | normalized timestamps | Any subset of the four may be enabled, and any single one may stand alone. The -default -- `index` alone -- is the historical behaviour: each segment's mean rate -of change is compared against its entry in `segment_thresholds` -- four segments -by default, one threshold per segment -- and the segments are **equal-count**: +default (`index` alone) is the historical behaviour: each segment's mean rate +of change is compared against its entry in `segment_thresholds` (four segments +by default, one threshold per segment), and the segments are **equal-count**: each holds the same number of observations, regardless of how much time they -cover. For bursty log sources that is misleading -- a variable that changed +cover. For bursty log sources that is misleading: a variable that changed constantly during a quiet night and then went silent under a flood of daytime traffic looks stable, because the flood supplies enough samples to dominate the later segments. Enabling `time` cuts the same segments at **equal @@ -337,12 +381,12 @@ durations** instead, so each segment covers the same amount of wall-clock time; the detector then needs an event time per record, which it reads from the log's named variables (`logFormatVariables`, i.e. the fields declared in the parser's `log_format`) under the name given by `timestamp_variable`. `slope_index` and -`slope_time` ask a different question -- whether the change centroid sits early -or late in the series -- on the index axis and the time axis respectively. +`slope_time` ask a different question (whether the change centroid sits early +or late in the series), on the index axis and the time axis respectively. These parameters live on every `VariableDetector` subclass (`NewValueDetector`, `NewValueComboDetector`, `ValueRangeDetector`, `CharsetDetector`, `BigramDetector`, …) -and go in the detector's `auto_config_params` block -- they are inputs to the +and go in the detector's `auto_config_params` block: they are inputs to the auto-configuration phase, read only while `auto_config` is `True`, and never consulted at detection time. @@ -388,7 +432,7 @@ requires strictly more than half of them to. Ties resolve to UNSTABLE. That keeps `majority` from ever being more lenient than a coin-flip, and makes it collapse onto `consensus` at one and two enabled -methods -- turning a third method on is the only place the rule starts to matter. +methods. Turning a third method on is the only place the rule starts to matter. **All four methods false is a config error**, rejected by a pydantic validator. It is not a harmless no-op: classification decides `INSUFFICIENT_DATA`, @@ -404,7 +448,7 @@ All of these live in the detector's `auto_config_params` block. | `use_stable_vars` | `bool` | `true` | Include variables classified `STABLE` in the generated configuration. | | `use_static_vars` | `bool` | `true` | Include variables classified `STATIC`. Defaults to `false` on `NewValueComboDetector`. | | `classification` | `ClassificationMethods` | see below | Which classification methods run and how their verdicts combine. | -| `timestamp_variable` | `str \| null` | `null` | Name of the field in `logFormatVariables` holding the record's event time. Required for `time` and `slope_time` to have any effect. Only named log-format fields are consulted -- never the positional `variables` list. | +| `timestamp_variable` | `str \| null` | `null` | Name of the field in `logFormatVariables` holding the record's event time. Required for `time` and `slope_time` to have any effect. Only named log-format fields are consulted, never the positional `variables` list. | | `timestamp_format` | `str \| null` | `null` | Explicit [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) pattern for parsing that field. When unset, `TimeFormatHandler` auto-detects the format (ISO 8601, Apache, syslog, numeric epoch seconds/milliseconds, and other common layouts). | Set `timestamp_format` when the source uses a layout the auto-detection does not @@ -441,13 +485,13 @@ Time-aware classification is best-effort and never fails a run: falls back to the index axis. * If timestamps stop lining up with the recorded observations, or the observed time span is zero, or they arrive out of order, `time` silently reuses the equal-index - cuts, and `slope_time` computes its centroid on the index axis instead -- it + cuts, and `slope_time` computes its centroid on the index axis instead, so it degrades to `slope_index`. * Under `majority`, a method that fell back still casts its own vote. If both methods of a pair (`slope_index`/`slope_time` or `index`/`time`) are enabled and timestamps are unusable, they compute the same verdict, which then counts twice. -In every fallback case classification still runs and produces a result -- only the +In every fallback case classification still runs and produces a result; only the axis behind it changes back to index. A segment with no observations in it is *not* a fallback: it scores a mean of 0.0, @@ -455,7 +499,7 @@ because nothing observed means nothing changed. Once the segment floor above is met, equal-index cuts never leave a segment empty; equal-duration cuts of a bursty variable still do, routinely, so `time` on its own is lenient towards a burst of churn followed by silence. Enable `index` and `time` together when that leniency -matters -- the index pass keeps every segment populated. +matters: the index pass keeps every segment populated. ### Saving state (persist) @@ -478,7 +522,7 @@ detectors: ... ``` -All fields are optional -- `persist: {}` uses all defaults. Omitting `persist:` entirely +All fields are optional: `persist: {}` uses all defaults. Omitting `persist:` entirely disables saving (backward compatible). The other detectors (Random, Rule, DeepLog, LogBERT) have no state to save: a `persist:` block on them fails at config load with `persist: Extra inputs are not permitted`. @@ -490,7 +534,7 @@ named `NewValueDetector` writes to `./state/NewValueDetector/`. The default `path` is CWD-relative. systemd services usually run with CWD `/`, so `./state` would resolve to `/state` (wrong location, needs root). To avoid -this, set `StateDirectory=` in your unit file -- systemd creates `/var/lib/` +this, set `StateDirectory=` in your unit file: systemd creates `/var/lib/` with the right ownership and exports `$STATE_DIRECTORY`, which the default `path` reads automatically. No explicit `path:` needed: @@ -514,7 +558,7 @@ Setting `path:` explicitly (e.g. an `s3://` URL) always overrides `$STATE_DIRECT #### Storage options examples -**Local filesystem** -- no `storage_options` needed: +**Local filesystem** (no `storage_options` needed): ```yaml persist: @@ -564,5 +608,5 @@ persist: ``` In practice, credentials are usually supplied via environment variables -(`AWS_ACCESS_KEY_ID`, etc.) or instance roles -- in which case `storage_options` +(`AWS_ACCESS_KEY_ID`, etc.) or instance roles, in which case `storage_options` stays empty or is omitted. diff --git a/docs/detectors/combo.md b/docs/detectors/combo.md index b88171a1..00b22642 100644 --- a/docs/detectors/combo.md +++ b/docs/detectors/combo.md @@ -1,6 +1,6 @@ # New Value Combo Detector -The New Value Combo Detector raises alerts when previously unseen combinations of values appear in configured fields (for example new user names, IP addresses, or process names). It is useful to detect novelty, configuration drift, or the appearance of new actors in the environment. +The New Value Combo Detector raises alerts when previously unseen combinations of values appear in configured fields (for example a known user logging in from a known IP address they have never used before). It is useful to detect unusual pairings of otherwise familiar values, which the [New Value Detector](new_value.md) would miss. ## In/out diff --git a/docs/detectors/ecvc_detector.md b/docs/detectors/ecvc_detector.md index 17e53b61..af292b69 100644 --- a/docs/detectors/ecvc_detector.md +++ b/docs/detectors/ecvc_detector.md @@ -21,7 +21,7 @@ Input and output schemas in the pipeline A count vector is formed by counting the number of appearances of each event ID in a sequence of a specific window size. -Count vectors learned during training are stored via [persistency](../auxiliar/persistency.md), so a trained model can be saved and restored with a `persist:` block. A count vector is only comparable within the window it was counted over, so restoring state at a different `window_size` logs a warning -- the restored vectors cannot match and every window would alert. +Count vectors learned during training are stored via [persistency](../auxiliar/persistency.md), so a trained model can be saved and restored with a `persist:` block. A count vector is only comparable within the window it was counted over, so restoring state at a different `window_size` logs a warning, because the restored vectors cannot match and every window would alert. ## Example diff --git a/docs/detectors/event_sequence.md b/docs/detectors/event_sequence.md index 1fade4b2..a8751bf4 100644 --- a/docs/detectors/event_sequence.md +++ b/docs/detectors/event_sequence.md @@ -21,7 +21,7 @@ Input and output schemas in the pipeline The detector slides a window of `fixed_window_size` event IDs over the log stream. During training every full window is stored as a known sequence; during detection a window whose exact sequence is not in that set is reported as an anomaly. -Because a novel event stays inside the window for `fixed_window_size` steps, a single unexpected event produces up to `fixed_window_size` consecutive alerts -- one per window it invalidates. This is intentional: each of those windows is a distinct sequence that was never trained. +Because a novel event stays inside the window for `fixed_window_size` steps, a single unexpected event produces up to `fixed_window_size` consecutive alerts, one per window it invalidates. This is intentional: each of those windows is a distinct sequence that was never trained. Sequences are stored as fixed-length n-grams, so a persisted model is only meaningful at the length it was trained with. When state is restored via [persistency](../auxiliar/persistency.md) at a different `fixed_window_size`, the detector logs a warning, adopts the persisted length, and skips auto-configuration. @@ -31,7 +31,7 @@ With `auto_config: True` the detector spends the configure phase feeding one win Candidates whose window never filled during the configure phase are skipped, so a short configure phase simply narrows the choice. -If no candidate is stable, no window length is meaningful for this log stream. Rather than fall back to an arbitrary length and alert on nearly every window, the detector generates an empty configuration: **no instance of the detector is created**, `fixed_window_size` stays `None`, and it neither trains nor alerts for the rest of the run. A warning names the range that was searched. The same applies to `auto_config: False` without a `fixed_window_size` -- the detector stays inert. +If no candidate is stable, no window length is meaningful for this log stream. Rather than fall back to an arbitrary length and alert on nearly every window, the detector generates an empty configuration: **no instance of the detector is created**, `fixed_window_size` stays `None`, and it neither trains nor alerts for the rest of the run. A warning names the range that was searched. The same applies to `auto_config: False` without a `fixed_window_size`: the detector stays inert. Longer windows are more specific and therefore alert more readily; if the auto-configured length is too sensitive, narrow the range or set `fixed_window_size` explicitly. diff --git a/docs/detectors/scvs_detector.md b/docs/detectors/scvs_detector.md index f3ee9609..ce75278a 100644 --- a/docs/detectors/scvs_detector.md +++ b/docs/detectors/scvs_detector.md @@ -21,7 +21,7 @@ Input and output schemas in the pipeline A count vector is formed by counting the number of appearances of each event ID in a sequence of a specific window size. -Count vectors learned during training are stored via [persistency](../auxiliar/persistency.md), so a trained model can be saved and restored with a `persist:` block. A count vector is only comparable within the window it was counted over, so restoring state at a different `window_size` logs a warning -- the restored vectors cannot match and every window would alert. +Count vectors learned during training are stored via [persistency](../auxiliar/persistency.md), so a trained model can be saved and restored with a `persist:` block. A count vector is only comparable within the window it was counted over, so restoring state at a different `window_size` logs a warning, because the restored vectors cannot match and every window would alert. ## Example diff --git a/docs/detectors/value_range.md b/docs/detectors/value_range.md index 44f24a6d..21220d98 100644 --- a/docs/detectors/value_range.md +++ b/docs/detectors/value_range.md @@ -1,6 +1,6 @@ # Value Range Detector -The Value Range Detector raises alerts when numerical values outside of known ranges appear in configured fields. It is useful to detect unexpected changes, configuration drift, or the appearance of new actors in the environment. +The Value Range Detector raises alerts when numerical values outside of known ranges appear in configured fields. It is useful to detect unexpected changes in quantities such as sizes, counts, or durations. ## In/out @@ -19,7 +19,7 @@ Input and output schemas in the pipeline ## Description -This detector maintains a lightweight set of observed values per monitored field and emits an alert when a value outside the learned range is seen (subject to configuration). +This detector keeps only the minimum and maximum value seen during training for each monitored field, so its memory use stays constant however many values it sees. It emits an alert when a value falls outside that range (subject to configuration). ## Example diff --git a/docs/development.md b/docs/development.md index ea498c8b..97ba47ce 100644 --- a/docs/development.md +++ b/docs/development.md @@ -74,7 +74,7 @@ def configure(self, input_): ``` The `set_configuration()` method queries the tracker results and writes the -final `events` block. It touches nothing else on the config -- everything the +final `events` block. It touches nothing else on the config: everything the operator set under `params` or `auto_config_params` must survive untouched, so `set_configuration` never rebuilds the config from scratch: @@ -92,8 +92,8 @@ def set_configuration(self): ### Why `auto_config_params` lives on `BasicConfig` Both `auto_config` and `Component.configure()` are declared on the shared base, -so `auto_config_params` is declared there too -- on `BasicConfig`, beside -`auto_config` -- rather than on the detector config alone. Detectors are the only +so `auto_config_params` is declared there too (on `BasicConfig`, beside +`auto_config`) rather than on the detector config alone. Detectors are the only component type with a real configure phase today, so they are the only ones that narrow the block with fields; parsers and alert aggregators inherit it empty, and an empty block is omitted from the serialized config, so their YAML is unaffected. @@ -109,7 +109,7 @@ files under `docs/examples/`, mirrored by category (`docs/examples/parsers/`, `docs/examples/detectors/`), and are pulled into the Markdown pages via [`pymdownx.snippets`](https://facelessuser.github.io/pymdown-extensions/extensions/snippets/). This way every snippet in the docs is an actual `.py` file that gets executed by -the test suite, so a broken example fails CI instead of silently shipping. +the test suite in CI, so a broken example fails CI instead of silently shipping. **1. Add the snippet file.** Put your example under `docs/examples//`. By convention the filename matches its documentation page (`charset.md` → @@ -135,18 +135,19 @@ from detectmatelibrary.parsers.logbatcher import LogBatcherParser, LogBatcherPar You can also include the whole file by dropping the `:section` suffix (`--8<-- "docs/examples/parsers/template_tree_matcher.py"`), but section markers are the norm. Because `check_paths: true` is set, the build aborts if the file or -marker doesn't exist -- a missing snippet is caught at build time. +marker doesn't exist, so a missing snippet is caught at build time. **3. Make sure it's testable.** The test (`tests/test_docs/test_doc_examples.py`) globs every `.py` under `docs/examples/` and runs each one as a script via `runpy.run_path(..., run_name="__main__")`. There is no plugin and no assert requirement: a snippet passes as long as it runs standalone without raising. If your example needs something unavailable in CI (e.g. an API key), comment out -those calls rather than letting them fail. Run the snippet tests together with -the rest of the suite: +those calls rather than letting them fail. The test is marked `ignored`, so a +plain `pytest` run skips it; pass `--run-ignored` (as CI does) to include it: ```bash -uv run --dev pytest +uv run --dev pytest --run-ignored # whole suite, snippets included +uv run --dev pytest tests/test_docs --run-ignored # snippets only ``` **4. Component pages: a YAML file, and generated argument tables.** Each parser and @@ -177,7 +178,7 @@ For a live local preview while editing: uv run --dev mkdocs serve ``` -`mkdocs` comes in transitively via `mike` in the `dev` group, so `--dev` is -required. There is no `--strict` mode configured; the hard check on the docs is +`mkdocs` comes in via `mkdocs-material` (a direct dependency of the `dev` group), +so `--dev` is required. There is no `--strict` mode configured; the hard check on the docs is `check_paths: true` from `pymdownx.snippets`, which fails the build on a missing snippet or marker. diff --git a/docs/examples/data/named_templates.csv b/docs/examples/data/named_templates.csv new file mode 100644 index 00000000..a0606052 --- /dev/null +++ b/docs/examples/data/named_templates.csv @@ -0,0 +1,3 @@ +EventId,EventTemplate +login_ok,Accepted password for from port +login_failure,Failed password for from port diff --git a/docs/examples/data/tutorial_audit.log b/docs/examples/data/tutorial_audit.log new file mode 100644 index 00000000..8d5b9f7a --- /dev/null +++ b/docs/examples/data/tutorial_audit.log @@ -0,0 +1,9 @@ +type=DAEMON_START msg=audit(1757673850.240:4506): op=start ver=3.0.9 format=enriched kernel=6.6.62+rpt-rpi-2712 auid=4294967295 pid=691 uid=0 ses=4294967295 res=successAUID="unset" UID="root" +type=CONFIG_CHANGE msg=audit(1757673850.283:3): op=set audit_backlog_limit=8192 old=64 auid=4294967295 ses=4294967295 res=1AUID="unset" +type=SYSCALL msg=audit(1757673850.283:3): arch=c00000b7 syscall=206 success=yes exit=60 a0=3 a1=7fffc27ad8f0 a2=3c a3=0 items=0 ppid=696 pid=710 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="auditctl" exe="/usr/sbin/auditctl" key=(null)ARCH=aarch64 SYSCALL=sendto AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" +type=PROCTITLE msg=audit(1757673850.283:3): proctitle=2F7362696E2F617564697463746C002D52002F6574632F61756469742F61756469742E72756C6573 +type=CONFIG_CHANGE msg=audit(1757673850.283:4): op=set audit_failure=1 old=1 auid=4294967295 ses=4294967295 res=1AUID="unset" +type=SYSCALL msg=audit(1757673850.283:4): arch=c00000b7 syscall=206 success=yes exit=60 a0=3 a1=7fffc27ad8f0 a2=3c a3=0 items=0 ppid=696 pid=710 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="auditctl" exe="/usr/sbin/auditctl" key=(null)ARCH=aarch64 SYSCALL=sendto AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" +type=PROCTITLE msg=audit(1757673850.283:4): proctitle=2F7362696E2F617564697463746C002D52002F6574632F61756469742F61756469742E72756C6573 +type=CONFIG_CHANGE msg=audit(1757673850.283:5): op=set audit_backlog_wait_time=60000 old=15000 auid=4294967295 ses=4294967295 res=1AUID="unset" +type=SYSCALL msg=audit(1757673850.283:5): arch=c00000b7 syscall=206 success=yes exit=60 a0=3 a1=7fffc27ad8f0 a2=3c a3=0 items=0 ppid=696 pid=710 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="auditctl" exe="/usr/sbin/auditctl" key=(null)ARCH=aarch64 SYSCALL=sendto AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" diff --git a/tests/test_data/logs_templates.txt b/docs/examples/data/tutorial_templates.txt similarity index 100% rename from tests/test_data/logs_templates.txt rename to docs/examples/data/tutorial_templates.txt diff --git a/docs/examples/detectors/named_events.py b/docs/examples/detectors/named_events.py new file mode 100644 index 00000000..1b8cbece --- /dev/null +++ b/docs/examples/detectors/named_events.py @@ -0,0 +1,42 @@ +# --8<-- [start:example] +from detectmatelibrary import schemas +from detectmatelibrary.parsers.template_matcher import MatcherParser +from detectmatelibrary.detectors.new_value_detector import NewValueDetector + +# Templates from a CSV file: the EventId column names each template, and +# named wildcards (, , ) name its variables. +parser = MatcherParser(name="MatcherParser", config={"parsers": {"MatcherParser": { + "method_type": "matcher_parser", + "path_templates": "docs/examples/data/named_templates.csv", +}}}) + +# The detector configuration uses those names instead of numbers. +detector = NewValueDetector(name="NewUserDetector", config={"detectors": {"NewUserDetector": { + "method_type": "new_value_detector", + "auto_config": False, + "params": {"data_use_training": 2}, + "events": { + "login_failure": { # EventId from the CSV + "failed_login": { + "variables": [{"pos": "user", "name": "user"}], # named wildcard + }, + }, + }, +}}}) + +# Resolve the names to the numbers the parser emits (login_failure -> 1, user -> 0). +# Without this step the detector monitors nothing. +detector.config.events = parser.template_matcher.compile_detector_config(detector.config.events) + +lines = [ + "Failed password for alice from 10.0.0.1 port 22", # training + "Failed password for alice from 10.0.0.2 port 22", # training + "Failed password for mallory from 10.0.0.3 port 22", # new user +] +for line in lines: + alert = detector.process(parser.process(schemas.LogSchema({"log": line}))) + print(dict(alert["alertsObtain"]) if alert else None) +# None +# None +# {'EventID 1 - user': "Unknown value: 'mallory'"} +# --8<-- [end:example] diff --git a/docs/examples/others/basic_usage.py b/docs/examples/others/basic_usage.py deleted file mode 100644 index dbf95c17..00000000 --- a/docs/examples/others/basic_usage.py +++ /dev/null @@ -1,75 +0,0 @@ -# flake8: noqa -log_path = "tests/test_data/logs.log" -templates_path = "tests/test_data/audit_templates.txt" -parsed_path = "parsed_logs.json" -log_json = "logs.json" -alert_path = "alerts.json" - - -# --8<-- [start:example_1] -from detectmatelibrary.parsers.template_matcher import MatcherParser -from detectmatelibrary.helper.from_to import From, To - - -config_dict = { - "parsers": { - "MatcherParser": { - "auto_config": True, - "method_type": "matcher_parser", - "path_templates": templates_path, - "log_format": r'type= msg=audit\(