Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test_changelog.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ permissions:
contents: read

jobs:
fragment:
changelog_fragment:
# Dependabot only bumps action and dependency versions, which needs no fragment.
if: >-
github.event.pull_request.draft == false
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test_docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ concurrency:
cancel-in-progress: true

jobs:
docs:
argument_specs_and_readme:
# For non-PR events (workflow_dispatch, etc.), github.event.pull_request.draft is null.
# GitHub Actions coerces null == false to true, so the job runs for all non-draft events.
if: github.event.pull_request.draft == false
Expand Down
2 changes: 1 addition & 1 deletion NOTICE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@

Here's a list of sponsors who contributed by having the collection improved via outsourcing to NETWAYS.

* CID GmbH : Thank you so much for sponsoring. Especially the feature to have different types of Elasticsearch nodes in the cluster and the integration of rolling upgrades.
* CID GmbH: Thank you so much for sponsoring. Especially the feature to have different types of Elasticsearch nodes in the cluster and the integration of rolling upgrades.
8 changes: 2 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The minimum required version is declared in [meta/runtime.yml](meta/runtime.yml)

| | |
|---|---|
| ansible-core | `>= 2.18`, tested with 2.19 and 2.20 |
| ansible-core | `>= 2.18` |
| Python on the control node | `>= 3.11` |
| Python on the target hosts | `>= 3.9` |
| Elastic Stack | release 8 |
Expand Down Expand Up @@ -119,10 +119,6 @@ The execution order of the roles matters, and the roles interact with hosts from

## Caveats and information for long time users

### Variable renaming

If you have been using this collection before version `1.0.0`, note that a significant number of variables had to be renamed because of naming schema changes made by Ansible. Please review the variables you set in your playbooks and variable files.

### Ulimit management for Elasticsearch

The role no longer configures `ulimit`. Make sure the system's open file limit is set correctly, packages usually handle this. Verify with `ulimit -n` or by checking `/proc/<pid>/limits`. If your version does not, please [open an issue](https://github.com/NETWAYS/ansible-collection-elasticstack/issues).
Expand All @@ -131,7 +127,7 @@ The role no longer configures `ulimit`. Make sure the system's open file limit i

Every kind of contribution is welcome. Open [issues](https://github.com/NETWAYS/ansible-collection-elasticstack/issues) or provide [pull requests](https://github.com/NETWAYS/ansible-collection-elasticstack/pulls).

Pull requests go against `main`. If you need a stable state, pin a release tag.
Pull requests go against `main`.

Please read [CONTRIBUTING.md](CONTRIBUTING.md) first. It describes what a pull request needs, including the changelog fragment that every pull request has to bring along, how role variables are documented and how to run the tests.

Expand Down
10 changes: 10 additions & 0 deletions changelogs/fragments/docs_wording_cleanup.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
trivial:
- The repository requirement is worded the same way in all four roles that need it, and the
requirements that stated the obvious are gone.
- The pipelines documentation no longer labels the generated Redis input block as an output,
and the prose around the examples says what the role actually does.
- "The three ``beats_<beat>_output`` descriptions now say what choosing each value changes
instead of repeating the variable name, the README no longer names the tested ansible-core
versions, and the stale variable renaming notice is removed."
- The two check jobs are named after what they verify, ``argument_specs_and_readme`` and
``changelog_fragment``, so the status check on a pull request says which condition failed.
18 changes: 9 additions & 9 deletions roles/beats/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Ansible Role: Beats

Installs and configures the Elastic [Beats](https://www.elastic.co/beats) — Filebeat, Auditbeat and Metricbeat — on Linux systems. The role can run standalone or together with the other Elastic Stack roles. Beats can ship to Logstash or directly to Elasticsearch.
Installs and configures the Elastic [Beats](https://www.elastic.co/beats) on Linux systems: Filebeat, Auditbeat and Metricbeat. The role can run standalone or together with the other Elastic Stack roles. Beats can ship to Logstash or directly to Elasticsearch.

## Requirements

* The `community.crypto` collection (used to check certificate expiration).
* The Beats you want to install must be available in your software repositories — use the [`repos`](../repos) role or provide them yourself.
* The Elastic repositories configured: use the [`repos`](../repos/README.md) role or provide them yourself.

## Example

Expand All @@ -27,9 +27,9 @@ Filebeat can read from several sources (log files, syslog, journald, Docker, mod

Run only parts of the role with `--tags`:

* `configuration` (alias `beats_configuration`) — (re)write the Filebeat and Auditbeat configuration without installing. Use `beats_filebeat_configuration` or `beats_auditbeat_configuration` to limit it to one of them.
* `certificates` — only generate and distribute the TLS certificates.
* `renew_beats_cert` / `renew_ca` — force renewal of the Beats certificate.
* `configuration` (alias `beats_configuration`): (re)write the Filebeat and Auditbeat configuration without installing. Use `beats_filebeat_configuration` or `beats_auditbeat_configuration` to limit it to one of them.
* `certificates`: only generate and distribute the TLS certificates.
* `renew_beats_cert` / `renew_ca`: force renewal of the Beats certificate.

<!-- ANSIBLE DOCSMITH MAIN START -->
## Role variables<a id="variables"></a>
Expand All @@ -40,12 +40,12 @@ Run only parts of the role with `--tags`:
| `beats_auditbeat`<br>**Type**: `bool`<br>**Default**: `false` | Install and manage Auditbeat. |
| `beats_metricbeat`<br>**Type**: `bool`<br>**Default**: `false` | Install and manage Metricbeat. |
| `beats_target_hosts`<br>**Type**: `list` of `str`<br>**Default**: `['localhost']` | Hosts the Beats ship to. Only used when the role runs standalone; with the other Elastic Stack roles the targets are determined automatically. |
| `beats_fields`<br>**Type**: `list` of `str` | Global fields added to the log and syslog (tcp/udp) inputs (not to the mysql, journald or docker inputs), given as a list of "key: value" strings — the global counterpart to the per-input fields. Unset by default. See the Filebeat inputs documentation. |
| `beats_fields`<br>**Type**: `list` of `str` | Global fields added to the log and syslog (tcp/udp) inputs (not to the mysql, journald or docker inputs), given as a list of "key: value" strings. This is the global counterpart to the per-input fields. Unset by default. See the Filebeat inputs documentation. |
| `beats_logging`<br>**Type**: `str`<br>**Default**: `"file"` | Where the Beats log. Set to "file" to log into beats_logpath; any other value leaves the Beats built-in logging. |
| `beats_loglevel`<br>**Type**: `str`<br>**Default**: `"info"` | Log level for all Beats. |
| `beats_logpath`<br>**Type**: `str`<br>**Default**: `"/var/log/beats"` | Directory for the log files when beats_logging is "file". |
| `beats_filebeat_enable`<br>**Type**: `bool`<br>**Default**: `true` | Start and enable the Filebeat service. |
| `beats_filebeat_output`<br>**Type**: `str`<br>**Default**: `"logstash"`<br>**Choices**: `logstash`, `elasticsearch` | Where Filebeat sends its events. |
| `beats_filebeat_output`<br>**Type**: `str`<br>**Default**: `"logstash"`<br>**Choices**: `logstash`, `elasticsearch` | Where Filebeat sends its events. With "logstash" they pass through the Logstash pipelines and can be filtered and enriched there, which is what log lines usually need. With "elasticsearch" they go straight into the cluster. |
| `beats_filebeat_elastic_monitoring`<br>**Type**: `bool`<br>**Default**: `false` | Report Filebeat monitoring data through the Elastic Stack monitoring features. |
| `beats_filebeat_loadbalance`<br>**Type**: `bool`<br>**Default**: `true` | Enable load balancing for the Filebeat Logstash output. |
| `beats_filebeat_log_input`<br>**Type**: `bool`<br>**Default**: `true` | Read log files with the inputs from beats_filebeat_log_inputs. |
Expand All @@ -61,11 +61,11 @@ Run only parts of the role with `--tags`:
| `beats_filebeat_mysql_slowlog_input`<br>**Type**: `bool`<br>**Default**: `false` | Collect the MySQL/MariaDB slow query log, including multiline handling. |
| `beats_filebeat_modules`<br>**Type**: `list` of `str` | List of Filebeat modules to enable (experimental). Unset by default. |
| `beats_auditbeat_enable`<br>**Type**: `bool`<br>**Default**: `true` | Start and enable the Auditbeat service. |
| `beats_auditbeat_output`<br>**Type**: `str`<br>**Default**: `"elasticsearch"`<br>**Choices**: `logstash`, `elasticsearch` | Where Auditbeat sends its events. |
| `beats_auditbeat_output`<br>**Type**: `str`<br>**Default**: `"elasticsearch"`<br>**Choices**: `logstash`, `elasticsearch` | Where Auditbeat sends its events. Defaults to "elasticsearch" because audit events already arrive structured and gain nothing from passing through Logstash. Set it to "logstash" to route them through the pipelines anyway. |
| `beats_auditbeat_setup`<br>**Type**: `bool`<br>**Default**: `true` | Run the Auditbeat setup (index management, ingest pipelines and Kibana dashboards). Only effective with the elasticsearch output; loading the dashboards additionally requires Kibana to be reachable. |
| `beats_auditbeat_loadbalance`<br>**Type**: `bool`<br>**Default**: `true` | Enable load balancing for the Auditbeat Logstash output. |
| `beats_metricbeat_enable`<br>**Type**: `bool`<br>**Default**: `true` | Start and enable the Metricbeat service. |
| `beats_metricbeat_output`<br>**Type**: `str`<br>**Default**: `"elasticsearch"`<br>**Choices**: `logstash`, `elasticsearch` | Where Metricbeat sends its events. |
| `beats_metricbeat_output`<br>**Type**: `str`<br>**Default**: `"elasticsearch"`<br>**Choices**: `logstash`, `elasticsearch` | Where Metricbeat sends its events. Defaults to "elasticsearch" because metrics already arrive structured and gain nothing from passing through Logstash. Set it to "logstash" to route them through the pipelines anyway. |
| `beats_metricbeat_modules`<br>**Type**: `list` of `str`<br>**Default**: `['system']` | Metricbeat modules to enable. |
| `beats_metricbeat_loadbalance`<br>**Type**: `bool`<br>**Default**: `true` | Enable load balancing for the Metricbeat Logstash output. |
| `beats_security`<br>**Type**: `bool`<br>**Default**: `false` | Activate TLS for the connections to the targets. Works with the other roles and elasticstack_full_stack to create certificates automatically, or with the beats_tls_* variables for custom certificates. |
Expand Down
8 changes: 4 additions & 4 deletions roles/beats/docs/filebeat-inputs.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Filebeat can read from several sources. Each source is turned into a Filebeat in

Set with `beats_filebeat_log_input` (default `true`) and `beats_filebeat_log_inputs`.

`beats_filebeat_log_inputs` is a **dictionary keyed by a free name**. That key becomes the id of the generated [`filestream`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream) input (`<key>-filestream`) — there is no separate `name` field. Each entry needs `paths`; `fields` and `multiline` are optional.
`beats_filebeat_log_inputs` is a **dictionary keyed by a free name**. That key becomes the id of the generated [`filestream`](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-filestream) input (`<key>-filestream`). There is no separate `name` field. Each entry needs `paths`; `fields` and `multiline` are optional.

The default reads the system log:

Expand Down Expand Up @@ -40,7 +40,7 @@ beats_filebeat_log_inputs:

There are two ways to add fields, and they use **different shapes**:

* **Per input** — the optional `fields` inside a log input is a **dictionary** (`key: value`), added only to that input:
* **Per input**: the optional `fields` inside a log input is a **dictionary** (`key: value`), added only to that input:

```yaml
beats_filebeat_log_inputs:
Expand All @@ -51,7 +51,7 @@ There are two ways to add fields, and they use **different shapes**:
environment: production
```

* **Globally** — `beats_fields` is a **list of `"key: value"` strings**. In the current templates it is added to the log, TCP and UDP inputs; the journald, Docker and MySQL slow-log inputs do not receive it:
* **Globally**: `beats_fields` is a **list of `"key: value"` strings**. In the current templates it is added to the log, TCP and UDP inputs; the journald, Docker and MySQL slow-log inputs do not receive it:

```yaml
beats_fields:
Expand All @@ -72,7 +72,7 @@ beats_filebeat_syslog_tcp_port: 514

Enable with `beats_filebeat_journald` (default `false`, available since Filebeat 7.16) and configure inputs with `beats_filebeat_journald_inputs`.

This is a **dictionary keyed by a free name**. Unlike the log inputs, here the key is only a label — the input's id comes from the `id` field, which each entry needs. `include_matches` is optional and is itself a dictionary whose values are the match expressions.
This is a **dictionary keyed by a free name**. Unlike the log inputs, here the key is only a label. The input's id comes from the `id` field, which each entry needs. `include_matches` is optional and is itself a dictionary whose values are the match expressions.

```yaml
beats_filebeat_journald_inputs:
Expand Down
17 changes: 13 additions & 4 deletions roles/beats/meta/argument_specs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ argument_specs:
description: >-
Global fields added to the log and syslog (tcp/udp) inputs (not to the
mysql, journald or docker inputs), given as a list of "key: value"
strings — the global counterpart to the per-input fields. Unset by
strings. This is the global counterpart to the per-input fields. Unset by
default. See the Filebeat inputs documentation.

# ----- Logging -----
Expand Down Expand Up @@ -88,7 +88,10 @@ argument_specs:
- logstash
- elasticsearch
default: logstash
description: Where Filebeat sends its events.
description: >-
Where Filebeat sends its events. With "logstash" they pass through the Logstash
pipelines and can be filtered and enriched there, which is what log lines usually
need. With "elasticsearch" they go straight into the cluster.

beats_filebeat_elastic_monitoring:
type: bool
Expand Down Expand Up @@ -183,7 +186,10 @@ argument_specs:
- logstash
- elasticsearch
default: elasticsearch
description: Where Auditbeat sends its events.
description: >-
Where Auditbeat sends its events. Defaults to "elasticsearch" because audit events
already arrive structured and gain nothing from passing through Logstash. Set it to
"logstash" to route them through the pipelines anyway.

beats_auditbeat_setup:
type: bool
Expand All @@ -210,7 +216,10 @@ argument_specs:
- logstash
- elasticsearch
default: elasticsearch
description: Where Metricbeat sends its events.
description: >-
Where Metricbeat sends its events. Defaults to "elasticsearch" because metrics
already arrive structured and gain nothing from passing through Logstash. Set it to
"logstash" to route them through the pipelines anyway.

beats_metricbeat_modules:
type: list
Expand Down
8 changes: 4 additions & 4 deletions roles/elasticsearch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The OSS variant is only available up to Elastic Stack 7; from release 8 on only

## Requirements

* The Elastic repositories configured — use the [`repos`](../repos) role.
* The Elastic repositories configured: use the [`repos`](../repos/README.md) role or provide them yourself.
* For the default security setup the role generates certificates with the bundled `elasticsearch-certutil` and `openssl`. The required Python libraries (`cryptography`, `elasticsearch`) are installed for you by the shared `elasticstack` role.

## Example
Expand All @@ -25,9 +25,9 @@ The OSS variant is only available up to Elastic Stack 7; from release 8 on only

Run only parts of the role with `--tags`:

* `certificates` — only generate and distribute the TLS certificates.
* `renew_es_cert` — force renewal of the Elasticsearch node certificate.
* `renew_ca` — force renewal of the stack CA and all component certificates.
* `certificates`: only generate and distribute the TLS certificates.
* `renew_es_cert`: force renewal of the Elasticsearch node certificate.
* `renew_ca`: force renewal of the stack CA and all component certificates.

<!-- ANSIBLE DOCSMITH MAIN START -->
## Role variables<a id="variables"></a>
Expand Down
2 changes: 1 addition & 1 deletion roles/elasticsearch/tasks/elasticsearch-security.yml
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
---

- name: Ensure ca exists

Check warning on line 3 in roles/elasticsearch/tasks/elasticsearch-security.yml

View workflow job for this annotation

GitHub Actions / lint_collection / lint

var-naming[no-role-prefix]

Variables names from within roles should use elasticsearch_ as a prefix. (register: elasticstack_ca_exists)
ansible.builtin.stat:
path: "{{ elasticstack_ca_dir }}/elastic-stack-ca.p12"
register: elasticstack_ca_exists
when: inventory_hostname == elasticstack_ca_host

- name: Get CA informations

Check warning on line 9 in roles/elasticsearch/tasks/elasticsearch-security.yml

View workflow job for this annotation

GitHub Actions / lint_collection / lint

var-naming[no-role-prefix]

Variables names from within roles should use elasticsearch_ as a prefix. (register: elasticstack_ca_infos)
cert_info:
path: "{{ elasticstack_ca_dir }}/elastic-stack-ca.p12"
passphrase: "{{ elasticstack_ca_pass | default(omit, true) }}"
register: elasticstack_ca_infos
when: inventory_hostname == elasticstack_ca_host and elasticstack_ca_exists.stat.exists | bool

- name: Set the ca expiration date in days

Check warning on line 16 in roles/elasticsearch/tasks/elasticsearch-security.yml

View workflow job for this annotation

GitHub Actions / lint_collection / lint

var-naming[no-role-prefix]

Variables names from within roles should use elasticsearch_ as a prefix. (set_fact: elasticstack_ca_expiration_days)
ansible.builtin.set_fact:
elasticstack_ca_expiration_days: "{{ ((elasticstack_ca_infos.not_valid_after | to_datetime()) - (ansible_date_time.date | to_datetime('%Y-%m-%d'))).days }}"
when: inventory_hostname == elasticstack_ca_host and elasticstack_ca_infos.skipped is not defined

- name: Set ca will expire soon to true

Check warning on line 21 in roles/elasticsearch/tasks/elasticsearch-security.yml

View workflow job for this annotation

GitHub Actions / lint_collection / lint

var-naming[no-role-prefix]

Variables names from within roles should use elasticsearch_ as a prefix. (set_fact: elasticstack_ca_will_expire_soon)
ansible.builtin.set_fact:
elasticstack_ca_will_expire_soon: true
when: >
Expand Down Expand Up @@ -231,7 +231,7 @@
# Pass passphrases via argv (a list), not a folded shell string. A string command is
# tokenized by shlex.split() (shell-like word splitting and quote parsing), which mangles
# passphrases containing spaces or quotes. With argv every element is handed to the binary
# literally — no shell, no shlex — so arbitrary special characters work.
# literally, with no shell and no shlex, so arbitrary special characters work.
- name: Configure ca on elasticstack_ca_host
ansible.builtin.command:
argv:
Expand Down Expand Up @@ -433,7 +433,7 @@
retries: 5
delay: 10

- name: Fetch Elastic password # noqa: risky-shell-pipe

Check warning on line 436 in roles/elasticsearch/tasks/elasticsearch-security.yml

View workflow job for this annotation

GitHub Actions / lint_collection / lint

var-naming[no-role-prefix]

Variables names from within roles should use elasticsearch_ as a prefix. (register: elasticstack_password)
ansible.builtin.shell: >
if test -n "$(ps -p $$ | grep bash)"; then set -o pipefail; fi;
grep "PASSWORD elastic" {{ elasticstack_initial_passwords }} |
Expand Down
4 changes: 2 additions & 2 deletions roles/elasticstack/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Ansible Role: elasticstack

The shared base role of the collection. Every component role (elasticsearch,logstash, kibana, beats, repos) imports it automatically on first use, so you normally do not call it directly. It installs the common dependencies and, above all, defines the collection-wide `elasticstack_*` variables — the settings shared across all roles: release and variant, the inventory host groups, ports, the package repositories, and the stack CA.
The shared base role of the collection. Every component role (elasticsearch, logstash, kibana, beats, repos) imports it automatically on first use, so you normally do not call it directly. It installs the common dependencies and, above all, defines the collection-wide `elasticstack_*` variables, the settings shared across all roles: release and variant, the inventory host groups, ports, the package repositories, and the stack CA.

Set these variables once (in `group_vars`, the play, or via `-e`) and every role picks them up. This page is the central reference for them.

## Requirements

You do not run this role directly — the component roles import it automatically. It installs the shared Python libraries (`cryptography` and the `elasticsearch` client) and packages (`openssl`, `unzip`) that the collection needs.
It installs the shared Python libraries (`cryptography` and the `elasticsearch` client) and the packages (`openssl`, `unzip`) that the collection needs.

## Example

Expand Down
Loading
Loading