Address mode

changelogs/fragments/address_mode.yml

@@ -0,0 +1,17 @@
+minor_changes:
+  - The new ``elasticstack_address_mode`` selects how the Ansible roles address a host of the
+    elasticsearch, logstash or kibana inventory group. ``inventory`` keeps the name from the
+    inventory as before, ``fqdn``, ``hostname`` and ``ip`` take the address from the facts of that
+    host. Each host is addressed in the mode set for it in the inventory, in group_vars or
+    host_vars, or with ``-e``; a value in the vars of a play or a role parameter stops the run
+    with an error. Facts of those hosts that are missing, for example in a run limited with
+    ``--limit``, are gathered first.
+  - Setting ``elasticstack_node_address`` in the host_vars of a host makes the Ansible roles reach that
+    host at this address, whatever ``elasticstack_address_mode`` says, for example through the
+    IP of a particular network card. The address is also added to the certificate of an
+    Elasticsearch host and to the certificate a Logstash host presents to the Beats. Existing
+    certificates get it when they are renewed.
+
+trivial:
+  - The full stack Molecule scenario sets ``elasticstack_address_mode`` to ``ip`` in its inventory
+    and no longer writes the container addresses into /etc/hosts.

docs/getting-started.md

@@ -10,7 +10,7 @@ Hosts in these groups get the respective services. Restricting your plays to onl
 
 Make sure every host that should be configured is part of your playbook, including hosts that only run Beats. The collection first gathers facts from all hosts and then uses facts such as host names and IP addresses to connect the tools to each other.
 
-You will want reliable DNS resolution, or all hosts of the stack in your systems' hosts files.
+By default Beats, Kibana and Logstash reach the other hosts by their names from your inventory, so these names have to resolve on those hosts, through DNS or the hosts files. Without that, set `elasticstack_address_mode` to `ip`, described with the [elasticstack role](../roles/elasticstack/README.md). Unlike the variables in the example playbook below, it belongs into your inventory, for example `group_vars/all`; in the `vars` of a play it stops the run.
 
 ## Before the first run
 

molecule/elasticstack_default/molecule.yml

@@ -40,6 +40,11 @@ provisioner:
   env:
     ANSIBLE_ROLES_PATH: $MOLECULE_EPHEMERAL_DIRECTORY/roles:${HOME}/.ansible/roles:$MOLECULE_PROJECT_DIRECTORY/roles
   inventory:
+    group_vars:
+      all:
+        # The containers cannot resolve each other's names on the GitHub runner. The roles read
+        # elasticstack_address_mode from the inventory, the vars of a play would stop the run.
+        elasticstack_address_mode: ip
     host_vars:
       elasticstack${ELASTIC_RELEASE}-cluster2-${MOLECULE_DISTRO}:
         elasticsearch_tls_key_passphrase: UniqueHostPassword

molecule/elasticstack_default/prepare.yml

@@ -59,20 +59,3 @@
         state: latest
         update_cache: yes
       when: ansible_os_family == "Debian"
-
-    # Temporary CI workaround: containers cannot resolve each other by name
-    # (the runner's resolver is Azure's 168.63.129.16, which does not know the
-    # container hostnames). Seed /etc/hosts from gathered facts so name-based
-    # connections work. Remove once #524 (elasticstack_address_mode) lands and
-    # molecule sets elasticstack_address_mode: ip.
-    - name: Seed /etc/hosts from facts for inter-container name resolution
-      ansible.builtin.blockinfile:
-        path: /etc/hosts
-        marker: "# {mark} ANSIBLE CLUSTER HOSTS"
-        # /etc/hosts is a Docker bind-mount, so the default atomic temp+rename
-        # write fails with EBUSY; write in place instead.
-        unsafe_writes: true
-        block: |-
-          {% for h in groups['all'] %}
-          {{ hostvars[h].ansible_default_ipv4.address | default(hostvars[h].ansible_all_ipv4_addresses[0]) }} {{ hostvars[h].ansible_fqdn }} {{ hostvars[h].ansible_hostname }} {{ h }}
-          {% endfor %}

roles/beats/README.md

@@ -39,8 +39,8 @@ Run only parts of the role with `--tags`:
 | `beats_filebeat`<br>**Type**: `bool`<br>**Default**: `true` | Install and manage Filebeat. |
 | `beats_auditbeat`<br>**Type**: `bool`<br>**Default**: `false` | Install and manage Auditbeat. |
 | `beats_metricbeat`<br>**Type**: `bool`<br>**Default**: `false` | Install and manage Metricbeat. |
-| `beats_elasticsearch_hosts`<br>**Type**: `list` of `str` | Elasticsearch hosts the Beats ship to. When elasticstack_full_stack is true and this variable is not set, the role fills it with the hosts of the inventory group named in elasticstack_elasticsearch_group_name. When elasticstack_full_stack is false, set it yourself as soon as a Beat uses the elasticsearch output, otherwise the run stops. |
-| `beats_logstash_hosts`<br>**Type**: `list` of `str` | Logstash hosts the Beats ship to. When elasticstack_full_stack is true and this variable is not set, the role fills it with the hosts of the inventory group named in elasticstack_logstash_group_name. When elasticstack_full_stack is false, set it yourself as soon as a Beat uses the logstash output, otherwise the run stops. |
+| `beats_elasticsearch_hosts`<br>**Type**: `list` of `str` | Elasticsearch hosts the Beats ship to. When elasticstack_full_stack is true and this variable is not set, the role fills it with the addresses of the hosts in the inventory group named in elasticstack_elasticsearch_group_name, picked by elasticstack_address_mode and elasticstack_node_address. When elasticstack_full_stack is false, set it yourself as soon as a Beat uses the elasticsearch output, otherwise the run stops. |
+| `beats_logstash_hosts`<br>**Type**: `list` of `str` | Logstash hosts the Beats ship to. When elasticstack_full_stack is true and this variable is not set, the role fills it with the addresses of the hosts in the inventory group named in elasticstack_logstash_group_name, picked by elasticstack_address_mode and elasticstack_node_address. When elasticstack_full_stack is false, set it yourself as soon as a Beat uses the logstash output, otherwise the run stops. |
 | `beats_fields`<br>**Type**: `list` of `str` | Global fields added to the log and syslog (tcp/udp) inputs (not to the mysql, journald or docker inputs), given as a list of "key: value" strings. This is the global counterpart to the per-input fields. Unset by default. See the Filebeat inputs documentation. |
 | `beats_logging`<br>**Type**: `str`<br>**Default**: `"file"` | Where the Beats log. Set to "file" to log into beats_logpath; any other value leaves the Beats built-in logging. |
 | `beats_loglevel`<br>**Type**: `str`<br>**Default**: `"info"` | Log level for all Beats. |

roles/beats/meta/argument_specs.yml

@@ -45,20 +45,22 @@ argument_specs:
         elements: str
         description: >-
           Elasticsearch hosts the Beats ship to. When elasticstack_full_stack is true and
-          this variable is not set, the role fills it with the hosts of the inventory group
-          named in elasticstack_elasticsearch_group_name. When elasticstack_full_stack is
-          false, set it yourself as soon as a Beat uses the elasticsearch output, otherwise
-          the run stops.
+          this variable is not set, the role fills it with the addresses of the hosts in the
+          inventory group named in elasticstack_elasticsearch_group_name, picked by
+          elasticstack_address_mode and elasticstack_node_address. When
+          elasticstack_full_stack is false, set it yourself as soon as a Beat uses the
+          elasticsearch output, otherwise the run stops.
 
       beats_logstash_hosts:
         type: list
         elements: str
         description: >-
           Logstash hosts the Beats ship to. When elasticstack_full_stack is true and this
-          variable is not set, the role fills it with the hosts of the inventory group
-          named in elasticstack_logstash_group_name. When elasticstack_full_stack is false,
-          set it yourself as soon as a Beat uses the logstash output, otherwise the run
-          stops.
+          variable is not set, the role fills it with the addresses of the hosts in the
+          inventory group named in elasticstack_logstash_group_name, picked by
+          elasticstack_address_mode and elasticstack_node_address. When
+          elasticstack_full_stack is false, set it yourself as soon as a Beat uses the
+          logstash output, otherwise the run stops.
 
       beats_fields:
         type: list

roles/beats/tasks/main.yml

@@ -3,7 +3,6 @@
 - name: Include global role
   ansible.builtin.import_role:
     name: netways.elasticstack.elasticstack
-  when: not hostvars[inventory_hostname]._elasticstack_role_imported | default(false)
 
 - name: Update apt cache.
   ansible.builtin.apt:
@@ -69,6 +68,7 @@
   ansible.builtin.set_fact:
     beats_elasticsearch_hosts: >-
       {{ groups[elasticstack_elasticsearch_group_name] | default([])
+         | map('extract', __elasticstack_node_address) | list
          if elasticstack_full_stack | bool else [] }}
   when: beats_elasticsearch_hosts is undefined
   tags: always
@@ -77,6 +77,7 @@
   ansible.builtin.set_fact:
     beats_logstash_hosts: >-
       {{ groups[elasticstack_logstash_group_name] | default([])
+         | map('extract', __elasticstack_node_address) | list
          if elasticstack_full_stack | bool else [] }}
   when: beats_logstash_hosts is undefined
   tags: always

roles/beats/templates/auditbeat.yml.j2

@@ -32,7 +32,7 @@ setup.template.settings:
   index.number_of_shards: 1
 setup.kibana:
 {% if elasticstack_full_stack | bool and groups[elasticstack_kibana_group_name] | default([]) | length > 0 %}
-  host: "http://{{ groups[elasticstack_kibana_group_name] | first }}:{{ elasticstack_kibana_port }}"
+  host: "http://{{ __elasticstack_node_address[groups[elasticstack_kibana_group_name] | first] }}:{{ elasticstack_kibana_port }}"
 {% if beats_security | bool %}
   username: "elastic"
   password: "{{ beats_writer_password.stdout }}"

roles/beats/templates/filebeat.yml.j2

@@ -116,7 +116,7 @@ setup.template.settings:
   index.number_of_shards: 1
 setup.kibana:
 {% if elasticstack_full_stack | bool and groups[elasticstack_kibana_group_name] | default([]) | length > 0 %}
-  host: "http://{{ groups[elasticstack_kibana_group_name] | first }}:{{ elasticstack_kibana_port }}"
+  host: "http://{{ __elasticstack_node_address[groups[elasticstack_kibana_group_name] | first] }}:{{ elasticstack_kibana_port }}"
 {% if beats_security | bool %}
   username: "elastic"
   password: "{{ beats_writer_password.stdout }}"

roles/beats/templates/metricbeat.yml.j2

@@ -7,7 +7,7 @@ setup.template.settings:
   index.number_of_shards: 1
 setup.kibana:
 {% if elasticstack_full_stack | bool and groups[elasticstack_kibana_group_name] | default([]) | length > 0 %}
-  host: "http://{{ groups[elasticstack_kibana_group_name] | first }}:{{ elasticstack_kibana_port }}"
+  host: "http://{{ __elasticstack_node_address[groups[elasticstack_kibana_group_name] | first] }}:{{ elasticstack_kibana_port }}"
 {% if beats_security | bool %}
   username: "elastic"
   password: "{{ beats_writer_password.stdout }}"

roles/elasticsearch/tasks/elasticsearch-security.yml

@@ -1,24 +1,24 @@
 ---
 
 - name: Ensure ca exists
   ansible.builtin.stat:
     path: "{{ elasticstack_ca_dir }}/elastic-stack-ca.p12"
   register: elasticstack_ca_exists
   when: inventory_hostname == elasticstack_ca_host
 
 - name: Get CA informations
   cert_info:
     path: "{{ elasticstack_ca_dir }}/elastic-stack-ca.p12"
     passphrase: "{{ elasticstack_ca_pass | default(omit, true) }}"
   register: elasticstack_ca_infos
   when: inventory_hostname == elasticstack_ca_host and elasticstack_ca_exists.stat.exists | bool
 
 - name: Set the ca expiration date in days
   ansible.builtin.set_fact:
     elasticstack_ca_expiration_days: "{{ ((elasticstack_ca_infos.not_valid_after | to_datetime()) - (ansible_date_time.date | to_datetime('%Y-%m-%d'))).days }}"
   when: inventory_hostname == elasticstack_ca_host and elasticstack_ca_infos.skipped is not defined
 
 - name: Set ca will expire soon to true
   ansible.builtin.set_fact:
     elasticstack_ca_will_expire_soon: true
   when: >
@@ -263,9 +263,16 @@
           - --name
           - "{{ hostvars[item].ansible_hostname }}"
           - --ip
-          - "{{ hostvars[item].ansible_default_ipv4.address | default(hostvars[item].ansible_all_ipv4_addresses[0]) }}"
+          - >-
+            {{ ([hostvars[item].ansible_default_ipv4.address | default(hostvars[item].ansible_all_ipv4_addresses[0])]
+                + ([hostvars[item].elasticstack_node_address | default('')] | select('match', '^[0-9.]+$') | list))
+               | join(',') }}
           - --dns
-          - "{{ hostvars[item].ansible_hostname }},{{ hostvars[item].ansible_fqdn }},{{ hostvars[item].inventory_hostname }}"
+          - >-
+            {{ ([hostvars[item].ansible_hostname, hostvars[item].ansible_fqdn, hostvars[item].inventory_hostname]
+                + ([hostvars[item].elasticstack_node_address | default('')]
+                   | reject('match', '^[0-9.]+$') | reject('equalto', '') | list))
+               | join(',') }}
           - --pass
           - "{{ (hostvars[item].elasticsearch_tls_key_passphrase is defined) | ternary(hostvars[item].elasticsearch_tls_key_passphrase, elasticsearch_tls_key_passphrase) }}"
           - --out
@@ -433,7 +440,7 @@
   retries: 5
   delay: 10
 
 - name: Fetch Elastic password # noqa: risky-shell-pipe
   ansible.builtin.shell: >
     if test -n "$(ps -p $$ | grep bash)"; then set -o pipefail; fi;
     grep "PASSWORD elastic" {{ elasticstack_initial_passwords }} |

roles/elasticsearch/tasks/main.yml

@@ -10,7 +10,6 @@
 - name: Include global role
   ansible.builtin.import_role:
     name: netways.elasticstack.elasticstack
-  when: not hostvars[inventory_hostname]._elasticstack_role_imported | default(false)
 
 - name: Update apt cache.
   ansible.builtin.apt:

roles/elasticstack/README.md

@@ -35,6 +35,8 @@ The role runs automatically the first time any component role is applied to a ho
 | `elasticstack_elasticsearch_group_name`<br>**Type**: `str`<br>**Default**: `"elasticsearch"` | Inventory group that holds the Elasticsearch hosts. |
 | `elasticstack_logstash_group_name`<br>**Type**: `str`<br>**Default**: `"logstash"` | Inventory group that holds the Logstash hosts. |
 | `elasticstack_kibana_group_name`<br>**Type**: `str`<br>**Default**: `"kibana"` | Inventory group that holds the Kibana hosts. |
+| `elasticstack_address_mode`<br>**Type**: `str`<br>**Default**: `"inventory"`<br>**Choices**: `inventory`, `fqdn`, `hostname`, `ip` | How the Ansible roles of this collection address a host of the elasticsearch, logstash or kibana inventory group when they configure the connections to it. "inventory" uses the host's name in the inventory, "fqdn" its fully qualified domain name, "hostname" its short host name and "ip" the IPv4 address of its network card with the default route; the last three come from the facts of that host. Each host is addressed in the mode set for it, so a value in the group_vars or host_vars of some hosts changes only how those hosts are addressed. The Ansible roles read the value from the inventory or from extra vars; a value in the vars of a play or a role parameter stops the run with an error. The certificates the collection creates contain the IP, the short name, the FQDN and the inventory name, so every mode passes the TLS hostname check. Host lists you set yourself, such as beats_elasticsearch_hosts, are used as they are. The setting does not change discovery.seed_hosts, through which the Elasticsearch nodes find each other. |
+| `elasticstack_node_address`<br>**Type**: `str` | Address the Ansible roles of this collection use for this host when they configure the connections to it, an IPv4 address or a DNS name that resolves to this host. Set it in the host_vars of that host. It takes precedence over elasticstack_address_mode, for example to reach a host with several network cards through the IP of a particular one. It is also added to the certificate of an Elasticsearch host and to the certificate a Logstash host presents to the Beats. A certificate that already exists gets it only when it is renewed, with the tag renew_es_cert or renew_logstash_cert. It does not change discovery.seed_hosts. |
 | `elasticstack_elasticsearch_http_port`<br>**Type**: `int`<br>**Default**: `9200` | Elasticsearch HTTP API port. |
 | `elasticstack_kibana_host`<br>**Type**: `str`<br>**Default**: `"{{ ansible_fqdn }}"` | Host name Kibana publishes itself under (server.publicBaseUrl). Defaults to the fully qualified domain name of the Kibana host. Set it when users reach Kibana under a different name, for example behind a reverse proxy or a load balancer. |
 | `elasticstack_kibana_port`<br>**Type**: `int`<br>**Default**: `5601` | Kibana HTTP port (also used to build the public base URL). |

roles/elasticstack/defaults/main.yml

@@ -3,6 +3,7 @@
 elasticstack_elasticsearch_group_name: elasticsearch
 elasticstack_logstash_group_name: logstash
 elasticstack_kibana_group_name: kibana
+elasticstack_address_mode: inventory
 
 elasticstack_beats_port: 5044
 elasticstack_ca_host: "{{ groups[elasticstack_elasticsearch_group_name][0] | default('') }}"

roles/elasticstack/meta/argument_specs.yml

@@ -81,6 +81,41 @@ argument_specs:
         default: kibana
         description: Inventory group that holds the Kibana hosts.
 
+      elasticstack_address_mode:
+        type: str
+        default: inventory
+        choices:
+          - inventory
+          - fqdn
+          - hostname
+          - ip
+        description: >-
+          How the Ansible roles of this collection address a host of the elasticsearch,
+          logstash or kibana inventory group when they configure the connections to it.
+          "inventory" uses the host's name in the inventory, "fqdn" its fully qualified
+          domain name, "hostname" its short host name and "ip" the IPv4 address of its
+          network card with the default route; the last three come from the facts of that
+          host. Each host is addressed in the mode set for it, so a value in the group_vars
+          or host_vars of some hosts changes only how those hosts are addressed. The Ansible
+          roles read the value from the inventory or from extra vars; a value in the vars of
+          a play or a role parameter stops the run with an error. The certificates the
+          collection creates contain the IP, the short name, the FQDN and the inventory
+          name, so every mode passes the TLS hostname check. Host lists you set yourself,
+          such as beats_elasticsearch_hosts, are used as they are. The setting does not
+          change discovery.seed_hosts, through which the Elasticsearch nodes find each other.
+
+      elasticstack_node_address:
+        type: str
+        description: >-
+          Address the Ansible roles of this collection use for this host when they configure
+          the connections to it, an IPv4 address or a DNS name that resolves to this host.
+          Set it in the host_vars of that host. It takes precedence over
+          elasticstack_address_mode, for example to reach a host with several network cards
+          through the IP of a particular one. It is also added to the certificate of an
+          Elasticsearch host and to the certificate a Logstash host presents to the Beats. A
+          certificate that already exists gets it only when it is renewed, with the tag
+          renew_es_cert or renew_logstash_cert. It does not change discovery.seed_hosts.
+
       # ----- Ports -----
       elasticstack_elasticsearch_http_port:
         type: int

roles/elasticstack/tasks/main.yml

@@ -1,4 +1,20 @@
 ---
+# The other hosts read this host's elasticstack_address_mode through hostvars, which holds neither
+# play vars nor role parameters. Stop instead of silently ignoring a value set there. Outside the
+# block below, so it runs on every start of a role and not only once per host; the roles therefore
+# import this role without a when.
+- name: Check that elasticstack_address_mode comes from the inventory
+  ansible.builtin.assert:
+    that:
+      # 'inventory' is the default from defaults/main.yml, hostvars does not see role defaults
+      - elasticstack_address_mode == (hostvars[inventory_hostname].elasticstack_address_mode | default('inventory'))
+    fail_msg: >-
+      elasticstack_address_mode has to be set in the inventory (group_vars, host_vars) or with -e.
+      The other hosts read it through hostvars, which does not contain the vars of a play or role
+      parameters, so a value set there would be ignored.
+    quiet: true
+  tags: always
+
 - name: Include elasticstack role
   when: not hostvars[inventory_hostname]._elasticstack_role_imported | default(false) # Ensure the global meta role is only called once for each host
   block:
@@ -22,6 +38,9 @@
     - name: Install common packages and dependencies
       ansible.builtin.import_tasks: packages.yml
 
+    - name: Build the addresses of the Elastic Stack hosts
+      ansible.builtin.import_tasks: node_address.yml
+
     - name: Register elasticstack role has run for that host
       ansible.builtin.set_fact:
         _elasticstack_role_imported: true