diff --git a/calico-enterprise/getting-started/install-on-clusters/kubernetes/options-install.mdx b/calico-enterprise/getting-started/install-on-clusters/kubernetes/options-install.mdx index 5998010b43..9fcf4d0285 100644 --- a/calico-enterprise/getting-started/install-on-clusters/kubernetes/options-install.mdx +++ b/calico-enterprise/getting-started/install-on-clusters/kubernetes/options-install.mdx @@ -29,7 +29,7 @@ Each API in the operator.tigera.io API group configures the installation of a di | **Installation** | The foundation of every deployment is the **Installation** resource. It deploys core networking and network policy enforcement features to the cluster, and is required for all other resources to function. | | **APIServer** | Installs the Tigera API server, which enables access to the full set of [$[prodname] APIs](../../../reference/installation/api.mdx). | | **LogStorage** | Configures installation of storage for $[prodname] flow and DNS logs. | -| **LogCollector** | Configures collection of those logs from the cluster, including optional configuration to backup logs in additional stores. | +| **LogCollector** | Configures collection of those logs from the cluster, including optional configuration to backup logs in additional stores or export them over OpenTelemetry. | | **IntrusionDetection** | Configures the $[prodname] intrusion detection feature. | | **Manager** | Installs the $[prodname] web console. | | **ManagementClusterConnection** | Configures a connection to the management cluster for managed clusters. | diff --git a/calico-enterprise/getting-started/install-on-clusters/windows-calico/flowlogs.mdx b/calico-enterprise/getting-started/install-on-clusters/windows-calico/flowlogs.mdx index 84a4490f06..65122d1b78 100644 --- a/calico-enterprise/getting-started/install-on-clusters/windows-calico/flowlogs.mdx +++ b/calico-enterprise/getting-started/install-on-clusters/windows-calico/flowlogs.mdx @@ -43,5 +43,5 @@ $[prodnameWindows] flow logs are enabled and configured the same way as [Flow lo - [Configure RBAC for Elasticsearch logs](../../../observability/elastic/rbac-elasticsearch.mdx) - [Configure flow log aggregation](../../../observability/elastic/flow/aggregation.mdx) - [Log storage recommendations](../../../operations/logstorage/log-storage-recommendations.mdx) -- [Archive logs](../../../observability/elastic/archive-storage.mdx) +- [Archive logs](../../../observability/archive-storage.mdx) - [Log collection options](../../../reference/installation/api.mdx#logcollectorspec) diff --git a/calico-enterprise/observability/elastic/archive-storage.mdx b/calico-enterprise/observability/archive-storage.mdx similarity index 66% rename from calico-enterprise/observability/elastic/archive-storage.mdx rename to calico-enterprise/observability/archive-storage.mdx index 803c18448e..907a2fd4f7 100644 --- a/calico-enterprise/observability/elastic/archive-storage.mdx +++ b/calico-enterprise/observability/archive-storage.mdx @@ -1,5 +1,5 @@ --- -description: Forward Calico Enterprise flow, DNS, audit, and L7 logs to Syslog, Splunk, or Amazon S3 to retain compliance data beyond in-cluster Elasticsearch retention. +description: Forward Calico Enterprise flow, DNS, audit, and L7 logs to Syslog, Splunk, Amazon S3, or an OpenTelemetry (OTLP) backend to retain compliance data beyond in-cluster Elasticsearch retention. --- # Archive logs @@ -9,11 +9,11 @@ import TabItem from '@theme/TabItem'; ## Big picture -Archive $[prodname] logs to SIEMs like Syslog, Splunk, or Amazon S3 to meet compliance storage requirements. +Archive $[prodname] logs to SIEMs like Syslog, Splunk, Amazon S3, or any backend that accepts OpenTelemetry (OTLP) to meet compliance storage requirements. ## Value -Archiving your $[prodname] Elasticsearch logs to storage services like Amazon S3, Syslog, or Splunk are reliable +Archiving your $[prodname] Elasticsearch logs to storage services like Amazon S3, Syslog, Splunk, or an OpenTelemetry backend are reliable options for maintaining and consolidating your compliance data long term. ## Before you begin @@ -23,11 +23,12 @@ options for maintaining and consolidating your compliance data long term. - Syslog - flow, dns, idsevents, audit - Splunk - flow, audit, dns - Amazon S3 - l7, flow, dns, audit +- OpenTelemetry (OTLP) - flow, dns, audit, and component metrics :::note If you're archiving logs for a system that includes Kubernetes clusters and non-cluster hosts or VMs, the default configuration archives all logs (both cluster and non-cluster) together. For information on how to configure this behaviour, see [Control which hosts have their logs archived](#control-which-hosts-have-their-logs-archived). -Non-cluster hosts only generate a subset of the log types generated by Kubernetes cluster hosts. For more information, see the [non-cluster hosts documentation](../../getting-started/bare-metal/about). +Non-cluster hosts only generate a subset of the log types generated by Kubernetes cluster hosts. For more information, see the [non-cluster hosts documentation](../getting-started/bare-metal/about). ::: ## How to @@ -53,8 +54,8 @@ Because $[prodname] and Kubernetes logs are integral to $[prodname] diagnostics, -n tigera-operator ``` -3. Update the [LogCollector](../../reference/installation/api.mdx#logcollector) - resource named, `tigera-secure` to include an [S3 section](../../reference/installation/api.mdx#s3storespec) +3. Update the [LogCollector](../reference/installation/api.mdx#logcollector) + resource named, `tigera-secure` to include an [S3 section](../reference/installation/api.mdx#s3storespec) with your information noted from above. Example: @@ -104,8 +105,8 @@ Because $[prodname] and Kubernetes logs are integral to $[prodname] diagnostics, -1. Update the [LogCollector](../../reference/installation/api.mdx#logcollector) - resource named `tigera-secure` to include a [Syslog section](../../reference/installation/api.mdx#syslogstorespec) +1. Update the [LogCollector](../reference/installation/api.mdx#logcollector) + resource named `tigera-secure` to include a [Syslog section](../reference/installation/api.mdx#syslogstorespec) with your syslog information. Example: ```yaml @@ -132,7 +133,7 @@ Because $[prodname] and Kubernetes logs are integral to $[prodname] diagnostics, kubectl edit logcollector tigera-secure ``` 2. You can control which types of $[prodname] log data you would like to send to syslog. - The [Syslog section](../../reference/installation/api.mdx#syslogstorespec) + The [Syslog section](../reference/installation/api.mdx#syslogstorespec) contains a field called `logTypes` which allows you to list which log types you would like to include. The allowable log types are: @@ -141,11 +142,11 @@ Because $[prodname] and Kubernetes logs are integral to $[prodname] diagnostics, - Flows - IDSEvents - Refer to the [Syslog section](../../reference/installation/api.mdx#syslogstorespec) for more details on what data each log type represents. + Refer to the [Syslog section](../reference/installation/api.mdx#syslogstorespec) for more details on what data each log type represents. :::note - The log type `IDSEvents` is only supported for a cluster that has [LogStorage](../../reference/installation/api.mdx#logstorage) configured. It is because intrusion detection event data is pulled from the corresponding LogStorage datastore directly. + The log type `IDSEvents` is only supported for a cluster that has [LogStorage](../reference/installation/api.mdx#logstorage) configured. It is because intrusion detection event data is pulled from the corresponding LogStorage datastore directly. ::: @@ -153,7 +154,7 @@ Because $[prodname] and Kubernetes logs are integral to $[prodname] diagnostics, **TLS configuration** -3. You can enable TLS option for syslog forwarding by including the "encryption" option in the [Syslog section](../../reference/installation/api.mdx#syslogstorespec). +3. You can enable TLS option for syslog forwarding by including the "encryption" option in the [Syslog section](../reference/installation/api.mdx#syslogstorespec). ```yaml apiVersion: operator.tigera.io/v1 @@ -192,9 +193,7 @@ Because $[prodname] and Kubernetes logs are integral to $[prodname] diagnostics, -**Support** In this release, only [Splunk Enterprise](https://www.splunk.com/en_us/products/splunk-enterprise.html) is supported. - -$[prodname] uses Splunk's **HTTP Event Collector** to send data to Splunk server. To copy the flow, audit, and dns logs to Splunk, follow these steps: +$[prodname] uses Splunk's **HTTP Event Collector** (HEC) to send data to Splunk, so you can use any Splunk deployment that provides an HEC endpoint, such as Splunk Enterprise or Splunk Cloud Platform. To copy the flow, audit, and dns logs to Splunk, follow these steps: 1. Create a HTTP Event Collector token by following the steps listed in Splunk's documentation for your specific Splunk version. Here is the link to do this for [Splunk version 8.0.0](https://docs.splunk.com/Documentation/Splunk/8.0.0/Data/UsetheHTTPEventCollector). @@ -208,9 +207,9 @@ $[prodname] uses Splunk's **HTTP Event Collector** to send data to Splunk server ``` 3. Update the - [LogCollector](../../reference/installation/api.mdx#logcollector) + [LogCollector](../reference/installation/api.mdx#logcollector) resource named `tigera-secure` to include - a [Splunk section](../../reference/installation/api.mdx#splunkstorespec) + a [Splunk section](../reference/installation/api.mdx#splunkstorespec) with your Splunk information. Example: @@ -245,12 +244,70 @@ $[prodname] uses Splunk's **HTTP Event Collector** to send data to Splunk server kubectl create configmap splunk-ca --from-file=tls.crt -n tigera-operator ``` + + + +$[prodname] sends logs over the OpenTelemetry Protocol (OTLP) through an OpenTelemetry Collector that the operator deploys and manages for you. +Your backend must accept OTLP over `https`, using gRPC or HTTP. + +1. If your backend needs an API key or token, create a Secret for it in the `tigera-operator` namespace. + + ```bash + kubectl create secret generic otel-backend-credentials \ + --from-literal=api-key= \ + -n tigera-operator + ``` + +2. Update the [LogCollector](../reference/installation/api.mdx#logcollector) + resource named `tigera-secure` to include an [openTelemetry section](../reference/installation/api.mdx#opentelemetryspec). + Unlike the other destinations, it goes directly under `spec`, not under `additionalStores`. + Example: + + ```yaml + apiVersion: operator.tigera.io/v1 + kind: LogCollector + metadata: + name: tigera-secure + spec: + openTelemetry: + logs: + types: + - Flows + - DNS + - Audit + exporters: + - name: my-backend + endpoint: https://otlp.example.com:4317 + auth: + headers: + - name: api-key + valueFrom: + secretKeyRef: + name: otel-backend-credentials + key: api-key + ``` + + This can be done during installation by editing the custom-resources.yaml + by applying it, or after installation by editing the resource with the command: + + ```bash + kubectl edit logcollector tigera-secure + ``` + +For metrics export, a private CA or mutual TLS, sending to more than one backend, and sizing the Collector, see [Export logs and metrics with OpenTelemetry](opentelemetry.mdx). + +:::note + +`hostScope` does not apply to OpenTelemetry export. Flow and DNS logs from both cluster hosts and non-cluster hosts are exported. + +::: + ### Control which hosts have their logs archived -By default, logs are archived from both Kubernetes cluster hosts and [non-cluster hosts](../../getting-started/bare-metal/about). To archive logs for only non-cluster hosts or VMs, use the [`hostScope`](../../reference/installation/api.mdx#additionallogstorespec) field when you set your additional storage spec on the `LogCollector` resource. +By default, logs are archived from both Kubernetes cluster hosts and [non-cluster hosts](../getting-started/bare-metal/about). To archive logs for only non-cluster hosts or VMs, use the [`hostScope`](../reference/installation/api.mdx#additionallogstorespec) field when you set your additional storage spec on the `LogCollector` resource. ```yaml title="Example spec for Splunk" apiVersion: operator.tigera.io/v1 diff --git a/calico-enterprise/observability/elastic/flow/aggregation.mdx b/calico-enterprise/observability/elastic/flow/aggregation.mdx index 84e48781d4..aa20f7547d 100644 --- a/calico-enterprise/observability/elastic/flow/aggregation.mdx +++ b/calico-enterprise/observability/elastic/flow/aggregation.mdx @@ -120,6 +120,6 @@ The underlying reason for this overlap is a dependency on Linux conntrack, which ## Additional resources -- [Archive logs to storage](../archive-storage.mdx) +- [Archive logs to storage](../../archive-storage.mdx) - [Configure RBAC for Elasticsearch logs](../rbac-elasticsearch.mdx) - [Configure data retention](../retention.mdx) diff --git a/calico-enterprise/observability/elastic/flow/filtering.mdx b/calico-enterprise/observability/elastic/flow/filtering.mdx index b1f6fd68f6..a8d6186e9c 100644 --- a/calico-enterprise/observability/elastic/flow/filtering.mdx +++ b/calico-enterprise/observability/elastic/flow/filtering.mdx @@ -111,5 +111,5 @@ The operator inserts the filters inline into the log collector configuration and ## Additional resources - [Flow log aggregation](aggregation.mdx) -- [Archive logs to storage](../archive-storage.mdx) +- [Archive logs to storage](../../archive-storage.mdx) - [Configure RBAC for Elasticsearch logs](../rbac-elasticsearch.mdx) diff --git a/calico-enterprise/observability/elastic/overview.mdx b/calico-enterprise/observability/elastic/overview.mdx index b0675ae8c9..c0d42c5dfd 100644 --- a/calico-enterprise/observability/elastic/overview.mdx +++ b/calico-enterprise/observability/elastic/overview.mdx @@ -19,7 +19,7 @@ Workloads and policies are highly dynamic. To troubleshoot Kubernetes clusters, - Collect/archive logs or subset of logs - Log aggregation for high-volume logs - Configure data retention settings to manage cluster disk space -- Integration with third-party tools like Amazon S3, Syslog, Splunk +- Integration with third-party tools like Amazon S3, Syslog, Splunk, and any backend that accepts OpenTelemetry (OTLP) ## Concepts @@ -45,7 +45,7 @@ Because of their high-volume, flow and DNS logs support aggregation. ### Default log configuration and security -$[prodname] automatically installs Fluent Bit on all nodes and collects flow, audit, and DNS logs. You can configure additional destinations like Amazon S3, Syslog, Splunk. +$[prodname] automatically installs Fluent Bit on all nodes and collects flow, audit, and DNS logs. You can configure additional destinations like Amazon S3, Syslog, Splunk, or an [OpenTelemetry backend](../opentelemetry.mdx). Fluent Bit buffers logs on each node's filesystem, under `/var/log/calico/calico-fluent-bit/storage`, so chunks that have not been delivered yet survive a log collector restart. Each destination gets its own buffer with its own cap: 500 MB for flow logs and for each additional destination, and 100 MB for each of the other log types. Account for that disk usage when you size your nodes. Fluent Bit retries a chunk until it is delivered, so a destination that stays unreachable long enough to fill its buffer starts losing data for that destination only; the others keep shipping. @@ -60,7 +60,7 @@ kubectl patch logcollector tigera-secure --type=merge \ --patch='{"spec":{"ingestionCompression":"Zstd"}}' ``` -The setting applies only to batches in transit. It does not change the compression of the rotated log files on disk, the [additional destinations](archive-storage.mdx), or the OpenTelemetry exporters. +The setting applies only to batches in transit. It does not change the compression of the rotated log files on disk, the [additional destinations](../archive-storage.mdx), or the OpenTelemetry exporters. ### RBAC and log access @@ -98,5 +98,5 @@ verbs: ['get'] - [Audit logs](audit-overview.mdx) - [BGP logs](bgp.mdx) - [DNS logs](dns/dns-logs.mdx) -- [Archive logs](archive-storage.mdx) +- [Archive logs](../archive-storage.mdx) - [Log collection options](../../reference/installation/api.mdx#logcollectorspec) diff --git a/calico-enterprise/observability/index.mdx b/calico-enterprise/observability/index.mdx index b23276a22f..d233648d30 100644 --- a/calico-enterprise/observability/index.mdx +++ b/calico-enterprise/observability/index.mdx @@ -25,13 +25,19 @@ See what's going on in your cluster with network observability tools and detaile - +## Export logs and metrics + + + + + + ## Flow logs diff --git a/calico-enterprise/observability/opentelemetry.mdx b/calico-enterprise/observability/opentelemetry.mdx new file mode 100644 index 0000000000..525786157e --- /dev/null +++ b/calico-enterprise/observability/opentelemetry.mdx @@ -0,0 +1,305 @@ +--- +description: Export Calico Enterprise flow, DNS, and audit logs and component metrics over OTLP to any OpenTelemetry-compatible backend with an operator-managed OpenTelemetry Collector. +--- + +# Export logs and metrics with OpenTelemetry + +You can send $[prodname] logs and metrics to any backend that accepts the OpenTelemetry Protocol (OTLP), such as an observability platform, a SIEM, or your own OpenTelemetry Collector. +When you turn this on, the Tigera Operator deploys an OpenTelemetry Collector in your cluster and manages its configuration, certificates, and network policy for you. +You choose what to export and where to send it; you don't write any Collector configuration. + +## About OpenTelemetry export + +You configure OpenTelemetry export in the `openTelemetry` section of the [LogCollector](../reference/installation/api.mdx#logcollector) resource named `tigera-secure`. +From that section, the operator creates an OpenTelemetry Collector StatefulSet named `otel-collector` in the `calico-system` namespace. + +You can export two kinds of data: + +- **Logs.** Fluent Bit sends the log types you choose to the Collector over mutual TLS, and the Collector forwards them to your exporters. You can export `Flows`, `DNS`, and `Audit` logs. Flow and DNS logs from [non-cluster hosts](../getting-started/bare-metal/about) are included. Each record carries a `service.name` resource attribute of `flows`, `dns`, or `audit`, so you can route or filter by log type in your backend. `service.name` is the only resource attribute. + + `Audit` covers both $[prodname] audit logs and Kubernetes API server audit logs, and both carry `service.name` `audit`. + To tell them apart, filter on `objectRef.apiGroup`: $[prodname] resources use `projectcalico.org`. + If your Kubernetes audit policy also logs `projectcalico.org` requests, the same change can appear in both logs, and this filter can't separate them. +- **Metrics.** The Collector federates every metric that the in-cluster $[prodname] Prometheus already scrapes, including metrics from $[noderunning], Typha, kube-controllers, Fluent Bit, Elasticsearch, and the operator. Series keep their original `job` and `instance` labels. + +Exported logs and metrics don't identify the cluster they come from. +Flow and DNS records have a `host` field with the node name, audit records have no node field, and metrics have no cluster label. +If several clusters export to the same backend, give each cluster an exporter that your backend can tell apart, such as a different endpoint or different credentials. + +Everything you export goes to every exporter that you configure. +Logs that go to OTLP still go to $[prodname]'s log storage and to any [additional destinations](archive-storage.mdx) as usual. + +## Before you begin + +- Each cluster that you configure must run $[prodname] v3.24.0-3.0 (3.24 early preview 3) or later. + Earlier releases don't have the `openTelemetry` field. + On an earlier release, `kubectl apply`, `kubectl edit`, and `kubectl apply --server-side` reject the change with an `unknown field` or `field not declared in schema` error. + `kubectl patch`, and other clients that don't ask for strict validation, only print `Warning: unknown field "spec.openTelemetry"` and save the resource without the field, so nothing is exported and no status reports an error. + After you upgrade the cluster, apply the configuration again. +- Your $[prodname] license must include the OpenTelemetry Collector feature. + If the license doesn't include it, or if it expires, the operator removes the Collector and export stops. +- To export metrics, the in-cluster $[prodname] Prometheus must be running. + For more information, see [Configure Prometheus](../operations/monitor/prometheus/configure-prometheus.mdx). +- Your backend must accept OTLP over `https`, using either gRPC or HTTP. + The Collector never sends telemetry or credentials over plain HTTP. +- If an exporter's endpoint is a domain name rather than an IP address, your license must also include egress access control. + The operator restricts the Collector's outbound traffic to your exporters with a network policy, and it can only write a rule for a domain name when that feature is available. +- Logs are exported from Linux nodes only. + +## Configure an exporter + +1. If your backend needs an API key or token, create a Secret for it in the `tigera-operator` namespace. + + ```bash + kubectl create secret generic otel-backend-credentials \ + --from-literal=api-key= \ + -n tigera-operator + ``` + + The Collector sends the value exactly as stored. + If your backend expects a bearer token in the `Authorization` header, include the `Bearer ` prefix in the Secret value. + +2. Add an `openTelemetry` section to the LogCollector resource. + + ```bash + kubectl edit logcollector tigera-secure + ``` + + The following example exports all three log types and component metrics to a single backend over gRPC: + + ```yaml + apiVersion: operator.tigera.io/v1 + kind: LogCollector + metadata: + name: tigera-secure + spec: + openTelemetry: + logs: + types: + - Flows + - DNS + - Audit + metrics: + state: Enabled + exporters: + - name: my-backend + endpoint: https://otlp.example.com:4317 + protocol: grpc + auth: + headers: + - name: api-key + valueFrom: + secretKeyRef: + name: otel-backend-credentials + key: api-key + ``` + + You must configure at least one exporter, and you must turn on at least one data source: list at least one log type, or set `metrics.state` to `Enabled`. + + | Field | Description | + | --- | --- | + | `logs.types` | The log types to export. Supported values: `Flows`, `DNS`, `Audit`. | + | `metrics.state` | Set to `Enabled` to export component metrics. Default: `Disabled`. | + | `exporters[].name` | A unique name for the exporter. Must be a valid DNS label: lowercase letters, numbers, and hyphens, up to 63 characters. | + | `exporters[].endpoint` | The OTLP endpoint URL. Must start with `https://`. | + | `exporters[].protocol` | The OTLP transport: `grpc` or `http`. Default: `grpc`. | + | `exporters[].auth.headers` | Request headers to send with every export, each read from a key in a Secret in the `tigera-operator` namespace. | + | `exporters[].tls` | Custom certificate authority and client certificate for this exporter. See [Configure TLS for an exporter](#configure-tls-for-an-exporter). | + + For the full API, see [OpenTelemetrySpec](../reference/installation/api.mdx#opentelemetryspec). + + :::note + + With `protocol: http`, set `endpoint` to the base URL of your backend's OTLP receiver. + The Collector appends the signal path, such as `/v1/logs` or `/v1/metrics`, to the URL for you. + + ::: + +3. Check that the operator deployed the Collector. + + ```bash + kubectl get tigerastatus log-collector-otel + ``` + + When the Collector is running, `AVAILABLE` is `True`. + If it's `DEGRADED`, run `kubectl get tigerastatus log-collector-otel -o yaml` to see the reason, such as a Secret or ConfigMap that an exporter refers to but that doesn't exist yet. + + `AVAILABLE` only means that the Collector process is running. + It stays `True` when exports fail, for example when the backend is down, its certificate doesn't verify, or the endpoint isn't an OTLP receiver. + To confirm that data is reaching your backend, see [Check that data is reaching your backend](#check-that-data-is-reaching-your-backend). + +## Check that data is reaching your backend + +The Collector reports its own metrics, and the in-cluster $[prodname] Prometheus scrapes them. +Each series has an `exporter` label of `otlp_grpc/` or `otlp_http/`, where `` is the exporter's name. + +| Metric | What it tells you | +| --- | --- | +| `otelcol_exporter_queue_size` | Batches waiting to be sent. It grows as soon as the backend stops accepting data. | +| `otelcol_exporter_send_failed_log_records` | Log records that were dropped after the Collector stopped retrying them. | +| `otelcol_exporter_send_failed_metric_points` | Metric points that were dropped after the Collector stopped retrying them. | +| `otelcol_exporter_sent_log_records`, `otelcol_exporter_sent_metric_points` | Data that the backend accepted. | + +The Collector retries a failed export for about 5 minutes before it drops the data and counts it as failed. +An outage shorter than that shows only as a growing queue, so alert on `otelcol_exporter_queue_size` as well as on the failure counters. + +The Collector also logs each failed attempt, at `info` level: + +```bash +kubectl logs -n calico-system otel-collector-0 | grep "Exporting failed" +``` + +## Export from managed clusters + +In a [multi-cluster management](../multicluster/index.mdx) setup, configure OpenTelemetry export on each cluster's own LogCollector resource, from that cluster. +Nothing is configured from the management cluster, and the management cluster runs a Collector only if you configure one there. +A managed cluster can run an earlier release than the management cluster, so upgrade each managed cluster to a release that supports OpenTelemetry export before you set `openTelemetry` on it. +See [Before you begin](#before-you-begin). + +A managed cluster's Collector sends directly to your backend, not through the management cluster. +Each managed cluster needs its own network route to the backend, and its own Secrets and ConfigMaps for credentials and certificates in its `tigera-operator` namespace. + +## Send data to more than one backend + +Add an entry to `exporters` for each backend. +Each exporter has its own protocol, credentials, and TLS settings, and each one receives all of the logs and metrics that you export. + +```yaml +apiVersion: operator.tigera.io/v1 +kind: LogCollector +metadata: + name: tigera-secure +spec: + openTelemetry: + logs: + types: + - Flows + - DNS + exporters: + - name: observability + endpoint: https://otlp.example.com:4317 + - name: siem + endpoint: https://siem.example.internal:4318 + protocol: http + auth: + headers: + - name: Authorization + valueFrom: + secretKeyRef: + name: siem-credentials + key: token +``` + +## Configure TLS for an exporter + +By default, the Collector verifies each exporter's server certificate against the system's public certificate authorities. +Certificate verification is never turned off. + +**Use a private certificate authority** + +If your backend's certificate is signed by a private certificate authority (CA), create a ConfigMap in the `tigera-operator` namespace with the CA certificate under the `tls.crt` key: + +```bash +kubectl create configmap siem-ca --from-file=tls.crt= -n tigera-operator +``` + +Then refer to it from the exporter: + +```yaml +exporters: + - name: siem + endpoint: https://siem.example.internal:4317 + tls: + caConfigMapName: siem-ca +``` + +When you set `caConfigMapName`, the Collector trusts only that CA for this exporter. +It no longer trusts the system's public CAs for this exporter. + +**Use mutual TLS** + +If your backend requires a client certificate, create a TLS Secret in the `tigera-operator` namespace with the client certificate and key under the `tls.crt` and `tls.key` keys: + +```bash +kubectl create secret tls siem-client-cert \ + --cert= \ + --key= \ + -n tigera-operator +``` + +Then refer to it from the exporter: + +```yaml +exporters: + - name: siem + endpoint: https://siem.example.internal:4317 + tls: + caConfigMapName: siem-ca + clientCertSecretName: siem-client-cert +``` + +TLS settings apply to one exporter only, so each backend can use a different CA and a different client identity. + +When you update a CA ConfigMap, client certificate Secret, or credentials Secret, the operator restarts the Collector to pick up the change. + +## Customize the Collector deployment + +The Collector runs as a single replica on the same nodes as other $[prodname] control plane components. +By default, the `otel-collector` container requests 128Mi of memory and is limited to 512Mi. +The Collector refuses incoming data when it uses about 60% of its memory limit, and forces garbage collection at about 80%. + +If you export a high volume of logs, raise the memory limit with `openTelemetryCollectorStatefulSet`. +Both thresholds scale with the limit you set. + +```yaml +apiVersion: operator.tigera.io/v1 +kind: LogCollector +metadata: + name: tigera-secure +spec: + openTelemetry: + # ... + openTelemetryCollectorStatefulSet: + spec: + template: + spec: + containers: + - name: otel-collector + resources: + requests: + memory: 256Mi + limits: + memory: 1Gi +``` + +You can also set `affinity`, `nodeSelector`, and `tolerations` for the Collector pod. +For the full list of fields, see [OpenTelemetryCollectorStatefulSet](../reference/installation/api.mdx#opentelemetrycollectorstatefulset). + +## Restarts and data loss + +The Collector is a single replica, and it holds data waiting to be sent in memory. +When it restarts, for example after you change its configuration or credentials, or when its node is drained, logs that Fluent Bit sends during the restart can be lost. +Fluent Bit retries a failed send to the Collector once, and then drops that batch. +This affects only the OpenTelemetry copy: the same logs still reach $[prodname] log storage. + +When you change `logs.types`, the operator restarts Fluent Bit on every node. +Fluent Bit also ships logs to $[prodname] log storage, so that path restarts too. + +## Turn off OpenTelemetry export + +Remove the `openTelemetry` section from the LogCollector resource: + +```bash +kubectl patch logcollector tigera-secure --type=json \ + -p='[{"op": "remove", "path": "/spec/openTelemetry"}]' +``` + +The operator removes the Collector and the resources it created for it, including its copies of your exporter certificates and credentials. +It doesn't delete the Secrets and ConfigMaps that you created in the `tigera-operator` namespace. + +## Additional resources + +- [OpenTelemetrySpec API reference](../reference/installation/api.mdx#opentelemetryspec) +- [Archive logs](archive-storage.mdx) +- [Prometheus metrics](../operations/monitor/metrics/index.mdx) +- [OpenTelemetry Protocol specification](https://opentelemetry.io/docs/specs/otlp/) diff --git a/calico-enterprise/release-notes/index.mdx b/calico-enterprise/release-notes/index.mdx index 02a169af7c..843a7c059b 100644 --- a/calico-enterprise/release-notes/index.mdx +++ b/calico-enterprise/release-notes/index.mdx @@ -80,7 +80,7 @@ The log collector changes from Fluentd to Fluent Bit. The `tigera-fluentd` names - **Rewrite your log filters before you upgrade.** Flow and DNS log filters are no longer read from the `fluentd-filters` ConfigMap, and Fluentd filter syntax is not translated for you. Recreate your filters as Fluent Bit filter lists in a `fluent-bit-filters` ConfigMap. Until you do, logs ship unfiltered. See [Filter flow logs](../observability/elastic/flow/filtering.mdx) and [Filter DNS logs](../observability/elastic/dns/filtering-dns.mdx). - **Update anything that names the old resources.** Log collector resources move to `calico-system` and are renamed: the DaemonSet and container become `calico-fluent-bit`, or `calico-fluent-bit-windows` on Windows nodes, and the TLS secret becomes `calico-fluent-bit-tls`. Commands such as `kubectl logs -c fluentd` no longer work. - **Re-point your dashboards, alerts, and scrape configuration.** Log collector metrics are now Fluent Bit's own Prometheus metrics, with `fluentbit_` names on plain HTTP port 2020, in place of `fluentd_` names on the mTLS endpoint on port 9081. The metrics services are renamed to `calico-fluent-bit-metrics` and `calico-fluent-bit-metrics-windows`, and the pod label is now `k8s-app: calico-fluent-bit`. See [Elasticsearch and Fluent Bit metrics](../operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx) and [Bring your own Prometheus](../operations/monitor/prometheus/byo-prometheus.mdx). -- **Update tooling that reads archived logs from S3.** The object keys change from a flat layout to one directory per log type, and flows from non-cluster hosts get their own directory. See [Archive logs](../observability/elastic/archive-storage.mdx). +- **Update tooling that reads archived logs from S3.** The object keys change from a flat layout to one directory per log type, and flows from non-cluster hosts get their own directory. See [Archive logs](../observability/archive-storage.mdx). - **Check your EKS log forwarder overrides.** The deployment no longer runs an `eks-log-forwarder-startup` init container, so overrides that target it by name no longer apply. See [Configure resource requests and limits](../reference/component-resources/configure-resources.mdx). ::: @@ -88,7 +88,7 @@ The log collector changes from Fluentd to Fluent Bit. The `tigera-fluentd` names Two defaults also change: - Logs are buffered on each node's filesystem instead of in memory. Logs that have not been delivered yet survive a restart of the collector, and node disk usage grows accordingly. See [Log collection overview](../observability/elastic/overview.mdx). -- When `spec.additionalStores.syslog.packetSize` is unset, the maximum syslog message size is 2048 bytes rather than Fluentd's 1024, so fewer long log lines are truncated. See [Archive logs](../observability/elastic/archive-storage.mdx). +- When `spec.additionalStores.syslog.packetSize` is unset, the maximum syslog message size is 2048 bytes rather than Fluentd's 1024, so fewer long log lines are truncated. See [Archive logs](../observability/archive-storage.mdx). ## Release details diff --git a/sidebars-calico-enterprise.js b/sidebars-calico-enterprise.js index 0258aa52a7..dec9093af8 100644 --- a/sidebars-calico-enterprise.js +++ b/sidebars-calico-enterprise.js @@ -392,6 +392,8 @@ module.exports = { 'observability/kibana', 'observability/packetcapture', 'observability/visualize-traffic', + 'observability/archive-storage', + 'observability/opentelemetry', { type: 'category', label: 'Calico Enterprise logs', @@ -399,7 +401,6 @@ module.exports = { items: [ 'observability/elastic/overview', 'observability/elastic/retention', - 'observability/elastic/archive-storage', 'observability/elastic/rbac-elasticsearch', 'observability/elastic/aggregation', { diff --git a/static/_redirects b/static/_redirects index fb0cbd0496..51cab6c406 100644 --- a/static/_redirects +++ b/static/_redirects @@ -465,3 +465,6 @@ /calico-cloud/next/network-policy/policy-tiers/allow-tigera /calico-cloud/next/network-policy/policy-tiers/calico-system 301 /calico-enterprise/latest/network-policy/policy-tiers/allow-tigera /calico-enterprise/latest/network-policy/policy-tiers/calico-system 301 /calico-enterprise/3.24/network-policy/policy-tiers/allow-tigera /calico-enterprise/3.24/network-policy/policy-tiers/calico-system 301 + +# Archive logs moved out of observability/elastic/ in Calico Enterprise 3.24 EP3 +# FOR 3.24 EP3: /calico-enterprise/3.24/observability/elastic/archive-storage /calico-enterprise/3.24/observability/archive-storage 301