diff --git a/docset.yml b/docset.yml index 31150af647..7823912405 100644 --- a/docset.yml +++ b/docset.yml @@ -393,5 +393,8 @@ subs: fedramp-mod: "FedRAMP Moderate" fedramp-high: "FedRAMP High" fedramp-il5: "FedRAMP IL5" + tsds: "time series data stream" + tsds-cap: "Time series data stream" + tsds-init: "TSDS" release_notes: - product: cloud-serverless \ No newline at end of file diff --git a/manage-data/_snippets/enable-backfill.md b/manage-data/_snippets/enable-backfill.md new file mode 100644 index 0000000000..4e4c40f0ba --- /dev/null +++ b/manage-data/_snippets/enable-backfill.md @@ -0,0 +1,14 @@ +{{es}} can create missing backing indices when you add data that precedes existing time ranges. +To enable this feature, update the cluster settings: + +```console +PUT _cluster/settings +{ + "persistent": { + "data_stream.past_tsdb_index_creation_enabled": true, + "data_streams.past_tsdb_index_interval": "2d" <1> + } +} +``` + +1. By default, each past backing index covers one day of data. Refer to [`data_streams.past_tsdb_index_interval`](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#time-series-data-stream). diff --git a/manage-data/data-store/data-streams.md b/manage-data/data-store/data-streams.md index 4743717dd9..c2611d8d52 100644 --- a/manage-data/data-store/data-streams.md +++ b/manage-data/data-store/data-streams.md @@ -15,7 +15,7 @@ products: A data stream acts as a layer of abstraction over a set of indices that are optimized for storing append-only time series data. It stores data across multiple backing indices while giving you a single named resource to use for requests. Data streams are well-suited for logs, events, metrics, and other continuously generated data. -You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream’s data. You can use [{{ilm}} ({{ilm-init}})](../lifecycle/index-lifecycle-management.md) to automate the management of these backing indices. For example, you can use {{ilm-init}} to automatically move older backing indices to less expensive hardware and delete unneeded indices. {{ilm-init}} can help you reduce costs and overhead as your data grows. +You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream's data. You can use [{{ilm}} ({{ilm-init}})](../lifecycle/index-lifecycle-management.md) to automate the management of these backing indices. For example, you can use {{ilm-init}} to automatically move older backing indices to less expensive hardware and delete unneeded indices. {{ilm-init}} can help you reduce costs and overhead as your data grows. You can also use a [data stream lifecycle](../lifecycle/data-stream.md) to automate lifecycle management according to your retention requirements. @@ -36,7 +36,7 @@ To determine whether you should use a data stream for your data, you should cons * You mostly perform indexing requests, with occasional updates and deletes. * You index documents without an `_id`, or when indexing documents with an explicit `_id` you expect first-write-wins behavior. -For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn’t fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See the tutorial [](../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) for more information. +For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn't fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See the tutorial [](../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) for more information. Keep in mind that some features such as [Time Series Data Streams (TSDS)](../data-store/data-streams/time-series-data-stream-tsds.md) and [data stream lifecycles](../lifecycle/data-stream.md) require a data stream. @@ -47,7 +47,6 @@ Keep in mind that some features such as [Time Series Data Streams (TSDS)](../dat [LogsDB](data-streams/logs-data-stream.md) and [time series (TSDS)](data-streams/time-series-data-stream-tsds.md) index modes are not available in {{vectordb}}; use an {{es}}, Observability, or Security project type for those workloads. ::: - ## Backing indices [backing-indices] A data stream consists of one or more [hidden](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-hidden), auto-generated backing indices. @@ -56,15 +55,14 @@ A data stream consists of one or more [hidden](elasticsearch://reference/elastic :alt: data streams diagram ::: -A data stream requires a matching [index template](templates.md). The template contains the mappings and settings used to configure the stream’s backing indices and defines the {{ilm-init}} policy that the data stream uses. +A data stream requires a matching [index template](templates.md). The template contains the mappings and settings used to configure the stream's backing indices and defines the {{ilm-init}} policy that the data stream uses. -Every document indexed to a data stream must contain a `@timestamp` field, mapped as a [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) field type. If the index template doesn’t specify a mapping for the `@timestamp` field, {{es}} maps `@timestamp` as a `date` field with default options. +Every document indexed to a data stream must contain a `@timestamp` field, mapped as a [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) field type. If the index template doesn't specify a mapping for the `@timestamp` field, {{es}} maps `@timestamp` as a `date` field with default options. The same index template can be used for multiple data streams. You cannot delete an index template in use by a data stream. The name pattern for the backing indices is an implementation detail and no intelligence should be derived from it. The only invariant the holds is that each data stream generation index will have a unique name. - ## Read requests [data-stream-read-requests] When you submit a read request to a data stream, the stream routes the request to all its backing indices. @@ -73,17 +71,15 @@ When you submit a read request to a data stream, the stream routes the request t :alt: data streams search request ::: - ## Write index [data-stream-write-index] -The most recently created backing index is the data stream’s write index. The stream adds new documents to this index only. +The most recently created backing index is the data stream's write index. The stream adds new documents to this index only. :::{image} /manage-data/images/elasticsearch-reference-data-streams-index-request.svg :alt: data streams index request ::: You cannot add new documents to other backing indices, even by sending requests directly to the index. - You also cannot perform operations on a write index that may hinder indexing, such as: * [Clone]({{es-apis}}operation/operation-indices-clone) @@ -91,10 +87,13 @@ You also cannot perform operations on a write index that may hinder indexing, su * [Shrink]({{es-apis}}operation/operation-indices-shrink) * [Split]({{es-apis}}operation/operation-indices-split) +:::{note} +[Time series data streams (TSDS)](data-streams/time-series-data-stream-tsds.md) are different: they write to the backing index whose time range includes the document's `@timestamp`, not only the latest one. +::: ## Rollover [data-streams-rollover] -A [rollover]({{es-apis}}operation/operation-indices-rollover) creates a new backing index that becomes the stream’s new write index. +A [rollover]({{es-apis}}operation/operation-indices-rollover) creates a new backing index that becomes the stream's new write index. We recommend using [{{ilm-init}}](../lifecycle/index-lifecycle-management.md) to automatically roll over data streams when the write index reaches a specified age or size. If needed, you can also [manually roll over](data-streams/use-data-stream.md#manually-roll-over-a-data-stream) a data stream. @@ -109,21 +108,18 @@ When a backing index is created, the index is named using the following conventi .ds--- ``` -`` is the backing index’s creation date. Backing indices with a higher generation contain more recent data. For example, the `web-server-logs` data stream has a generation of `34`. The stream’s most recent backing index, created on 7 March 2099, is named `.ds-web-server-logs-2099.03.07-000034`. +`` is the backing index's creation date. Backing indices with a higher generation contain more recent data. For example, the `web-server-logs` data stream has a generation of `34`. The stream's most recent backing index, created on 7 March 2099, is named `.ds-web-server-logs-2099.03.07-000034`. -Some operations, such as a [shrink]({{es-apis}}operation/operation-indices-shrink) or [restore](../../deploy-manage/tools/snapshot-and-restore/restore-snapshot.md), can change a backing index’s name. These name changes do not remove a backing index from its data stream. +Some operations, such as a [shrink]({{es-apis}}operation/operation-indices-shrink) or [restore](../../deploy-manage/tools/snapshot-and-restore/restore-snapshot.md), can change a backing index's name. These name changes do not remove a backing index from its data stream. The generation of the data stream can change without a new index being added to the data stream (for example, when an existing backing index is shrunk). This means the backing indices for some generations will never exist. You should not derive any intelligence from the backing indices names. - ## Append-only (mostly) [data-streams-append-only] -Data streams are designed for use cases where existing data is rarely updated. You cannot send update or deletion requests for existing documents directly to a data stream. However, you can still [update or delete documents](data-streams/use-data-stream.md#update-delete-docs-in-a-backing-index) in a data stream by submitting requests directly to the document’s backing index. +Data streams are designed for use cases where existing data is rarely updated. You cannot send update or deletion requests for existing documents directly to a data stream. However, you can still [update or delete documents](data-streams/use-data-stream.md#update-delete-docs-in-a-backing-index) in a data stream by submitting requests directly to the document's backing index. If you need to update a larger number of documents in a data stream, you can use the [update by query](data-streams/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](data-streams/use-data-stream.md#delete-docs-in-a-data-stream-by-query) APIs. ::::{tip} If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See the tutorial [](../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md). :::: - - diff --git a/manage-data/data-store/data-streams/advanced-topics-tsds.md b/manage-data/data-store/data-streams/advanced-topics-tsds.md index 4bee425e2a..45d4e0a479 100644 --- a/manage-data/data-store/data-streams/advanced-topics-tsds.md +++ b/manage-data/data-store/data-streams/advanced-topics-tsds.md @@ -16,6 +16,7 @@ products: This section contains information about advanced concepts and operations for [time series data streams](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md): - [](/manage-data/data-store/data-streams/time-bound-tsds.md) +- [](/manage-data/data-store/data-streams/load-historical-tsds.md) - [](/manage-data/data-store/data-streams/metric-temporality.md) - [](/manage-data/data-store/data-streams/reindex-tsds.md) - [](/manage-data/data-store/data-streams/tsds-ingest-otlp.md) diff --git a/manage-data/data-store/data-streams/load-historical-tsds.md b/manage-data/data-store/data-streams/load-historical-tsds.md new file mode 100644 index 0000000000..873f3f21d7 --- /dev/null +++ b/manage-data/data-store/data-streams/load-historical-tsds.md @@ -0,0 +1,150 @@ +--- +navigation_title: "Load historical data" +applies_to: + stack: ga 9.5 +products: + - id: elasticsearch +--- + +# Load historical data into a {{tsds}} [load-historical-tsds] + +By default, a {{tsds-cap}} ({{tsds-init}}) works well for continuous, near-real-time ingestion. +Only documents with `@timestamp` values that fall inside the time range of existing backing indices are accepted. + +To import historical data into an existing {{tsds-init}}, enable the [data_stream.past_tsdb_index_creation_enabled](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#time-series-data-stream) cluster setting. You can then use the same APIs you use for live data. +{{es}} creates the past backing indices needed to store past documents as they arrive. +The documents must fall within the [eligible write window](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-past-index-creation). This is the period of time between "current" and the data stream retention limit or the first occurrence of a lifecycle action that makes a backing index read-only, whichever occurs first. +Write-time deduplication and {{tsds-init}} storage optimizations apply to historical data the same way they apply to live data. + +:::{note} +Users who trigger past index creation need the `auto_configure` index privilege. +For details, refer to [Secure a {{tsds-init}}](/manage-data/data-store/data-streams/set-up-tsds.md#secure-tsds). +::: + +## Load data within the eligible write window + +For data that falls within the eligible write window, point your migration or replay pipeline at the live {{tsds}}. +{{es}} creates past backing indices as needed. + +This approach works well when you're backfilling recent history alongside live ingestion, such as late-arriving metrics or a short bootstrap period. + +Go to [](/manage-data/data-store/data-streams/set-up-tsds.md) for an example of setting up and loading historical data into a new {{tsds}}. + +## Load data beyond the eligible write window + +You can't load data older than the eligible write window directly into a {{tsds-init}}. +For example, if downsampling makes indices read-only after seven days, you can't backfill eighteen months of history into that same data stream. + +Instead, create a separate historical {{tsds-init}} without a lifecycle, load the data, then add a [data stream lifecycle](/manage-data/lifecycle/data-stream.md) when the load is complete. + +:::::{stepper} +::::{step} Create an index template for the historical data stream + +Use the same mappings as your live {{tsds-init}}, but don't include a lifecycle policy in the template. +For example, use the [create index template]({{es-apis}}operation/operation-indices-put-index-template) API: + +```console +PUT _index_template/metrics-historical +{ + "index_patterns": ["metrics-historical-*"], + "data_stream": {}, + "template": { + "settings": { + "index.mode": "time_series" + }, + "mappings": { + "properties": { + "@timestamp": { "type": "date" }, + "sensor_id": { "type": "keyword", "time_series_dimension": true }, + "temperature": { "type": "half_float", "time_series_metric": "gauge" } + } + } + } +} +``` + +:::: + +::::{step} Create the historical data stream + +Create a data stream with a name that matches the pattern in the index template. +For example, use the [create a data stream]({{es-apis}}operation/operation-indices-create-data-stream) API: + +```console +PUT _data_stream/metrics-historical-2024 +``` + +:::: + +::::{step} Index historical data + +Index historical data into the historical data stream while current data continues flowing into the original {{tsds-init}}. + +:::{important} +Historical data must fit on the target tier as a whole before you enable data stream lifecycle. +If you're importing a large data set, split it into batches. +Each batch should fit within available disk space at indexing time. +::: +:::: + +::::{step} Add data stream lifecycle + +When the load is complete, add a [data stream lifecycle](/manage-data/lifecycle/data-stream.md) to the historical data stream. +For example, use the [update data stream lifecycles]({{es-apis}}operation/operation-indices-put-data-lifecycle) API: + +```console +PUT _data_stream/metrics-historical-2024/_lifecycle +{ + "enabled": true, + "data_retention": "365d", + "downsampling": [ + { + "after": "7d", + "fixed_interval": "10m" + } + ] +} +``` + +Processing begins immediately and creates a backlog of downsampling work. +If you include `data_retention` settings, data stream lifecycle deletes expired backing indices but does not remove the data stream itself. +:::: + +::::{step} Query across both data streams + +Query both streams with a wildcard pattern or a [data stream alias](/manage-data/data-store/aliases.md). +For example, use the [search]({{es-apis}}operation/operation-search) API: + +```console +GET metrics-*/_search +{ + "size": 10, + "sort": [{ "@timestamp": "desc" }] +} +``` + +:::: +::::: +Delete historical data streams manually when their data is no longer needed. + +## Protect the cluster during large loads + +Loading months of historical data can trigger significant storage use, force merge activity, and lifecycle processing in parallel. +Verify that your cluster has enough available resources before you start. + +When you enable a lifecycle on a data stream with many indices that qualify for downsampling, data stream lifecycle can queue multiple downsampling operations at once. +To limit concurrent downsampling per data stream, configure the [`data_streams.lifecycle.downsampling.max_indices_in_progress`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-downsampling-max-indices-in-progress) cluster setting. +For details, refer to [Downsample with a data stream lifecycle](/manage-data/data-store/data-streams/run-downsampling.md#downsample-with-a-data-stream-lifecycle). + +## Limitations + +Backfill and creation of past indices have the following limitations: + +- System data streams are excluded. +- {{ccr-cap}} ({{ccr-init}}) follower data streams rely on the leader data stream, so you can't backfill follower streams directly. + +## Next steps + +- [Time-bound indices](/manage-data/data-store/data-streams/time-bound-tsds.md) for eligible write window and past index creation details +- [Downsampling a time series data stream](/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) to reduce storage after historical data ages +- [Reindex a time series data stream](/manage-data/data-store/data-streams/reindex-tsds.md) if you need to copy data to a new {{tsds-init}} instead of backfilling in place diff --git a/manage-data/data-store/data-streams/quickstart-tsds.md b/manage-data/data-store/data-streams/quickstart-tsds.md index e70a901a89..14a0f16c44 100644 --- a/manage-data/data-store/data-streams/quickstart-tsds.md +++ b/manage-data/data-store/data-streams/quickstart-tsds.md @@ -150,7 +150,7 @@ A successful request returns `"errors": false` and a `create` item for each docu ::: :::{tip} -If you get an error about timestamp values, check the error response for the valid timestamp range and run the bulk API again with appropriate `@timestamp` values. +If you get an error about timestamp values, check the error response for the valid timestamp range. For more details, refer to [Accepted time range for adding data](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-accepted-time-range). ::: diff --git a/manage-data/data-store/data-streams/run-downsampling.md b/manage-data/data-store/data-streams/run-downsampling.md index 8fc17469c8..873ef51125 100644 --- a/manage-data/data-store/data-streams/run-downsampling.md +++ b/manage-data/data-store/data-streams/run-downsampling.md @@ -31,7 +31,7 @@ In most cases, you can choose the data stream lifecycle option. If you're using :::{tab-item} Data stream lifecycle -## Downsample with a data stream lifecycle +## Downsample with a data stream lifecycle [downsample-with-a-data-stream-lifecycle] ```{applies_to} stack: ga serverless: ga @@ -61,7 +61,11 @@ PUT _data_stream/my-data-stream/_lifecycle } ``` -The downsampling action runs after the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed. +The downsampling action runs after the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed. + +{applies_to}`stack: ga 9.5` {applies_to}`serverless: ga` Data stream lifecycle limits how many backing indices per data stream it submits for downsampling at once. When the number of indices actively being downsampled reaches the threshold configured by [`data_streams.lifecycle.downsampling.max_indices_in_progress`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-downsampling-max-indices-in-progress), data stream lifecycle pauses queuing further downsampling operations for that data stream until the count drops. Deferred indices are recorded in the error store with a warning and retried in a subsequent lifecycle run. Other data streams are not affected. + +This limit helps prevent cluster overload when many indices become eligible for downsampling at the same time, such as after [loading historical metrics](/manage-data/data-store/data-streams/load-historical-tsds.md) and enabling lifecycle on the data stream. ::: :::{tab-item} Index lifecycle diff --git a/manage-data/data-store/data-streams/set-up-tsds.md b/manage-data/data-store/data-streams/set-up-tsds.md index 0a218ded5a..c5052a3688 100644 --- a/manage-data/data-store/data-streams/set-up-tsds.md +++ b/manage-data/data-store/data-streams/set-up-tsds.md @@ -1,5 +1,5 @@ --- -navigation_title: "Set up a TSDS" +navigation_title: "Set up a {{tsds-init}}" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-tsds.html applies_to: @@ -13,29 +13,30 @@ products: - id: elasticsearch --- -# Set up a time series data stream [set-up-tsds] +# Set up a {{tsds}} [set-up-tsds] -This page shows you how to manually set up a [time series data stream](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md) (TSDS). +This page shows you how to manually set up a [{{tsds}}](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md) ({{tsds-init}}). ## Before you begin [tsds-prereqs] -- Before you create a time series data stream, review [](../data-streams.md) and [TSDS concepts](time-series-data-stream-tsds.md). You can also try the [quickstart](/manage-data/data-store/data-streams/quickstart-tsds.md) for a hands-on introduction. +- Before you create a {{tsds}}, review [](../data-streams.md) and [{{tsds-init}} concepts](time-series-data-stream-tsds.md). You can also try the [quickstart](/manage-data/data-store/data-streams/quickstart-tsds.md) for a hands-on introduction. - Make sure you have the following permissions: - - [Cluster privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-cluster) - - `manage_index_templates` for creating a template to base the TSDS on - - {applies_to}`stack: ga` `manage_ilm` if you're using [index lifecycle management](#tsds-ilm-policy) - - [Index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices) - - `create_doc` and `create_index` for creating or converting a TSDS - - `manage` to [roll over](#convert-existing-data-stream-to-tsds) a TSDS + - [Cluster privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-cluster) + - `manage_index_templates` for creating a template to base the {{tsds-init}} on + - {applies_to}`stack: ga` `manage_ilm` if you're using [{{ilm}}](#tsds-ilm-policy) + - [Index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices) + - `create_doc` and `create_index` for creating or converting a {{tsds-init}} + - `manage` to [roll over](#convert-existing-data-stream-to-tsds) a {{tsds-init}} + - `auto_configure` if you choose to turn on past index creation ::::{note} If you're working with OpenTelemetry data, try the [OpenTelemetry quickstarts](/solutions/observability/get-started/opentelemetry/quickstart/index.md). :::: -## Set up a TSDS +## Set up a {{tsds-init}} :::::{stepper} -:::{step} Create an index lifecycle policy (optional) +::::{step} Create an index lifecycle policy (optional) :anchor: tsds-ilm-policy ```{applies_to} @@ -43,17 +44,17 @@ stack: ga serverless: unavailable ``` -In most cases, you can use a [data stream lifecycle](/manage-data/lifecycle/data-stream.md) to manage your time series data stream. If you're using [data tiers](/manage-data/lifecycle/data-tiers.md) in {{stack}}, you can use [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md). +In most cases, you can use a [data stream lifecycle](/manage-data/lifecycle/data-stream.md) to manage your {{tsds}}. If you're using [data tiers](/manage-data/lifecycle/data-tiers.md) in {{stack}}, you can use [index lifecycle management](/manage-data/lifecycle/index-lifecycle-management.md). -::::{note} +:::{note} :applies_to: {"stack": "ga 9.5"} {{ilm-init}} isn't required for frozen-tier {{search-snaps}}. Data stream lifecycle can manage them directly. Refer to [](/manage-data/lifecycle/data-stream/dlm-searchable-snapshots.md). -:::: +::: -:::{dropdown} Create an ILM policy +:::{dropdown} Create an {{ilm-init}} policy -If you're using {{stack}}, {{ilm-init}} can help you manage a time series data stream's backing indices. {{ilm-init}} requires an index lifecycle policy. +If you're using {{stack}}, {{ilm-init}} can help you manage the {{tsds}} backing indices. {{ilm-init}} requires an index lifecycle policy. For best results, specify a `max_age` for the `rollover` action in the policy. This ensures the [`timestamp` ranges](/manage-data/data-store/data-streams/time-bound-tsds.md) for the backing indices are consistent. For example, setting a `max_age` of `1d` for the `rollover` action ensures your backing indices consistently contain one day's worth of data. @@ -79,14 +80,14 @@ PUT _ilm/policy/my-weather-sensor-lifecycle-policy ``` ::: +:::: - -::::{step} Create an index template +::::{step} Create an index template :anchor: create-tsds-index-template -The structure of a time series data stream is defined by an index template. Create an index template with the following required elements and settings: +The structure of a {{tsds}} is defined by an index template. Create an index template with the following required elements and settings: -- **Index patterns:** One or more wildcard patterns matching the name of your TSDS, such as `weather-sensors-*`. For best results, use the [data stream naming scheme](/reference/fleet/data-streams.md#data-streams-naming-scheme). +- **Index patterns:** One or more wildcard patterns matching the name of your {{tsds-init}}, such as `weather-sensors-*`. For best results, use the [data stream naming scheme](/reference/fleet/data-streams.md#data-streams-naming-scheme). - **Data stream object:** The template must include `"data_stream": {}`. - **Time series mode:** Set `index.mode: time_series`. - **Field mappings:** Define at least one dimension field and typically one or more metric fields: @@ -145,17 +146,18 @@ PUT _index_template/my-weather-sensor-index-template } } ``` + 1. {{stack}} only -2. {{stack}} only +2. {{stack}} only, enabled by default in {{serverless-short}} :::{important} :applies_to: stack: ga -Without lifecycle management enabled, time series data streams can grow into very large indices that never roll over. This can lead to performance issues. Always configure lifecycle management for {{stack}} production deployments. +Without lifecycle management enabled, a {{tsds}} can grow into very large indices that never roll over. This can lead to performance issues. Always configure lifecycle management for {{stack}} production deployments. ::: :::{dropdown} Component templates (optional) -If you're using component templates with a time series data stream, check the following requirements: +If you're using component templates with a {{tsds}}, check the following requirements: - Each component template is valid on its own - The `index.routing_path` setting and its referenced dimension fields are defined in the same component template @@ -164,13 +166,33 @@ If you're using component templates with a time series data stream, check the fo :::: -::::{step} Create the time series data stream and add data +::::{step} Turn on past index creation (optional) +```{applies_to} +stack: ga 9.5+ +``` +:::{include} /manage-data/_snippets/enable-backfill.md +::: +:::: + +::::{step} Create the {{tsds}} and add data :anchor: create-tsds -After creating the index template, you can create a time series data stream by [indexing a document](use-data-stream.md#add-documents-to-a-data-stream). The TSDS is created automatically when you index the first document, as long as the index name matches the index template pattern. You can use a bulk API request or a POST request. +You can create a {{tsds}} explicitly with the [create a data stream]({{es-apis}}operation/operation-indices-create-data-stream) API. +You must give it a name that matches the `index_patterns` in your index template. +For example: + +```console +PUT _data_stream/metrics-weather-sensors +``` + +Alternatively, you can create it implicitly by indexing a document. +The {{tsds-init}} is created automatically as long as the index name in your API matches the index template pattern. :::{important} -To test the following `_bulk` example, update the timestamps to within two hours of your current time. Data added to a TSDS must fit the [accepted time range](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-accepted-time-range). +To add data to your data stream with the following bulk API request or POST request, you must update the timestamps to within two hours of your current time. +Data added to a {{tsds-init}} must fit the [accepted time range](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-accepted-time-range). + +{applies_to}`stack: ga 9.5` If you turned on past index creation, {{es}} creates backing indices automatically as historical documents are added within the eligible write window. For more details, go to [How time-bound indices work](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-accepted-time-range). ::: ```console @@ -179,9 +201,7 @@ PUT metrics-weather-sensors/_bulk { "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "SENSOR-001", "location": "warehouse-A", "temperature": 26.7,"humidity": 49.9 } { "create":{ } } { "@timestamp": "2099-05-06T16:25:42.000Z", "sensor_id": "SENSOR-002", "location": "warehouse-B", "temperature": 32.4, "humidity": 88.9 } -``` -```console POST metrics-weather-sensors/_doc { "@timestamp": "2099-05-06T16:21:15.000Z", @@ -191,10 +211,13 @@ POST metrics-weather-sensors/_doc "humidity": 88.9 } ``` + +For more information about adding documents to data streams, go to [](use-data-stream.md#add-documents-to-a-data-stream). + :::: ::::{step} Verify setup -To make sure your time series data stream is working, try some GET requests. +To make sure your {{tsds}} is working, try some GET requests. View data stream details: @@ -202,7 +225,7 @@ View data stream details: GET _data_stream/metrics-weather-sensors ``` -Check the document count in a time series data stream: +Check the document count in a {{tsds}}: ```console GET metrics-weather-sensors/_count @@ -218,15 +241,14 @@ GET metrics-weather-sensors/_search } ``` - :::: - +::::: ## Advanced setup -### Convert an existing data stream to a TSDS [convert-existing-data-stream-to-tsds] +### Convert an existing data stream to a {{tsds-init}} [convert-existing-data-stream-to-tsds] -You can convert an existing regular data stream to a TSDS. Follow these steps: +You can convert an existing regular data stream to a {{tsds-init}}. Follow these steps: 1. Update your existing index template and component templates (if any) to include time series settings. For {{stack}}, configure lifecycle management. 2. Use the [rollover API]({{es-apis}}operation/operation-indices-rollover) to manually roll over the existing data stream's write index, to apply the changes you made in step 1: @@ -236,20 +258,24 @@ POST metrics-weather-sensors/_rollover ``` :::{note} -After the rollover, new backing indices will have time series functionality. Existing backing indices are not affected by the rollover (because their `index.mode` cannot be changed). +After the rollover, new backing indices will have time series functionality. Existing backing indices are not affected by the rollover because their `index.mode` cannot be changed. ::: -### Secure a time series data stream [secure-tsds] +### Secure a {{tsds}} [secure-tsds] -To control access to a TSDS, use [index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices). Privileges set on a TSDS also apply to the backing indices. +To control access to a {{tsds-init}}, use [index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices). Privileges set on a {{tsds-init}} also apply to the backing indices. For an example, refer to [Data stream privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges). +{applies_to}`stack: ga 9.5` Users who write documents that trigger creation of [past backing indices](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-past-index-creation) need the `auto_configure` index privilege on the data stream, in addition to privileges that allow indexing (such as `create_doc` or `index`). +Users with only the `index` privilege receive a `security_exception` when a write would create a past backing index. + ## Next steps [set-up-tsds-whats-next] -Now that you've set up a time series data stream, you can manage and use it like a regular data stream. For more information, refer to: +Now that you've set up a {{tsds}}, you can manage and use it like a regular data stream. For more information, refer to: * [Use a data stream](use-data-stream.md) for indexing and searching * [Change data stream settings](modify-data-stream.md#data-streams-change-mappings-and-settings) as needed * Query time series data using the {{esql}} [`TS` command](elasticsearch://reference/query-languages/esql/commands/ts.md) +* {applies_to}`stack: ga 9.5` [Load historical data into a {{tsds-init}}](/manage-data/data-store/data-streams/load-historical-tsds.md) * Use [data stream APIs]({{es-apis}}group/endpoint-data-stream) diff --git a/manage-data/data-store/data-streams/time-bound-tsds.md b/manage-data/data-store/data-streams/time-bound-tsds.md index 1d666cb36d..7f8e5ba8cb 100644 --- a/manage-data/data-store/data-streams/time-bound-tsds.md +++ b/manage-data/data-store/data-streams/time-bound-tsds.md @@ -13,54 +13,82 @@ products: # Time-bound indices and dimension-based routing [time-bound-indices] -Unlike regular data streams that only write to the most recent backing index, time series data streams (TSDS) use time-bound backing indices that accept documents based on their timestamp values. This page provides details and best practices to help you work with time-bound indices. +Unlike regular data streams that write only to the most recent backing index, {{tsds}} ({{tsds-init}}) are backed by a set of time-bound indices, each covering a contiguous, non-overlapping time range. +{{es}} enforces that the `@timestamp` for each ingested document falls within the time range of exactly one backing index. -## How time-bound indices work +## How time-bound indices work [tsds-accepted-time-range] -Each TSDS backing index has a time range for accepted `@timestamp` values, defined by two settings: +Each {{tsds-init}} backing index has a range of `@timestamp` values that it accepts, which are tracked in index settings. +When the {{tsds-init}} is created, the first backing index has the following range: -- [`index.time_series.start_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-start-time): The earliest accepted timestamp (inclusive) -- [`index.time_series.end_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time): The latest accepted timestamp (exclusive) +- Its [`index.time_series.start_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-start-time), which is the earliest accepted timestamp (inclusive), is set to `now` minus the [`index.look_back_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-look-back-time). +- Its [`index.time_series.end_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time), which is the latest accepted timestamp (exclusive), is set to `now` plus [`index.look_ahead_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-look-ahead-time). -When you add a document to a TSDS, {{es}} adds the document to the appropriate backing index based on its `@timestamp` value. This means a TSDS can write to multiple backing indices simultaneously, not just the most recent one. +Thereafter, {{es}} automatically configures the settings for backing indices as part of the index creation and [rollover](/manage-data/lifecycle/index-lifecycle-management/rollover.md) process. +Each new backing index starts at the previous index's `end_time` and extends further ahead using `look_ahead_time`. + +When you add a document to the TSDS, {{es}} routes it to the appropriate backing index based on its `@timestamp` value. +This means a TSDS can write to multiple backing indices simultaneously, not only the most recent one. :::{image} /manage-data/images/elasticsearch-reference-time-bound-indices.svg -:alt: time bound indices +:alt: Time bound indices ::: -If no backing index can accept a document's `@timestamp` value, {{es}} rejects the document. - -{{es}} automatically configures `index.time_series.start_time` and `index.time_series.end_time` settings as part of the index creation and rollover process. +Late-arriving data can still be indexed into an older backing index, as long as that index exists, remains writable, and its accepted time range includes the timestamp. +To inspect the accepted time ranges of TSDS backing indices, use the [get data stream API]({{es-apis}}operation/operation-indices-get-data-stream): -### Accepted time range for adding data [tsds-accepted-time-range] +```console +GET _data_stream/my-tsds +``` -A TSDS is designed to ingest current metrics data. When the TSDS is first created, the initial backing index has the following settings: +By default, if no existing backing index can accept a document's `@timestamp`, {{es}} rejects the document. +{{es}} does not create missing past backing indices unless you [turn on past index creation](#tsds-past-index-creation). -- An `index.time_series.start_time` value set to `now - index.look_back_time` -- An `index.time_series.end_time` value set to `now + index.look_ahead_time` +::::{tip} +Writes might still be rejected even when a timestamp fits the accepted time range of a backing index. The following [actions](elasticsearch://reference/elasticsearch/index-lifecycle-actions.md) can affect the writable time range, either because they make a backing index read-only or remove it: -Only data that falls within this range is indexed. +- [Delete](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) +- [Downsample](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) +- [Force merge](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) +- [Read only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) +- [Searchable snapshot](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) +- [Shrink](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md), which might revert the read-only status at the end of the action -To check the accepted time range for writing to a TSDS, use the [get data stream API]({{es-apis}}operation/operation-indices-get-data-stream): +{{ilm-cap}} will **not** proceed with running these actions until [`index.time_series.end_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed. +:::: -```console -GET _data_stream/my-tsds +## Past index creation [tsds-past-index-creation] +```{applies_to} +stack: ga 9.5 ``` -::::{tip} -Writes within the accepted time range might still be rejected. The following actions can affect the writable time range, either because they make a backing index read-only or remove it: - - [Delete](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) - - [Downsample](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - - [Force merge](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - - [Read only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - - [Searchable snapshot](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - - [Shrink](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md), which might then revert the read-only status at the end of the action - - {{ilm-cap}} will **not** proceed with executing these actions until [`index.time_series.end_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed. -:::: +In addition to accepted time ranges for each backing index, a {{tsds}} has an _eligible write window_. +It is the period of time that extends from the present back to whichever comes first: +- the first lifecycle action that makes a backing index read-only (such as [downsampling](/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) or a {{search-snap}} transition), or +- the data stream retention limit (configured in a [data stream lifecycle](/manage-data/lifecycle/data-stream.md), for example) -### Dimension-based routing [dimension-based-routing] +{{es}} can +automatically create missing past backing indices for an existing {{tsds}} while indexing documents that fall within the eligible write window. To enable this, set the [`data_stream.past_tsdb_index_creation_enabled`](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#time-series-data-stream) cluster setting to `true`. + +Timestamps outside the eligible write window or in the future are still rejected. +If a [failure store](/manage-data/data-store/data-streams/failure-store.md) is enabled, rejected timestamp failures can be redirected there. + +:::{admonition} Lifecycle age for past indices + +Past backing indices hold old data but are new indices. +{{es}} sets [`index.lifecycle.origination_date`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-data-stream-lifecycle-origination-date) from `index.time_series.end_time` so that data stream lifecycle and [{{ilm}}](/manage-data/lifecycle/index-lifecycle-management.md) treat the index age based on the data it contains, not when the index was created. + +::: + +Each new past backing index covers a configurable time interval. +Use the [`data_streams.past_tsdb_index_interval`](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#time-series-data-stream) cluster setting to control the interval. + +When the gap between existing indices is up to 1.3 times the configured interval, {{es}} might create a single bridging index instead of many small indices. + +For guidance on loading historical data, refer to [Load historical data into a TSDS](/manage-data/data-store/data-streams/load-historical-tsds.md). + +## Dimension-based routing [dimension-based-routing] In addition to time-based routing, time series data streams use dimension-based routing to determine which shard to route data to. Documents with the same dimensions are routed to the same shards, using one of two strategies: @@ -85,5 +113,3 @@ or manually set the [`index.routing_path`](elasticsearch://reference/elasticsear Documents with the same dimension values are routed to the same shard, improving compression and query performance for time series data. The `index.routing_path` setting supports wildcards (for example, `dim.*`) and can dynamically match new fields. - - diff --git a/manage-data/data-store/data-streams/time-series-data-stream-tsds.md b/manage-data/data-store/data-streams/time-series-data-stream-tsds.md index 17d7e795b2..ce7991104a 100644 --- a/manage-data/data-store/data-streams/time-series-data-stream-tsds.md +++ b/manage-data/data-store/data-streams/time-series-data-stream-tsds.md @@ -57,7 +57,7 @@ Dimension fields often correspond to characteristics of the items you're measuri :::{tip} {{es}} uses dimensions and timestamps to generate time series document `_id` values. Two documents with the same dimensions and timestamp are considered duplicates. Duplicates are rejected during ingestion with a `409 Conflict` status. -::: +::: To mark a field as a dimension, set the Boolean `time_series_dimension` mapping parameter to `true`. The following field types support the `time_series_dimension` parameter: @@ -105,10 +105,12 @@ A time series data stream works like a regular data stream, with some key differ * One or more [metric fields](#time-series-metric) * An auto-generated document `_id` (custom `_id` values are not supported) * **Backing indices:** A TSDS uses [time-bound indices](/manage-data/data-store/data-streams/time-bound-tsds.md) to store data from the same time period in the same backing index. + + {applies_to}`stack: ga 9.5` When past index creation is turned on, {{es}} can create missing past backing indices on demand in TSDS. * **Dimension-based routing:** The routing logic uses dimension fields to map all data points of a time series to the same shard, improving storage efficiency and query performance. Duplicate data points are rejected. * **Sorting:** A TSDS uses internal [index sorting](elasticsearch://reference/elasticsearch/index-settings/sorting.md) to order shard segments by `_tsid` and `@timestamp`, for better compression. Time series data streams do not use `index.sort.*` settings. * **Source field:** A TSDS uses [synthetic `_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source), and as a result is subject to some [restrictions](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-restrictions) and [modifications](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications) applied to the `_source` field. -* {applies_to}`stack: ga 9.3` **Doc values skippers:** A TSDS enables [docvalue skippers](elasticsearch://reference/elasticsearch/mapping-reference/doc-values.md#doc-values-skippers) on its `_tsid`, `@timestamp`, [dimension](#time-series-dimension), and [metric](#time-series-metric) fields. Because `tsid` and `@timestamp` are part of the index sort, the skippers allow \{\{es}} to avoid building backing indexes for these fields, meaning lower disk usage and faster ingest speed. +* {applies_to}`stack: ga 9.3` **Doc values skippers:** A TSDS enables [docvalue skippers](elasticsearch://reference/elasticsearch/mapping-reference/doc-values.md#doc-values-skippers) on its `_tsid`, `@timestamp`, [dimension](#time-series-dimension), and [metric](#time-series-metric) fields. Because `tsid` and `@timestamp` are part of the index sort, the skippers allow {{es}} to avoid building backing indexes for these fields, meaning lower disk usage and faster ingest speed. * {applies_to}`stack: ga 9.4` **Sequence numbers are disabled:** A TSDS [disables sequence numbers](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-disable-sequence-numbers) by default to substantially improve storage efficiency (up to 2x). When sequence numbers are disabled, [optimistic concurrency control](elasticsearch://reference/elasticsearch/rest-apis/optimistic-concurrency-control.md) gets disabled, causing [update-by-query]({{es-apis}}/operation/operation-update-by-query) and [delete-by-query]({{es-apis}}/operation/operation-delete-by-query) operations to execute with weaker consistency. These capabilities are normally not relevant for time series workloads, but if you need them for your application, you can restore sequence numbers by setting `index.disable_sequence_numbers: false` in the index template of the relevant TSDS. @@ -127,4 +129,5 @@ You can use the {{esql}} [`TS` command](elasticsearch://reference/query-language * [Ingest data using the OpenTelemetry Protocol (OTLP)](/manage-data/data-store/data-streams/tsds-ingest-otlp.md) * [Ingest data using Prometheus remote write](/manage-data/data-store/data-streams/tsds-ingest-prometheus-remote-write.md) * Learn about [metric temporality](/manage-data/data-store/data-streams/metric-temporality.md) (delta versus cumulative) +* {applies_to}`stack: ga 9.5` [Load historical data into a TSDS](/manage-data/data-store/data-streams/load-historical-tsds.md) * Learn about [downsampling](/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) to reduce storage footprint diff --git a/manage-data/lifecycle/data-stream.md b/manage-data/lifecycle/data-stream.md index aa0b95e3c0..be6d844c7b 100644 --- a/manage-data/lifecycle/data-stream.md +++ b/manage-data/lifecycle/data-stream.md @@ -42,8 +42,8 @@ In intervals configured by [`data_streams.lifecycle.poll_interval`](elasticsearc 1. Checks if the data stream has a data stream lifecycle configured, skipping any indices not part of a managed data stream. 2. Rolls over the write index of the data stream, if it fulfills the conditions defined by [`cluster.lifecycle.default.rollover`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#cluster-lifecycle-default-rollover). -3. After an index is not the write index anymore (that is, the data stream has been rolled over), automatically tail merges the index. Data stream lifecycle executes a merge operation that only targets the long tail of small segments instead of the whole shard. As the segments are organised into tiers of exponential sizes, merging the long tail of small segments is only a fraction of the cost of force merging to a single segment. The small segments would usually hold the most recent data so tail merging will focus the merging resources on the higher-value data that is most likely to keep being queried. -4. If [downsampling]({{es-apis}}operation/operation-indices-put-data-lifecycle) is configured it will execute all the configured downsampling rounds. +3. After an index is not the write index anymore (that is, the data stream has been rolled over), automatically tail merges the index. Data stream lifecycle runs a merge operation that only targets the long tail of small segments instead of the whole shard. As the segments are organised into tiers of exponential sizes, merging the long tail of small segments is only a fraction of the cost of force merging to a single segment. The small segments would usually hold the most recent data so tail merging will focus the merging resources on the higher-value data that is most likely to keep being queried. +4. If [downsampling]({{es-apis}}operation/operation-indices-put-data-lifecycle) is configured it will run all the configured downsampling rounds. {applies_to}`stack: ga 9.5` {applies_to}`serverless: ga` Data stream lifecycle limits concurrent downsampling per data stream using the [`data_streams.lifecycle.downsampling.max_indices_in_progress`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-downsampling-max-indices-in-progress) cluster setting (default `10`). For details, refer to [Downsample with a data stream lifecycle](/manage-data/data-store/data-streams/run-downsampling.md#downsample-with-a-data-stream-lifecycle). 5. {applies_to}`stack: ga 9.5+` {applies_to}`serverless: unavailable` Transitions eligible backing indices to partially mounted {{search-snaps}}. Refer to [](/manage-data/lifecycle/data-stream/dlm-searchable-snapshots.md). 6. Applies retention to the remaining backing indices. This means deleting the backing indices whose `generation_time` is longer than the effective retention period (read more about the [effective retention calculation](data-stream/tutorial-data-stream-retention.md#effective-retention-calculation)). The `generation_time` is only applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time optionally configured in the [`index.lifecycle.origination_date`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-data-stream-lifecycle-origination-date) setting. Retention can delete indices even after they have been moved to {{search-snaps}}. diff --git a/manage-data/toc.yml b/manage-data/toc.yml index 9423f44462..0381113fad 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -23,6 +23,7 @@ toc: - file: data-store/data-streams/advanced-topics-tsds.md children: - file: data-store/data-streams/time-bound-tsds.md + - file: data-store/data-streams/load-historical-tsds.md - file: data-store/data-streams/metric-temporality.md - file: data-store/data-streams/reindex-tsds.md - file: data-store/data-streams/tsds-ingest-otlp.md