Document TSDB backfill behaviour

docset.yml

@@ -393,5 +393,8 @@ subs:
   fedramp-mod: "FedRAMP Moderate"
   fedramp-high: "FedRAMP High"
   fedramp-il5: "FedRAMP IL5"
+  tsds: "time series data stream"
+  tsds-cap: "Time series data stream"
+  tsds-init: "TSDS"
 release_notes:
   - product: cloud-serverless

manage-data/_snippets/enable-backfill.md

@@ -0,0 +1,14 @@
+{{es}} can create missing backing indices when you add data that precedes existing time ranges.
+To enable this feature, update the cluster settings:
+
+```console
+PUT _cluster/settings
+{
+  "persistent": {
+    "data_stream.past_tsdb_index_creation_enabled": true,
+    "data_streams.past_tsdb_index_interval": "2d" <1>
+  }
+}
+```
+
+1. By default, each past backing index covers one day of data. Refer to [`data_streams.past_tsdb_index_interval`](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#time-series-data-stream).

manage-data/data-store/data-streams.md

@@ -15,7 +15,7 @@
 
 A data stream acts as a layer of abstraction over a set of indices that are optimized for storing append-only time series data. It stores data across multiple backing indices while giving you a single named resource to use for requests. Data streams are well-suited for logs, events, metrics, and other continuously generated data.
 
-You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream’s data. You can use [{{ilm}} ({{ilm-init}})](../lifecycle/index-lifecycle-management.md) to automate the management of these backing indices. For example, you can use {{ilm-init}} to automatically move older backing indices to less expensive hardware and delete unneeded indices. {{ilm-init}} can help you reduce costs and overhead as your data grows.
+You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream's data. You can use [{{ilm}} ({{ilm-init}})](../lifecycle/index-lifecycle-management.md) to automate the management of these backing indices. For example, you can use {{ilm-init}} to automatically move older backing indices to less expensive hardware and delete unneeded indices. {{ilm-init}} can help you reduce costs and overhead as your data grows.
 
 You can also use a [data stream lifecycle](../lifecycle/data-stream.md) to automate lifecycle management according to your retention requirements.
 
@@ -36,7 +36,7 @@
 * You mostly perform indexing requests, with occasional updates and deletes.
 * You index documents without an `_id`, or when indexing documents with an explicit `_id` you expect first-write-wins behavior.
 
-For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn’t fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See the tutorial [](../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) for more information.
+For most time series data use-cases, a data stream will be a good fit. However, if you find that your data doesn't fit into these categories (for example, if you frequently send multiple documents using the same `_id` expecting last-write-wins), you may want to use an index alias with a write index instead. See the tutorial [](../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) for more information.
 
 Keep in mind that some features such as [Time Series Data Streams (TSDS)](../data-store/data-streams/time-series-data-stream-tsds.md) and [data stream lifecycles](../lifecycle/data-stream.md) require a data stream.
 
@@ -47,7 +47,6 @@
 [LogsDB](data-streams/logs-data-stream.md) and [time series (TSDS)](data-streams/time-series-data-stream-tsds.md) index modes are not available in {{vectordb}}; use an {{es}}, Observability, or Security project type for those workloads.
 :::
 
-
 ## Backing indices [backing-indices]
 
 A data stream consists of one or more [hidden](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-hidden), auto-generated backing indices.
@@ -56,15 +55,14 @@
 :alt: data streams diagram
 :::
 
-A data stream requires a matching [index template](templates.md). The template contains the mappings and settings used to configure the stream’s backing indices and defines the {{ilm-init}} policy that the data stream uses.
+A data stream requires a matching [index template](templates.md). The template contains the mappings and settings used to configure the stream's backing indices and defines the {{ilm-init}} policy that the data stream uses.
 
-Every document indexed to a data stream must contain a `@timestamp` field, mapped as a [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) field type. If the index template doesn’t specify a mapping for the `@timestamp` field, {{es}} maps `@timestamp` as a `date` field with default options.
+Every document indexed to a data stream must contain a `@timestamp` field, mapped as a [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) field type. If the index template doesn't specify a mapping for the `@timestamp` field, {{es}} maps `@timestamp` as a `date` field with default options.
 
 The same index template can be used for multiple data streams. You cannot delete an index template in use by a data stream.
 
 The name pattern for the backing indices is an implementation detail and no intelligence should be derived from it. The only invariant the holds is that each data stream generation index will have a unique name.
 
-
 ## Read requests [data-stream-read-requests]
 
 When you submit a read request to a data stream, the stream routes the request to all its backing indices.
@@ -73,28 +71,29 @@
 :alt: data streams search request
 :::
 
-
 ## Write index [data-stream-write-index]
 
-The most recently created backing index is the data stream’s write index. The stream adds new documents to this index only.
+The most recently created backing index is the data stream's write index. The stream adds new documents to this index only.
 
 :::{image} /manage-data/images/elasticsearch-reference-data-streams-index-request.svg
 :alt: data streams index request
 :::
 
 You cannot add new documents to other backing indices, even by sending requests directly to the index.
-
 You also cannot perform operations on a write index that may hinder indexing, such as:
 
 * [Clone]({{es-apis}}operation/operation-indices-clone)
 * [Delete]({{es-apis}}operation/operation-indices-delete)
 * [Shrink]({{es-apis}}operation/operation-indices-shrink)
 * [Split]({{es-apis}}operation/operation-indices-split)
 
+:::{note}
+[Time series data streams (TSDS)](data-streams/time-series-data-stream-tsds.md) are different: they write to the backing index whose time range includes the document's `@timestamp`, not only the latest one.
+:::
 
 ## Rollover [data-streams-rollover]
 
-A [rollover]({{es-apis}}operation/operation-indices-rollover) creates a new backing index that becomes the stream’s new write index.
+A [rollover]({{es-apis}}operation/operation-indices-rollover) creates a new backing index that becomes the stream's new write index.
 
 We recommend using [{{ilm-init}}](../lifecycle/index-lifecycle-management.md) to automatically roll over data streams when the write index reaches a specified age or size. If needed, you can also [manually roll over](data-streams/use-data-stream.md#manually-roll-over-a-data-stream) a data stream.
 
@@ -109,21 +108,18 @@
 .ds-<data-stream>-<yyyy.MM.dd>-<generation>
 ```
 
-`<yyyy.MM.dd>` is the backing index’s creation date. Backing indices with a higher generation contain more recent data. For example, the `web-server-logs` data stream has a generation of `34`. The stream’s most recent backing index, created on 7 March 2099, is named `.ds-web-server-logs-2099.03.07-000034`.
+`<yyyy.MM.dd>` is the backing index's creation date. Backing indices with a higher generation contain more recent data. For example, the `web-server-logs` data stream has a generation of `34`. The stream's most recent backing index, created on 7 March 2099, is named `.ds-web-server-logs-2099.03.07-000034`.
 
-Some operations, such as a [shrink]({{es-apis}}operation/operation-indices-shrink) or [restore](../../deploy-manage/tools/snapshot-and-restore/restore-snapshot.md), can change a backing index’s name. These name changes do not remove a backing index from its data stream.
+Some operations, such as a [shrink]({{es-apis}}operation/operation-indices-shrink) or [restore](../../deploy-manage/tools/snapshot-and-restore/restore-snapshot.md), can change a backing index's name. These name changes do not remove a backing index from its data stream.
 
 The generation of the data stream can change without a new index being added to the data stream (for example, when an existing backing index is shrunk). This means the backing indices for some generations will never exist. You should not derive any intelligence from the backing indices names.
 
-
 ## Append-only (mostly) [data-streams-append-only]
 
-Data streams are designed for use cases where existing data is rarely updated. You cannot send update or deletion requests for existing documents directly to a data stream. However, you can still [update or delete documents](data-streams/use-data-stream.md#update-delete-docs-in-a-backing-index) in a data stream by submitting requests directly to the document’s backing index.
+Data streams are designed for use cases where existing data is rarely updated. You cannot send update or deletion requests for existing documents directly to a data stream. However, you can still [update or delete documents](data-streams/use-data-stream.md#update-delete-docs-in-a-backing-index) in a data stream by submitting requests directly to the document's backing index.
 
 If you need to update a larger number of documents in a data stream, you can use the [update by query](data-streams/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](data-streams/use-data-stream.md#delete-docs-in-a-data-stream-by-query) APIs.
 
 ::::{tip}
 If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. See the tutorial [](../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md).
 ::::
-
-

manage-data/data-store/data-streams/advanced-topics-tsds.md

@@ -16,6 +16,7 @@ products:
 This section contains information about advanced concepts and operations for [time series data streams](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md):
 
 - [](/manage-data/data-store/data-streams/time-bound-tsds.md)
+- [](/manage-data/data-store/data-streams/load-historical-tsds.md)
 - [](/manage-data/data-store/data-streams/metric-temporality.md)
 - [](/manage-data/data-store/data-streams/reindex-tsds.md)
 - [](/manage-data/data-store/data-streams/tsds-ingest-otlp.md)

manage-data/data-store/data-streams/load-historical-tsds.md

@@ -0,0 +1,150 @@
+---
+navigation_title: "Load historical data"
+applies_to:
+  stack: ga 9.5
+products:
+  - id: elasticsearch
+---
+
+# Load historical data into a {{tsds}} [load-historical-tsds]
+
+By default, a {{tsds-cap}} ({{tsds-init}}) works well for continuous, near-real-time ingestion.
+Only documents with `@timestamp` values that fall inside the time range of existing backing indices are accepted.
+
+To import historical data into an existing {{tsds-init}}, enable the [data_stream.past_tsdb_index_creation_enabled](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#time-series-data-stream) cluster setting. You can then use the same APIs you use for live data.
+{{es}} creates the past backing indices needed to store past documents as they arrive.
+The documents must fall within the [eligible write window](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-past-index-creation). This is the period of time between "current" and the data stream retention limit or the first occurrence of a lifecycle action that makes a backing index read-only, whichever occurs first.
+Write-time deduplication and {{tsds-init}} storage optimizations apply to historical data the same way they apply to live data.
+
+:::{note}
+Users who trigger past index creation need the `auto_configure` index privilege.
+For details, refer to [Secure a {{tsds-init}}](/manage-data/data-store/data-streams/set-up-tsds.md#secure-tsds).
+:::
+
+## Load data within the eligible write window
+
+For data that falls within the eligible write window, point your migration or replay pipeline at the live {{tsds}}.
+{{es}} creates past backing indices as needed.
+
+This approach works well when you're backfilling recent history alongside live ingestion, such as late-arriving metrics or a short bootstrap period.
+
+Go to [](/manage-data/data-store/data-streams/set-up-tsds.md) for an example of setting up and loading historical data into a new {{tsds}}.
+
+## Load data beyond the eligible write window
+
+You can't load data older than the eligible write window directly into a {{tsds-init}}.
+For example, if downsampling makes indices read-only after seven days, you can't backfill eighteen months of history into that same data stream.
+
+Instead, create a separate historical {{tsds-init}} without a lifecycle, load the data, then add a [data stream lifecycle](/manage-data/lifecycle/data-stream.md) when the load is complete.
+
+:::::{stepper}
+::::{step} Create an index template for the historical data stream
+
+Use the same mappings as your live {{tsds-init}}, but don't include a lifecycle policy in the template.
+For example, use the [create index template]({{es-apis}}operation/operation-indices-put-index-template) API:
+
+```console
+PUT _index_template/metrics-historical
+{
+  "index_patterns": ["metrics-historical-*"],
+  "data_stream": {},
+  "template": {
+    "settings": {
+      "index.mode": "time_series"
+    },
+    "mappings": {
+      "properties": {
+        "@timestamp": { "type": "date" },
+        "sensor_id": { "type": "keyword", "time_series_dimension": true },
+        "temperature": { "type": "half_float", "time_series_metric": "gauge" }
+      }
+    }
+  }
+}
+```
+
+::::
+
+::::{step} Create the historical data stream
+
+Create a data stream with a name that matches the pattern in the index template.
+For example, use the [create a data stream]({{es-apis}}operation/operation-indices-create-data-stream) API:
+
+```console
+PUT _data_stream/metrics-historical-2024
+```
+
+::::
+
+::::{step} Index historical data
+
+Index historical data into the historical data stream while current data continues flowing into the original {{tsds-init}}.
+
+:::{important}
+Historical data must fit on the target tier as a whole before you enable data stream lifecycle.
+If you're importing a large data set, split it into batches.
+Each batch should fit within available disk space at indexing time.
+:::
+::::
+
+::::{step} Add data stream lifecycle
+
+When the load is complete, add a [data stream lifecycle](/manage-data/lifecycle/data-stream.md) to the historical data stream.
+For example, use the [update data stream lifecycles]({{es-apis}}operation/operation-indices-put-data-lifecycle) API:
+
+```console
+PUT _data_stream/metrics-historical-2024/_lifecycle
+{
+  "enabled": true,
+  "data_retention": "365d",
+  "downsampling": [
+    {
+      "after": "7d",
+      "fixed_interval": "10m"
+    }
+  ]
+}
+```
+
+Processing begins immediately and creates a backlog of downsampling work.
+If you include `data_retention` settings, data stream lifecycle deletes expired backing indices but does not remove the data stream itself.
+::::
+
+::::{step} Query across both data streams
+
+Query both streams with a wildcard pattern or a [data stream alias](/manage-data/data-store/aliases.md).
+For example,  use the [search]({{es-apis}}operation/operation-search) API:
+
+```console
+GET metrics-*/_search
+{
+  "size": 10,
+  "sort": [{ "@timestamp": "desc" }]
+}
+```
+
+::::
+:::::
+Delete historical data streams manually when their data is no longer needed.
+
+## Protect the cluster during large loads
+
+Loading months of historical data can trigger significant storage use, force merge activity, and lifecycle processing in parallel.
+Verify that your cluster has enough available resources before you start.
+
+When you enable a lifecycle on a data stream with many indices that qualify for downsampling, data stream lifecycle can queue multiple downsampling operations at once.
+To limit concurrent downsampling per data stream, configure the [`data_streams.lifecycle.downsampling.max_indices_in_progress`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-downsampling-max-indices-in-progress) cluster setting.
+For details, refer to [Downsample with a data stream lifecycle](/manage-data/data-store/data-streams/run-downsampling.md#downsample-with-a-data-stream-lifecycle).
+
+## Limitations
+
+Backfill and creation of past indices have the following limitations:
+
+- System data streams are excluded.
+- {{ccr-cap}} ({{ccr-init}}) follower data streams rely on the leader data stream, so you can't backfill follower streams directly.
+
+## Next steps
+
+- [Time-bound indices](/manage-data/data-store/data-streams/time-bound-tsds.md) for eligible write window and past index creation details
+- [Downsampling a time series data stream](/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) to reduce storage after historical data ages
+- [Reindex a time series data stream](/manage-data/data-store/data-streams/reindex-tsds.md) if you need to copy data to a new {{tsds-init}} instead of backfilling in place

manage-data/data-store/data-streams/quickstart-tsds.md

@@ -150,7 +150,7 @@ A successful request returns `"errors": false` and a `create` item for each docu
 :::
 
 :::{tip}
-If you get an error about timestamp values, check the error response for the valid timestamp range and run the bulk API again with appropriate `@timestamp` values.
+If you get an error about timestamp values, check the error response for the valid timestamp range.
 For more details, refer to [Accepted time range for adding data](/manage-data/data-store/data-streams/time-bound-tsds.md#tsds-accepted-time-range).
 :::