Add data tier removal guidance for self-managed and ECK - #8213
Conversation
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
🔍 Preview links for changed docs |
Elastic Docs Style Checker (Vale)Summary: 2 warnings, 16 suggestions found
|
| File | Line | Rule | Message |
|---|---|---|---|
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 169 | Elastic.Spelling | 'substeps' is a possible misspelling. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-self-managed-eck.md | 153 | Elastic.Spelling | 'substeps' is a possible misspelling. |
💡 Suggestions (16): Optional style improvements. Apply when helpful.
| File | Line | Rule | Message |
|---|---|---|---|
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 91 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 97 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 198 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 199 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 290 | Elastic.WordChoice | Consider using 'deactivates, deselects, hides, turns off, makes unavailable' instead of 'disables', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 301 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 313 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 320 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 331 | Elastic.WordChoice | Consider using 'deactivates, deselects, hides, turns off, makes unavailable' instead of 'disables', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 339 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 426 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 440 | Elastic.WordChoice | Consider using 'deactivate, deselect, hide, turn off' instead of 'disable', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 444 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 446 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 456 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece.md | 463 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
@stefnestor : I'm adding you here as a reviewer because you also collaborated when we created the ECH/ECE version of the procedure. Note that I didn't merge all into a single document because the resulting document would need a lot of applies switches and part of the narrative should be modified due to operational differences between Elastic Cloud + ECE and self-managed systems and ECK, but this could be evaluated in the future. Also note that I found a few logical and technical caveats in the original procedure and I've tried to improve them. Hoping you like the result Stef and @yetanothertw |
| ### Remove a tier with regular indices [non-searchable-snapshot-data-tier] | ||
|
|
||
| The frozen tier only stores [partially mounted](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) {{search-snaps}}. [Fully mounted](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) {{search-snaps}} can be allocated to the hot or cold tier depending on the {{ilm-init}} phase, while the cold tier can also hold regular indices. Use the checks in [Before you remove a data tier](#before-you-remove-a-data-tier) if you are unsure what is on the tier. | ||
| This section covers the removal of a tier that holds regular indices. The goal is to ensure all shard allocation rules allow the data to move to other tiers before you disable the tier. You also need to temporarily stop {{ilm-init}} to prevent new indices from being routed to the tier while you work. |
There was a problem hiding this comment.
We in Support do not disable ILM 👻. Especially where drain off time can be extensive & you might full disk hot nodes when you can't stop ingest.
Instead we normally recommend users exclude nodes, ex:
PUT _cluster/settings
{"transient": {"cluster": {"routing": {"allocation": {"exclude": {"_name": "instance-0000000043"} } } } } }
Note: this will incur ILM errors (and if you're hot>cold>frozen and removing frozen then agreed you'd still fill up hot, which is why we tell users to disable tiers the same as you'd rolling restart: from lowest-to-highest temp).
| ``` | ||
| 1. The example represents an index in the `warm` tier. | ||
|
|
||
| Before disabling the tier, update `_tier_preference` so that the tier where you want the data to move is the first available tier in the list. This allows {{es}} to begin relocating the shards before the deployment plan removes the tier. |
There was a problem hiding this comment.
TBC shards will not allocate to a fallback list-item unless/until no nodes of preceding exist in cluster. So this sentence reads catch-22 IMO. You must update _tier_preference to migrate shards off current deprecating tier, it is not to "begin relocating" early, it is a required step.
ref: https://www.elastic.co/docs/manage-data/lifecycle/data-tiers#data-tier-allocation
"This setting also accepts multiple tiers in order of preference. This prevents indices from remaining unallocated if there are no nodes in the cluster for the preferred tier."
rel: #1903
|
|
||
| Updating allocation inclusion rules will trigger a shard re-allocation, moving the shards from the nodes to be removed. | ||
| :::{note} | ||
| Do not use the frozen tier as a fallback for regular indices. It is reserved for partially mounted {{search-snaps}}. |
There was a problem hiding this comment.
| Do not use the frozen tier as a fallback for regular indices. It is reserved for partially mounted {{search-snaps}}. | |
| Do not use the frozen tier as a fallback for regular nor fully mounted indices. It is reserved for partially mounted {{search-snaps}}. |
| ::: | ||
|
|
||
| 2. Updating node attribute allocation requirement rules. | ||
| 2. Update node attribute allocation requirement rules. |
There was a problem hiding this comment.
Potential noop comment : This is considered user customization responsible and not default anymore. This is why https://www.elastic.co/docs/manage-data/lifecycle/data-tiers#data-tier-allocation-troubleshooting was simplified down to only ~"by now you should have migrated to data tiers & ensure you don't induce allocation conflicts". From Support's side, we usually say, "update tier preferences, let shards drain off, check allocation explain for anybody stubborn not moving off to see why".
This is the first common example: https://www.elastic.co/docs/troubleshoot/elasticsearch/cluster-allocation-api-examples#conflicting-settings
| Confirm that no shards are left on the nodes to be removed after the allocation completes: `GET /_cat/shards` (filter by `node` as needed) should show that the tier is empty. Updating settings starts the relocation process, but you must wait until [shard allocation and recovery](/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery.md) finish. If shards stay on the original tier, use the [cluster allocation explain](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-allocation-explain) API to determine the cause. Common reasons can be [disk watermarks](/troubleshoot/elasticsearch/fix-watermark-errors.md) or [`index.routing.allocation.total_shards_per_node`](elasticsearch://reference/elasticsearch/index-settings/total-shards-per-node.md#total-shards-per-node) on the destination nodes. | ||
| If your allocation setting changes start relocation, wait until [shard allocation and recovery](/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery.md) finish. Use `GET /_cat/allocation?v=true&s=node` to monitor the nodes that the plan will remove. Shards might remain if you only removed a `require` rule because that change does not force them to move. The deployment plan relocates them when it disables the tier. | ||
|
|
||
| If shards that you expect to move remain on the original tier, use the [cluster allocation explain]({{es-apis}}operation/operation-cluster-allocation-explain) API to determine the cause. Common causes include [disk watermarks](/troubleshoot/elasticsearch/fix-watermark-errors.md) or the [`index.routing.allocation.total_shards_per_node`](elasticsearch://reference/elasticsearch/index-settings/total-shards-per-node.md#total-shards-per-node) limit on the destination nodes. |
There was a problem hiding this comment.
These are the common examples: https://www.elastic.co/docs/troubleshoot/elasticsearch/cluster-allocation-api-examples
| 6. Once the plan change completes, confirm that there are no remaining nodes associated with the disabled tier and that `GET _cluster/health` reports `green`. If this is the case, re-enable {{ilm-init}}. | ||
| 6. Once the plan change completes, confirm that `GET /_cat/nodes?v` shows no nodes associated with the disabled tier and that `GET /_cluster/health` reports `green`. | ||
|
|
||
| 7. Review your {{ilm-init}} policies and consider removing references to the disabled tier to keep them consistent with the deployment topology. This is especially important in older deployments where {{ilm-init}} uses node-attribute-based allocation, as those policies cannot run phases that target nodes that no longer exist. |
There was a problem hiding this comment.
FYI: Should probably do this before you start.
👻 Also need to review/modify index+legacy+component templates
| GET /_all/_ilm/explain?human=true&expand_wildcards=all&only_errors=true | ||
| ``` | ||
|
|
||
| Confirm that `operation_mode` is `RUNNING`. Investigate any reported errors and verify that no policy still attempts to allocate data to the disabled tier. |
There was a problem hiding this comment.
If you just ran a plan, you likely just start by doing a global Retry API
| * **[Partially mounted {{search-snaps}}](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) on the frozen tier:** The only way to keep the data available as indices when removing the frozen tier is to restore all partially mounted indices as regular indices on another tier. Follow the steps in this section to restore the indices and remove the original {{search-snap}} indices. | ||
| * **[Fully mounted {{search-snaps}}](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) on the cold tier:** To keep the indices as {{search-snaps}}, move them to another tier by following [Remove a tier with regular indices](#non-searchable-snapshot-data-tier). This works because fully mounted indices follow the same shard placement rules as regular indices. Alternatively, follow the steps in this section to restore them as regular indices on another tier. | ||
|
|
||
| If you do not need to preserve the data, delete the {{search-snap}} indices and continue from the shard verification before disabling the tier. |
There was a problem hiding this comment.
You'd also delete the index-backing searchable snapshot. If you only delete the index, you'll leave stray data in your repository (which adds to your bill/disk).
| ::: | ||
|
|
||
| 5. Restore indices from the {{search-snaps}}. | ||
| 5. Restore each index in the inventory from its source snapshot. |
There was a problem hiding this comment.
I'm a bit lost by this; it doesn't seem updated. This is the way old steps Support had raised concern on. Can we cross-compare https://support.elastic.dev/knowledge/view/39606517 instead. (Ex: you don't add rollover_alias you set indexing_complete otherwise you'll ILM error 😕)
|
👋 @eedugon, thanks for this! It looks great. Added notes about technical truths of certain lines and/or more links to share. Housekeeping: Apologies. As of the top of this month, my GitHub notification flow has gotten broken. So please feel free to ping me if you don't hear back as soon as you're expecting. Or TBH, if I probably missed anywhere else you or your team pinged me. Sorry! 🙈 |
2 participants
Summary
Main procedure improvements
_tier_preference, legacyrequirerules, and custom allocation filters affect relocation, including the difference between relocation triggered by index settings and by the deployment or orchestration plan.Closes #6186
Validation
git diff --check