Add documentation for failure stores. #1368

jbaiera · 2025-05-05T22:26:38Z

Adds a new section to the documentation to explain new failure store functionality.

Preview:
https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/1368/manage-data/data-store/data-streams/failure-store

This PR relies on links to updates in:

Add docs for index component selector api convention elasticsearch#128786
[Failure store] Add data stream options to the es specs elasticsearch-specification#4414

TBD on recipes. Most links are not complete and need updating from "???".

manage-data/data-store/data-streams/failure-store.md

Co-authored-by: Lee Hinman <[email protected]>

manage-data/data-store/data-streams/failure-store-recipes.md

kilfoyle

@jbaiera this is such a superb doc! I've left a bunch of super small suggestions but overall it looks great to me! 🚀

Co-authored-by: David Kilfoyle <[email protected]>

jbaiera · 2025-06-02T21:27:50Z

Adding @slobodanadamovic as reviewer for the new roles in the reference docs. Is there anywhere else that we should expand with more failure store info for security beyond that reference?

slobodanadamovic · 2025-06-03T20:16:28Z

manage-data/data-store/data-streams/failure-store.md

+:::{warning}
+Documents redirected to the failure store in the event of a failed ingest pipeline will be stored in their original, unprocessed form. If an ingest pipeline normally redacts sensitive information from a document, then failed documents in their original, unprocessed form may contain sensitive information.
+
+Furthermore, failed documents are likely to be structured differently than normal data in a data stream, and thus are not supported by [document level security](../../../deploy-manage/users-roles/cluster-or-deployment-auth/controlling-access-at-document-field-level.md#document-level-security) or [field level security](../../../deploy-manage/users-roles/cluster-or-deployment-auth/controlling-access-at-document-field-level.md#field-level-security).


This is actually not true. We decided to support FLS and DLS against failure store. The main reason is because it would not be possible to prevent using FLS/DLS when users define implicit read access to backing .fs* indices. It felt wrong to prevent using FLS/DLS when users defined explicit read_failure_store access to data streams with FLS/DLS restrictions. Our biggest concern was that users would be expecting the DLS/FLS to stop certain docs/fields from being visible and it wouldn't.

Right now, if users include FLS/DLS when granting access to failure store then we'll try to honour it. We should just make sure to highlight it (which you already did) that these documents are structured differently, and because they may contain sensitive information, the users should take extra care when defining access to them.

slobodanadamovic · 2025-06-03T20:23:45Z

deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md

@@ -381,6 +384,8 @@ To learn how to assign privileges to a role, refer to [](/deploy-manage/users-ro

    This privilege is not available in {{serverless-full}}.

+`read_failure_store`
+:   Read-only access to actions performed on a data stream's failure store. Required for access to failure store data (count, explain, get, mget, get indexed scripts, more like this, multi percolate/search/termvector, percolate, scroll, clear_scroll, search, suggest, tv).


The manage_failure_store and read_failure_store privileges are special in terms that they only grant access to failure store when accessed through data stream names using ::failures selector. We should try and highlight somehow that these privileges cannot be used to grant direct read/manage access to failure store backing indices (.fs*) or any other regular indices. Hence, they should only be granted to data streams that have failure store enabled.

slobodanadamovic · 2025-06-03T20:33:40Z

Is there anywhere else that we should expand with more failure store info for security beyond that reference?

No, that's the only place we should document them.

I left some comments. I think we should try to point out that the new privileges are only granting access to failure store when accessed using ::failures selector, but I'm blanking on a concrete suggestion right now. Will try to think overnight.

dakrone

LGTM, I left some comments, but nothing major. This is a monumental amount of work and I think it's great, thanks Jimmy.

dakrone · 2025-06-16T17:25:26Z

manage-data/data-store/data-streams/failure-store-recipes.md

+  },
+  "_seq_no": 2,
+  "_primary_term": 1,
+  "failure_store": "used" // The document was sent to the failure store.


In this document, can you use the <1> annotation to put the captions at the bottom? I think it's easier and we have (I think) a special CSS syntax so that it doesn't show up in copy-and-pasting.

dakrone · 2025-06-16T17:25:54Z

manage-data/data-store/data-streams/failure-store-recipes.md

+          "@timestamp": "2025-05-09T06:24:48.381Z",
+          "document": {
+            "index": "my-datastream-ingest",
+            "source": { // When an ingest pipeline fails, the document stored is what was originally sent to the cluster.


Same here (and below) about using <1>, <2>, etc.

dakrone · 2025-06-16T17:33:58Z

manage-data/data-store/data-streams/failure-store-recipes.md

+
+**Separate your failures beforehand.** As described in the previous [failure document source](./failure-store.md#use-failure-store-document-source) section, failure documents are structured differently depending on when the document failed during ingestion. We recommend to separate documents by ingest pipeline failures and indexing failures at minimum. Ingest pipeline failures often need to have the original pipeline re-run, while index failures should skip any pipelines. Further separating failures by index or specific failure type may also be beneficial.
+
+**Perform a failure store rollover.** Consider rolling over the failure store before attempting to remediate failures. This will create a new failure index that will collect any new failures during the remediation process.


Can you link to the section you wrote about rolling over the failure store here?

dakrone · 2025-06-16T17:34:43Z

manage-data/data-store/data-streams/failure-store-recipes.md

+
+**Use an ingest pipeline to convert failure documents back into their original document.** Failure documents store failure information along with the document that failed ingestion. The first step for remediating documents should be to use an ingest pipeline to extract the original source from the failure document and then discard any other information about the failure.
+
+**Simulate first to avoid repeat failures.** If you must run a pipeline as part of your remediation process, it is best to simulate the pipeline against the failure first. This will catch any unforeseen issues that may fail the document a second time. Remember, ingest pipeline failures will capture the document before an ingest pipeline is applied to it, which can further complicate remediation when a failure document becomes nested inside a new failure.


Same here about linking to the simulate ingest API.

dakrone · 2025-06-16T22:21:39Z

manage-data/data-store/data-streams/failure-store-recipes.md

+1. The failures have been remediated.
+
+:::{tip}
+Since the failure store is enabled on this data stream, it would be wise to check it for any further failures from the reindexing process. Failures that happen at this point in the process may end up as nested failures in the failure store. Remediating nested failures can quickly become a hassle as the original document gets nested multiple levels deep in the failure document. For this reason, it is suggested to remediate data during a quiet period when no other failures are likely to arise. Furthermore, rolling over the failure store before executing the remediation would allow easier discarding of any new nested failures and only operate on the original failure documents.


I think it would also be good to mention that reindexing the failed documents into the data stream does not remove the failures, so that needs to be done after the reindexing.

dakrone · 2025-06-16T22:30:00Z

manage-data/data-store/data-streams/failure-store.md

+---
+applies_to:
+  stack: ga 8.19.0
+  serverless: ga 9.1.0


Does Serverless have versions for the docs?

dakrone · 2025-06-16T22:30:49Z

manage-data/data-store/data-streams/failure-store.md

+
+A failure store is a secondary set of indices inside a data stream, dedicated to storing failed documents. A failed document is any document that, without the failure store enabled, would cause an ingest pipeline exception or that has a structure that conflicts with a data stream's mappings. In the absence of the failure store, a failed document would cause the indexing operation to fail, with an error message returned in the operation response.
+
+When a data stream's failure store is enabled, these failures are instead captured in a separate index and persisted to be analysed later. Clients receive a successful response with a flag indicating the failure was redirected. Failure stores do not capture failures caused by backpressure or document version conflicts. These failures are always returned as-is since they warrant specific action by the client.


I think it would be worth separating out the Failure stores do not capture failures caused by backpressure or document version conflicts. These failures are always returned as-is since they warrant specific action by the client. section into a "note" or "important" block, so it doesn't get missed.

dakrone · 2025-06-16T22:32:02Z

manage-data/data-store/data-streams/failure-store.md

+You can specify in a data stream's [index template](../templates.md) if it should enable the failure store when it is first created.
+
+:::{note}
+Unlike the `settings` and `mappings` fields on an [index template](../templates.md) which are repeatedly applied to new data stream write indices on rollover, the `data_stream_options` section of a template is applied to a data stream only once when the data stream is first created. To configure existing data streams, use the put [data stream options API](indices-put-data-stream-options).


Will the indices-put-data-stream-options link work? Or do we need to fully-qualify it with the URL?

dakrone · 2025-06-16T22:33:02Z

manage-data/data-store/data-streams/failure-store.md

+```
+```console


I think these two lines need to be removed to make this one continuous console block, since we use <2> below

dakrone · 2025-06-16T22:34:10Z

manage-data/data-store/data-streams/failure-store.md

+
+### Searching failures [use-failure-store-searching]
+
+Once you have accumulated some failures, they can be searched much like a regular index.


Suggested change

Once you have accumulated some failures, they can be searched much like a regular index.

Once you have accumulated some failures, the failure store can be searched much like a regular data stream.

sabarasaba · 2025-06-19T16:29:44Z

@dakrone Any chance we might get this before FF for 8.19? We've made some work in the kibana UI and would be great if we can add a link to the docs

shainaraskas

hey @jbaiera! just adding a couple of comments to address the metadata of these pages / improve wayfinding between the pages. also approving to make sure that you're unblocked. let me know if you need a hand with anything.

shainaraskas · 2025-06-20T17:22:17Z

manage-data/data-store/data-streams/failure-store.md

+---
+applies_to:
+  stack: ga 8.19.0
+  serverless: ga 9.1.0
+---


to add to @dakrone's comments, we need to tweak this a little

8.19 docs for this feature should be written in the old docs system so we should be just marking this as 9.1+ in this system

when we have the 8.19 docs, we can use a mapped_pages tag to create a link from the old system to the new system

serverless is unversioned, so no need to specify a version #

we added a products tag since this PR was opened which will be used to filter search results

Suggested change

---

applies_to:

stack: ga 8.19.0

serverless: ga 9.1.0

---

---

mapped_pages:

- (8.19 docs)

applies_to:

stack: ga 9.1

serverless: ga

products:

- id: elasticsearch

- id: elastic-stack

- id: cloud-serverless

---

shainaraskas · 2025-06-20T17:23:03Z

manage-data/data-store/data-streams/failure-store-recipes.md

@@ -0,0 +1,1154 @@
+# Failure store recipes and use cases [failure-store-recipes]


This page also needs front matter. the front matter helps people who land on the page from google to understand if the content applies to them. this page could also have a mapped_pages if you're creating an equivalent page in the 8.19 docs - that can always be added in a subsequent PR as well

Suggested change

# Failure store recipes and use cases [failure-store-recipes]

---

mapped_pages:

- (8.19 docs)

applies_to:

stack: ga 9.1

serverless: ga

products:

- id: elasticsearch

- id: elastic-stack

- id: cloud-serverless

---

# Failure store recipes and use cases [failure-store-recipes]

:2c: I don't love the word "recipes" here - it's not great for an international audience. Perhaps "Using failure stores to address ingestion issues" or something.

shainaraskas · 2025-06-20T17:26:58Z

manage-data/data-store/data-streams/failure-store.md

+
+A failure store is a secondary set of indices inside a data stream, dedicated to storing failed documents. A failed document is any document that, without the failure store enabled, would cause an ingest pipeline exception or that has a structure that conflicts with a data stream's mappings. In the absence of the failure store, a failed document would cause the indexing operation to fail, with an error message returned in the operation response.
+
+When a data stream's failure store is enabled, these failures are instead captured in a separate index and persisted to be analysed later. Clients receive a successful response with a flag indicating the failure was redirected. Failure stores do not capture failures caused by backpressure or document version conflicts. These failures are always returned as-is since they warrant specific action by the client.


try to link from this overview page to your recipes page - otherwise, people might not know to look for it. we should try not to rely on the sidebar.

I suggest introducing it in the overview here - we can explain the scope of this page, and then explain that there is another page and what they can find on it.

Suggested change

When a data stream's failure store is enabled, these failures are instead captured in a separate index and persisted to be analysed later. Clients receive a successful response with a flag indicating the failure was redirected. Failure stores do not capture failures caused by backpressure or document version conflicts. These failures are always returned as-is since they warrant specific action by the client.

When a data stream's failure store is enabled, these failures are instead captured in a separate index and persisted to be analysed later. Clients receive a successful response with a flag indicating the failure was redirected. Failure stores do not capture failures caused by backpressure or document version conflicts. These failures are always returned as-is since they warrant specific action by the client.

On this page, you'll learn how to set up, use, and manage a failure store, as well as the structure of failure store documents.

For examples of how to use failure stores to identify and fix errors in ingest pipelines and your data, refer to [](/manage-data/data-store/data-streams/failure-store-recipes.md).

shainaraskas · 2025-06-20T17:27:34Z

manage-data/data-store/data-streams/failure-store.md

+
+## Using a failure store [use-failure-store]
+
+The failure store is meant to ease the burden of detecting and handling failures when ingesting data to {{es}}. Clients are less likely to encounter unrecoverable failures when writing documents, and developers are more easily able to troubleshoot faulty pipelines and mappings.


Suggested change

The failure store is meant to ease the burden of detecting and handling failures when ingesting data to {{es}}. Clients are less likely to encounter unrecoverable failures when writing documents, and developers are more easily able to troubleshoot faulty pipelines and mappings.

The failure store is meant to ease the burden of detecting and handling failures when ingesting data to {{es}}. Clients are less likely to encounter unrecoverable failures when writing documents, and developers are more easily able to troubleshoot faulty pipelines and mappings.

For examples of how to use failure stores to identify and fix errors in ingest pipelines and your data, refer to [](/manage-data/data-store/data-streams/failure-store-recipes.md).

github-actions · 2025-06-20T17:41:41Z

🔍 Preview links for changed docs:

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

shainaraskas

to fix the failing test

shainaraskas · 2025-06-20T17:45:02Z

manage-data/data-store/data-streams/failure-store.md

+You can specify in a data stream's [index template](../templates.md) if it should enable the failure store when it is first created.
+
+:::{note}
+Unlike the `settings` and `mappings` fields on an [index template](../templates.md) which are repeatedly applied to new data stream write indices on rollover, the `data_stream_options` section of a template is applied to a data stream only once when the data stream is first created. To configure existing data streams, use the put [data stream options API](indices-put-data-stream-options).


Suggested change

Unlike the `settings` and `mappings` fields on an [index template](../templates.md) which are repeatedly applied to new data stream write indices on rollover, the `data_stream_options` section of a template is applied to a data stream only once when the data stream is first created. To configure existing data streams, use the put [data stream options API](indices-put-data-stream-options).

Unlike the `settings` and `mappings` fields on an [index template](../templates.md) which are repeatedly applied to new data stream write indices on rollover, the `data_stream_options` section of a template is applied to a data stream only once when the data stream is first created. To configure existing data streams, use the put [data stream options API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-stream-options).

wonder if we need links to the serverless versions of these APIs too
https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-put-data-stream-options

[WIP] Add documentation for failure stores.

b3f8af4

TBD on recipes. Most links are not complete and need updating from "???".

jbaiera added documentation Improvements or additions to documentation wip labels May 5, 2025

github-actions bot deployed to docs-preview May 5, 2025 22:27 View deployment

Change ??? to "wip"

32d9336

github-actions bot deployed to docs-preview May 6, 2025 06:45 View deployment

Fix wip links

356cd62

github-actions bot deployed to docs-preview May 6, 2025 14:03 View deployment

fix wip please?

6c18018

github-actions bot deployed to docs-preview May 6, 2025 17:35 View deployment

fix

075abe2

github-actions bot deployed to docs-preview May 6, 2025 19:45 View deployment

dakrone reviewed May 7, 2025

View reviewed changes

felixbarny reviewed May 8, 2025

View reviewed changes

Update manage-data/data-store/data-streams/failure-store.md

eac4005

Co-authored-by: Lee Hinman <[email protected]>

github-actions bot deployed to docs-preview May 8, 2025 19:44 View deployment

jbaiera and others added 3 commits May 8, 2025 15:51

clarify that we do not redirect backpressure or version conflicts

dfbefc1

Update manage-data/data-store/data-streams/failure-store.md

2019367

Co-authored-by: Lee Hinman <[email protected]>

Update manage-data/data-store/data-streams/failure-store.md

2847fe9

Co-authored-by: Lee Hinman <[email protected]>

github-actions bot deployed to docs-preview May 8, 2025 19:52 View deployment

suspended -> disabled

840a8be

github-actions bot deployed to docs-preview May 8, 2025 19:58 View deployment

reword document.source docs

1591e7b

github-actions bot deployed to docs-preview May 8, 2025 20:28 View deployment

jbaiera added 2 commits May 8, 2025 16:29

clarify default retention application

cec305f

Add placeholder link for put data streams api

f1cb727

github-actions bot deployed to docs-preview May 8, 2025 20:33 View deployment

Mention flags in the intro

28287fe

github-actions bot deployed to docs-preview May 8, 2025 21:11 View deployment

Note lazy creation

08b2ac3

kilfoyle reviewed May 29, 2025

View reviewed changes

manage-data/data-store/data-streams/failure-store-recipes.md Outdated Show resolved Hide resolved

kilfoyle reviewed May 29, 2025

View reviewed changes

manage-data/data-store/data-streams/failure-store-recipes.md Outdated Show resolved Hide resolved

kilfoyle reviewed May 29, 2025

View reviewed changes

manage-data/data-store/data-streams/failure-store-recipes.md Outdated Show resolved Hide resolved

kilfoyle approved these changes May 29, 2025

View reviewed changes

Apply suggestions from code review

959bfd8

Co-authored-by: David Kilfoyle <[email protected]>

github-actions bot deployed to docs-preview May 30, 2025 19:40 View deployment

jbaiera added 2 commits May 30, 2025 15:42

typo

fd8939e

Update links and add privilege info

ccda059

github-actions bot deployed to docs-preview June 2, 2025 20:58 View deployment

jbaiera mentioned this pull request Jun 2, 2025

Add docs for index component selector api convention elastic/elasticsearch#128786

Open

Update example

86c4c7e

github-actions bot deployed to docs-preview June 2, 2025 21:25 View deployment

jbaiera marked this pull request as ready for review June 2, 2025 21:26

jbaiera requested review from a team as code owners June 2, 2025 21:26

jbaiera requested a review from slobodanadamovic June 2, 2025 21:26

jbaiera removed the wip label Jun 2, 2025

slobodanadamovic reviewed Jun 3, 2025

View reviewed changes

dakrone self-requested a review June 16, 2025 16:59

dakrone approved these changes Jun 16, 2025

View reviewed changes

github-actions bot deployed to docs-preview June 20, 2025 15:00 View deployment

shainaraskas approved these changes Jun 20, 2025

View reviewed changes

Merge branch 'main' into failure-store-docs

c338df6

github-actions bot deployed to docs-preview June 20, 2025 17:42 View deployment

shainaraskas reviewed Jun 20, 2025

View reviewed changes


		Separate your failures beforehand. As described in the previous [failure document source](./failure-store.md#use-failure-store-document-source) section, failure documents are structured differently depending on when the document failed during ingestion. We recommend to separate documents by ingest pipeline failures and indexing failures at minimum. Ingest pipeline failures often need to have the original pipeline re-run, while index failures should skip any pipelines. Further separating failures by index or specific failure type may also be beneficial.

		Perform a failure store rollover. Consider rolling over the failure store before attempting to remediate failures. This will create a new failure index that will collect any new failures during the remediation process.


		Use an ingest pipeline to convert failure documents back into their original document. Failure documents store failure information along with the document that failed ingestion. The first step for remediating documents should be to use an ingest pipeline to extract the original source from the failure document and then discard any other information about the failure.

		Simulate first to avoid repeat failures. If you must run a pipeline as part of your remediation process, it is best to simulate the pipeline against the failure first. This will catch any unforeseen issues that may fail the document a second time. Remember, ingest pipeline failures will capture the document before an ingest pipeline is applied to it, which can further complicate remediation when a failure document becomes nested inside a new failure.


		A failure store is a secondary set of indices inside a data stream, dedicated to storing failed documents. A failed document is any document that, without the failure store enabled, would cause an ingest pipeline exception or that has a structure that conflicts with a data stream's mappings. In the absence of the failure store, a failed document would cause the indexing operation to fail, with an error message returned in the operation response.

		When a data stream's failure store is enabled, these failures are instead captured in a separate index and persisted to be analysed later. Clients receive a successful response with a flag indicating the failure was redirected. Failure stores do not capture failures caused by backpressure or document version conflicts. These failures are always returned as-is since they warrant specific action by the client.


		### Searching failures [use-failure-store-searching]

		Once you have accumulated some failures, they can be searched much like a regular index.

----
-applies_to:
-  stack: ga 8.19.0
-  serverless: ga 9.1.0
----
+---
+mapped_pages:
+  - (8.19 docs)
+applies_to:
+  stack: ga 9.1
+  serverless: ga
+products:
+  - id: elasticsearch
+  - id: elastic-stack
+  - id: cloud-serverless
+---

		@@ -0,0 +1,1154 @@
		# Failure store recipes and use cases [failure-store-recipes]


		## Using a failure store [use-failure-store]

		The failure store is meant to ease the burden of detecting and handling failures when ingesting data to {{es}}. Clients are less likely to encounter unrecoverable failures when writing documents, and developers are more easily able to troubleshoot faulty pipelines and mappings.

		```
		```console

Add documentation for failure stores. #1368

Are you sure you want to change the base?

Add documentation for failure stores. #1368

Uh oh!

Conversation

jbaiera commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kilfoyle left a comment

Choose a reason for hiding this comment

Uh oh!

jbaiera commented Jun 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slobodanadamovic commented Jun 3, 2025

Uh oh!

dakrone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sabarasaba commented Jun 19, 2025

Uh oh!

shainaraskas left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shainaraskas Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shainaraskas Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 20, 2025

Uh oh!

jbaiera commented May 5, 2025 •

edited

Loading

shainaraskas left a comment •

edited

Loading

shainaraskas Jun 20, 2025 •

edited

Loading

shainaraskas Jun 20, 2025 •

edited

Loading