Add documentation for failure stores. #1368

jbaiera · 2025-05-05T22:26:38Z

Adds a new section to the documentation to explain new failure store functionality.

Preview:
https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/1368/manage-data/data-store/data-streams/failure-store

This PR relies on links to updates in:

Add docs for index component selector api convention elasticsearch#128786
[Failure store] Add data stream options to the es specs elasticsearch-specification#4414

TBD on recipes. Most links are not complete and need updating from "???".

manage-data/data-store/data-streams/failure-store.md

Co-authored-by: Lee Hinman <[email protected]>

PeteGillinElastic

This is looking good so far! I have one drive-by comment.

manage-data/data-store/data-streams/failure-store.md

bvader · 2025-05-19T19:33:54Z

Hi @jbaiera I noticed a small issue

https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/1368/manage-data/data-store/data-streams/failure-store#use-failure-store

This example

POST my-datastream/_bulk
{"create":{}}
{"@timestamp": "2025-05-01T00:00:00Z", "id": 1234}
{"create":{}}
{"@timestamp": "2025-05-01T00:00:00Z", "id": "invalid_text"}

Should be add the -new (<namespace>) otherwise it does not match the template at the top of the page and thus does not match and thus does not work :)

POST my-datastream-new/_bulk
{"create":{}}
{"@timestamp": "2025-05-01T00:00:00Z", "id": 1234}
{"create":{}}
{"@timestamp": "2025-05-01T00:00:00Z", "id": "invalid_text"}

manage-data/data-store/data-streams/failure-store.md

deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md

slobodanadamovic · 2025-06-03T20:33:40Z

Is there anywhere else that we should expand with more failure store info for security beyond that reference?

No, that's the only place we should document them.

I left some comments. I think we should try to point out that the new privileges are only granting access to failure store when accessed using ::failures selector, but I'm blanking on a concrete suggestion right now. Will try to think overnight.

dakrone

LGTM, I left some comments, but nothing major. This is a monumental amount of work and I think it's great, thanks Jimmy.

manage-data/data-store/data-streams/failure-store-recipes.md

manage-data/data-store/data-streams/failure-store.md

sabarasaba · 2025-06-19T16:29:44Z

@dakrone Any chance we might get this before FF for 8.19? We've made some work in the kibana UI and would be great if we can add a link to the docs

shainaraskas

hey @jbaiera! just adding a couple of comments to address the metadata of these pages / improve wayfinding between the pages. also approving to make sure that you're unblocked. let me know if you need a hand with anything.

manage-data/data-store/data-streams/failure-store.md

manage-data/data-store/data-streams/failure-store-recipes.md

manage-data/data-store/data-streams/failure-store.md

github-actions · 2025-06-20T17:41:41Z

🔍 Preview links for changed docs:

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

shainaraskas

to fix the failing test

manage-data/data-store/data-streams/failure-store.md

…ream (#221644) Added a tooltip and a link with documentation for Failed docs column when dataset does not have failure store enabled. https://github.com/user-attachments/assets/be65db9a-15c8-4087-b175-752b2fabab6e For now it awaits for the documentation PR to be merged : elastic/docs-content#1368 --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Yngrid Coello <[email protected]>

…ream (elastic#221644) Added a tooltip and a link with documentation for Failed docs column when dataset does not have failure store enabled. https://github.com/user-attachments/assets/be65db9a-15c8-4087-b175-752b2fabab6e For now it awaits for the documentation PR to be merged : elastic/docs-content#1368 --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Yngrid Coello <[email protected]> (cherry picked from commit 36cbbb9)

…data stream (#221644) (#225007) # Backport This will backport the following commits from `main` to `8.19`: - [[Dataset Quality] Indicate if failure store isn't enabled for data stream (#221644)](#221644)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Robert Stelmach <[email protected]> Co-authored-by: Yngrid Coello <[email protected]>

…ream (elastic#221644) Added a tooltip and a link with documentation for Failed docs column when dataset does not have failure store enabled. https://github.com/user-attachments/assets/be65db9a-15c8-4087-b175-752b2fabab6e For now it awaits for the documentation PR to be merged : elastic/docs-content#1368 --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Yngrid Coello <[email protected]>

Co-authored-by: Lee Hinman <[email protected]>

Co-authored-by: shainaraskas <[email protected]>

manage-data/data-store/data-streams/failure-store-recipes.md

shainaraskas

removing links to anchors to get the PR green

deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md

manage-data/data-store/data-streams/failure-store.md

fix for #1368

[WIP] Add documentation for failure stores.

b3f8af4

TBD on recipes. Most links are not complete and need updating from "???".

jbaiera added documentation Improvements or additions to documentation wip labels May 5, 2025

github-actions bot deployed to docs-preview May 5, 2025 22:27 View deployment

Change ??? to "wip"

32d9336

github-actions bot deployed to docs-preview May 6, 2025 06:45 View deployment

jbaiera added 3 commits May 6, 2025 10:03

Fix wip links

356cd62

fix wip please?

6c18018

fix

075abe2

dakrone reviewed May 7, 2025

View reviewed changes

felixbarny reviewed May 8, 2025

View reviewed changes

jbaiera and others added 12 commits May 8, 2025 15:43

Update manage-data/data-store/data-streams/failure-store.md

eac4005

Co-authored-by: Lee Hinman <[email protected]>

clarify that we do not redirect backpressure or version conflicts

dfbefc1

Update manage-data/data-store/data-streams/failure-store.md

2019367

Co-authored-by: Lee Hinman <[email protected]>

Update manage-data/data-store/data-streams/failure-store.md

2847fe9

Co-authored-by: Lee Hinman <[email protected]>

suspended -> disabled

840a8be

reword document.source docs

1591e7b

clarify default retention application

cec305f

Add placeholder link for put data streams api

f1cb727

Mention flags in the intro

28287fe

Note lazy creation

08b2ac3

edit configuration section for more clarity

e978f6e

mention override directly

2db178a

PeteGillinElastic reviewed May 12, 2025

View reviewed changes

manage-data/data-store/data-streams/failure-store.md Outdated Show resolved Hide resolved

bvader mentioned this pull request May 19, 2025

Redirect failed ingest node operations to a failure store when available elastic/elasticsearch#103481

Merged

bvader reviewed May 19, 2025

View reviewed changes

manage-data/data-store/data-streams/failure-store.md Outdated Show resolved Hide resolved

rStelmach mentioned this pull request May 27, 2025

[Dataset Quality] Indicate if failure store isn't enabled for data stream elastic/kibana#221644

Merged

Add recipes to the failure store narrative docs

87f3a47

kilfoyle reviewed May 28, 2025

View reviewed changes

manage-data/data-store/data-streams/failure-store.md Outdated Show resolved Hide resolved

slobodanadamovic reviewed Jun 3, 2025

View reviewed changes

deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md Outdated Show resolved Hide resolved

dakrone self-requested a review June 16, 2025 16:59

dakrone approved these changes Jun 16, 2025

View reviewed changes

shainaraskas approved these changes Jun 20, 2025

View reviewed changes

Merge branch 'main' into failure-store-docs

c338df6

shainaraskas reviewed Jun 20, 2025

View reviewed changes

manage-data/data-store/data-streams/failure-store.md Outdated Show resolved Hide resolved

sabarasaba mentioned this pull request Jun 23, 2025

[Index Management] Remove doc link from failure store configuration modal elastic/kibana#224941

Merged

jbaiera and others added 4 commits June 25, 2025 15:03

Update manage-data/data-store/data-streams/failure-store.md

8675b51

Co-authored-by: Lee Hinman <[email protected]>

Updates for security docs

3a88cc7

Review comments

d521fd7

Updates from docs team

72de0d1

Co-authored-by: shainaraskas <[email protected]>

shainaraskas reviewed Jun 26, 2025

View reviewed changes

manage-data/data-store/data-streams/failure-store-recipes.md Outdated Show resolved Hide resolved

shainaraskas mentioned this pull request Jun 26, 2025

Strategy for linking to serverless vs stateful APIs in narrative/procedue docs #1942

Open

jbaiera added 2 commits June 26, 2025 14:48

Fix link

d741c52

Merge branch 'main' into failure-store-docs

9ba44ae

jbaiera enabled auto-merge (squash) June 26, 2025 20:48

slobodanadamovic approved these changes Jun 26, 2025

View reviewed changes

shainaraskas reviewed Jun 26, 2025

View reviewed changes

Apply suggestions from code review

a99f884

jbaiera merged commit 2dfdbe0 into main Jun 26, 2025
7 of 8 checks passed

jbaiera deleted the failure-store-docs branch June 26, 2025 21:21

shainaraskas mentioned this pull request Jun 26, 2025

readd cs anchors #1947

Merged

shainaraskas added a commit that referenced this pull request Jun 26, 2025

readd cs anchors (#1947)

b89cc3f

fix for #1368

Add documentation for failure stores. #1368

Add documentation for failure stores. #1368

Uh oh!

Conversation

jbaiera commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PeteGillinElastic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bvader commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slobodanadamovic commented Jun 3, 2025

Uh oh!

dakrone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sabarasaba commented Jun 19, 2025

Uh oh!

shainaraskas left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shainaraskas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shainaraskas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbaiera commented May 5, 2025 •

edited

Loading

bvader commented May 19, 2025 •

edited

Loading

shainaraskas left a comment •

edited

Loading

github-actions bot commented Jun 20, 2025 •

edited

Loading