Skip to content

Commit 14205f9

Browse files
rylndjmikell821
rylnd
and
jmikell821
authored
Document Signals Migration APIs (#435)
Co-authored-by: jmikell821 <[email protected]>
1 parent d0f24d9 commit 14205f9

File tree

3 files changed

+220
-6
lines changed

3 files changed

+220
-6
lines changed

‎docs/detections/api/signals-migration-api.asciidoc

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
[[signals-migration-api]]
2+
[role="xpack"]
3+
== Detection Alerts Migration API
4+
5+
After an upgrade of {kib}, the latest {es-sec} features will be available for any new <<detection-alert-def, detection alerts>> that are generated. However, in order to enable new features on existing detection alerts, migration may be necessary. See <<release-notes, Release Notes>> for instructions specific to your upgrade.
6+
7+
Migrating detection alerts is performed at the index level and requires the following steps:
8+
9+
1. <<migration-1, Determine which indices to migrate>>
10+
2. <<migration-2, Initiate migrations>>
11+
3. <<migration-3, Finalize migrations>>
12+
4. <<migration-4, Migration cleanup>> (optional)
13+
14+
[[migration-1]]
15+
[float]
16+
=== Determine which indices to migrate
17+
You can use the Migration Status API to determine which indices contain detection alerts of a particular age, along with migration info for each of those indices.
18+
19+
[float]
20+
==== Request
21+
22+
`GET <kibana host>:<port>/api/detection_engine/signals/migration_status?from=now-30d`
23+
24+
[float]
25+
==== Request query parameters
26+
27+
[width="100%",options="header"]
28+
|==============================================
29+
|Name |Type |Description
30+
31+
|`from` |datemath|Maximum age of qualifying detection alerts
32+
|==============================================
33+
34+
[float]
35+
==== Response example
36+
37+
[source,json]
38+
--------------------------------------------------
39+
{
40+
"indices": [
41+
{
42+
"index": ".siem-signals-default-000002",
43+
"version": 15,
44+
"signal_versions": [
45+
{
46+
"version": 15,
47+
"count": 100
48+
},
49+
{
50+
"version": 16,
51+
"count": 87
52+
}
53+
],
54+
"migrations": [
55+
{
56+
"id": "924f7c50-505f-11eb-ae0a-3fa2e626a51d",
57+
"status": "pending",
58+
"version": 16,
59+
"updated": "2021-01-06T20:41:37.173Z"
60+
}
61+
],
62+
"is_outdated": true
63+
},
64+
{
65+
"index": ".siem-signals-default-000003",
66+
"version": 16,
67+
"signal_versions": [
68+
{
69+
"version": 16,
70+
"count": 54
71+
}
72+
],
73+
"migrations": [],
74+
"is_outdated": false
75+
}
76+
]
77+
}
78+
--------------------------------------------------
79+
The above response shows two indices: `.siem-signals-default-000002` is outdated, with multiple versions of detection alerts and a pending migration, and `.siem-signals-default-000003`, which is up to date as the current write index.
80+
81+
NOTE: Indices that do not contain detection alerts in the specified range will be filtered from the response.
82+
83+
With the above info, we can compile a list of indices that we wish to migrate.
84+
85+
[[migration-2]]
86+
[float]
87+
=== Initiate migrations
88+
89+
Migrations are initiated per index. While the process is neither destructive nor interferes with existing data, it may be resource-intensive. As such, it is recommended that you plan your migrations accordingly.
90+
91+
[float]
92+
==== Request
93+
94+
`POST <kibana host>:<port>/api/detection_engine/signals/migration`
95+
96+
[float]
97+
==== Request body
98+
99+
[width="100%",options="header"]
100+
|==============================================
101+
|Name |Type |Description | Required
102+
103+
|`index` |String[]|Array of index names to migrate|Yes
104+
|`size`|Integer|Number of alerts to migrate per batch. Corresponds to the `source.size` option on the {ref}/docs-reindex.html[Reindex API]|No
105+
|`requests_per_second`|Integer|The throttle for the migration task in sub-requests per second. Corresponds to `requests_per_second` on the {ref}/docs-reindex.html[Reindex API]| No
106+
|`slices`|Integer|The number of subtasks for the migration task. Corresponds to `slices` on the {ref}/docs-reindex.html[Reindex API]|No
107+
|==============================================
108+
109+
[float]
110+
==== Response example
111+
112+
[source,json]
113+
--------------------------------------------------
114+
{
115+
"indices": [
116+
{
117+
"index": ".siem-signals-default-000001",
118+
"migration_id": "923f7c50-505f-11eb-ae0a-3fa2e626a51d",
119+
"migration_index": ".siem-signals-default-000001-r000016"
120+
}
121+
]
122+
}
123+
--------------------------------------------------
124+
The response will include, for each index specified, an ID and destination index for the migration, and an error if unsuccessful.
125+
126+
[[migration-3]]
127+
[float]
128+
=== Finalize migrations
129+
130+
The finalization endpoint replaces the original index's alias with the successfully migrated index's alias. The endpoint is idempotent; therefore, it can safely be used to poll a given migration and, upon completion, finalize it.
131+
132+
NOTE: The original indices are not removed as part of this step. After verifying the integrity of the migrated index, you can use the <<migration-4, Migration cleanup>> endpoint to apply a 30-day deletion policy to the original, outdated index.
133+
134+
NOTE: If an unsuccessful migration is finalized, a deletion policy will be applied to its index, causing it to be deleted after 30 days.
135+
136+
[float]
137+
==== Request
138+
139+
`POST <kibana host>:<port>/api/detection_engine/signals/finalize_migration`
140+
141+
[float]
142+
==== Request body
143+
144+
[width="100%",options="header"]
145+
|==============================================
146+
|Name |Type |Description | Required
147+
148+
|`migration_ids` |String[]|Array of `migration_id`s to finalize|Yes
149+
|==============================================
150+
151+
[float]
152+
==== Response example
153+
154+
[source,json]
155+
--------------------------------------------------
156+
{
157+
"migrations": [
158+
{
159+
"id": "924f7c50-505f-11eb-ae0a-3fa2e626a51d",
160+
"completed": true,
161+
"destinationIndex": ".siem-signals-default-000002-r000016",
162+
"status": "success",
163+
"sourceIndex": ".siem-signals-default-000002",
164+
"version": 16,
165+
"updated": "2021-01-06T22:05:56.859Z"
166+
}
167+
]
168+
}
169+
--------------------------------------------------
170+
Finalized migrations will show a response of `completed: true`, with a corresponding `status`. If the migration is still running when you attempt to finalize it, its response will show as `completed: false`.
171+
172+
[float]
173+
[[migration-4]]
174+
=== Migration cleanup
175+
176+
Migrations favor data integrity over shard size. Consequently, unused or orphaned indices are artifacts of the migration process. A successful migration will result in both the old and new indices being present. As such, the old, orphaned index can (and likely should) be deleted.
177+
178+
While you can delete these indices manually, the endpoint accomplishes this task by applying a deletion policy to the relevant index, causing it to be deleted after 30 days. It also deletes other artifacts specific to the migration implementation.
179+
180+
[float]
181+
==== Request
182+
183+
`DELETE <kibana host>:<port>/api/detection_engine/signals/migration`
184+
185+
[float]
186+
==== Request body
187+
188+
[width="100%",options="header"]
189+
|==============================================
190+
|Name |Type |Description | Required
191+
192+
|`migration_ids` |String[]|Array of `migration_id`s to finalize|Yes
193+
|==============================================
194+
195+
[float]
196+
==== Response example
197+
198+
[source,json]
199+
--------------------------------------------------
200+
{
201+
"migrations": [
202+
{
203+
"id": "924f7c50-505f-11eb-ae0a-3fa2e626a51d",
204+
"destinationIndex": ".siem-signals-default-000002-r000016",
205+
"status": "success",
206+
"sourceIndex": ".siem-signals-default-000002",
207+
"version": 16,
208+
"updated": "2021-01-06T22:05:56.859Z"
209+
}
210+
]
211+
}
212+
--------------------------------------------------
213+
The response will include all migrations that were successfully deleted.

‎docs/post-upgrade-req.asciidoc

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,10 @@
11
[[post-upgrade-req]]
22
[role="xpack"]
3-
= Enable process analyzer after upgrade
3+
= Enable process analyzer after an upgrade
44

5-
To enable <<alerts-analyze-events, graphical representations of process relationships>>
6-
after upgrading to {stack} version 7.9.0 or 7.9.1 from a
7-
previous minor release (7.8.x, 7.7.x, and so on), you need to update
8-
`.siem-signals*` system index mappings.
5+
After upgrading from {stack} version 7.9.x to >= 7.10.x from a previous minor release (e.g., 7.8.x, etc.), you need to update `.siem-signals*` system index mappings to enable <<alerts-analyze-events, graphical representations of process relationships>>.
6+
7+
NOTE: If you are upgrading from a minor release to {stack} version >= 7.11.0, there is now a <<signals-migration-api>> that you can use instead of the manual process described below.
98

109
To update the `.siem-signals*` index:
1110

‎docs/siem-apis.asciidoc

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ You can use these APIs to interface with {es-sec} features:
66

77
* <<rule-api-overview>>: Manage detection rules and alerts
88
* <<exceptions-api-overview>>: Create and manage rule exceptions
9-
* <<lists-api-overview>>: Create source event value lists for use with rule exceptions
9+
* <<lists-api-overview>>: Create source event value lists for use with rule exceptions
1010
* <<timeline-api-overview>>: Import and export timelines
1111
* <<cases-api-overview>>: Open and manage cases
1212

@@ -80,6 +80,8 @@ include::detections/api/exceptions-api-index.asciidoc[]
8080

8181
include::detections/api/lists-api-index.asciidoc[]
8282

83+
include::detections/api/signals-migration-api.asciidoc[]
84+
8385
include::events/api/timeline-api-index.asciidoc[]
8486

8587
include::cases/api/cases-api/cases-api-index.asciidoc[]

0 commit comments

Comments
 (0)