-
Notifications
You must be signed in to change notification settings - Fork 99
Ingest pipeline best practices #1381
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Ingest pipeline best practices #1381
Conversation
There are a couple of things I need help with.
|
Thanks a lot for opening this Philipp! I've added the "Team:Obs" label since under the new docs organization that's where the ingest content will land. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@philippkahr I started reviewing this PR, but I didn't get very far (yet!). There's a lot of content to get into! I'm just going to post the comments/questions/suggestions I have so far so I can see if I'm on the right track. I can jump back in next week.
Some themes in my early feedback include:
- I see some opportunities to simplify the examples to really emphasize the point you're making in each section.
- It might be helpful to write out in plain language what the example is trying to achieve before jumping into a code snippet. (I provided a couple suggestions below.)
- There are probably opportunities to remove redundant information.
|
||
## if statements | ||
|
||
### Contains and lots of ORs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This just seems like a best practice for writing code in general. There's nothing specific about ingest pipelines (or am I missing something)?
### Contains operation and null check | ||
|
||
This includes an initial null check, which is not necessary. | ||
|
||
```painless | ||
"if": "ctx.event?.action !=null | ||
&& ['bandwidth','spoofed syn flood prevention','dns authentication','tls attack prevention', | ||
'tcp syn flood detection','tcp connection limiting','http rate limiting', | ||
'block malformed dns traffic','tcp connection reset','udp flood detection', | ||
'dns rate limiting','malformed http filtering','icmp flood detection', | ||
'dns nxdomain rate limiting','invalid packets'].contains(ctx.event.action)" | ||
``` | ||
|
||
This behaves nearly the same: | ||
|
||
```painless | ||
"if": "['bandwidth','spoofed syn flood prevention','dns authentication','tls attack prevention', | ||
'tcp syn flood detection','tcp connection limiting','http rate limiting', | ||
'block malformed dns traffic','tcp connection reset','udp flood detection', | ||
'dns rate limiting','malformed http filtering','icmp flood detection', | ||
'dns nxdomain rate limiting','invalid packets'].contains(ctx.event?.action)" | ||
``` | ||
|
||
The difference is in the execution itself which should not matter since it is Java under the hood and pretty fast as this. In reality what happens is the following when doing the first one with the initial: `ctx.event?.action != null` If action is null, then it will exit here and not even perform the contains operation. In our second example we basically run the contains operation x times, for every item in the array and have `valueOfarray.contains('null')` then. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This example confuses me. Why would you want to run the contains operation n times if you already know ctx.event.action
is null
and it's going to return false
.
Co-authored-by: Colleen McGinnis <[email protected]>
☝️ Updating with the latest on |
Cross linking with #1727, so we can ensure the new docs complement each other properly. |
} | ||
} | ||
], | ||
"on_failure": [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feel free to ignore, but IME when users want global error handling they also want to reroute documents to a different failure index which might be nice to include.
|
||
There are various ways to handle data in ingest pipelines, and while they all produce similar results, some methods might be more suitable depending on the specific case. This section provides guidance to ensure that your ingest pipelines are consistent, readable, and maintainable. While we won't focus heavily on performance optimizations, the goal is to create pipelines that are easy to understand and manage. | ||
|
||
## Accessing Fields in `if` Statements |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"if
statements" are officially called conditionals
AND I'd like to request this be flushed out to include context:
- that errors inside
if
s don't go toignore_failure
Conditional ingest processors should respectignore_failure
for errors inif
condition elasticsearch#126005 - cross-linking to top support volume is Painless exception handling for missing keys & missing values
|
||
```painless | ||
if (ctx.user_name != null) { | ||
ctx.user.name = ctx.user_name |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
per above this if
will error if field doesn't exist (vs check is field exists but its value is NULL)
🔍 Preview links for changed docs:
🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes. |
based on the discussions here: #1052
this is my first PR against the docs, and I am building a couple of new pages. I think it makes sense to split it out. I am putting it into that part of the docs. https://www.elastic.co/docs/manage-data/ingest/transform-enrich/ingest-pipelines The tips and tricks are generic and not specific to just o11y, or security.