Ingest pipeline best practices #1381
Open
+1,130
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
There are a couple of things I need help with.
|
philippkahr
added
Team:Platform
kilfoyle
added
Team:Obs
|
Thanks a lot for opening this Philipp! I've added the "Team:Obs" label since under the new docs organization that's where the ingest content will land. |
There was a problem hiding this comment.
@philippkahr I started reviewing this PR, but I didn't get very far (yet!). There's a lot of content to get into! I'm just going to post the comments/questions/suggestions I have so far so I can see if I'm on the right track. I can jump back in next week.
Some themes in my early feedback include:
- I see some opportunities to simplify the examples to really emphasize the point you're making in each section.
- It might be helpful to write out in plain language what the example is trying to achieve before jumping into a code snippet. (I provided a couple suggestions below.)
- There are probably opportunities to remove redundant information.
|
|
||
| ## if statements | ||
|
|
||
| ### Contains and lots of ORs |
There was a problem hiding this comment.
This just seems like a best practice for writing code in general. There's nothing specific about ingest pipelines (or am I missing something)?
Comment on lines
56
to
79
| ### Contains operation and null check | ||
|
|
||
| This includes an initial null check, which is not necessary. | ||
|
|
||
| ```painless | ||
| "if": "ctx.event?.action !=null | ||
| && ['bandwidth','spoofed syn flood prevention','dns authentication','tls attack prevention', | ||
| 'tcp syn flood detection','tcp connection limiting','http rate limiting', | ||
| 'block malformed dns traffic','tcp connection reset','udp flood detection', | ||
| 'dns rate limiting','malformed http filtering','icmp flood detection', | ||
| 'dns nxdomain rate limiting','invalid packets'].contains(ctx.event.action)" | ||
| ``` | ||
|
|
||
| This behaves nearly the same: | ||
|
|
||
| ```painless | ||
| "if": "['bandwidth','spoofed syn flood prevention','dns authentication','tls attack prevention', | ||
| 'tcp syn flood detection','tcp connection limiting','http rate limiting', | ||
| 'block malformed dns traffic','tcp connection reset','udp flood detection', | ||
| 'dns rate limiting','malformed http filtering','icmp flood detection', | ||
| 'dns nxdomain rate limiting','invalid packets'].contains(ctx.event?.action)" | ||
| ``` | ||
|
|
||
| The difference is in the execution itself which should not matter since it is Java under the hood and pretty fast as this. In reality what happens is the following when doing the first one with the initial: `ctx.event?.action != null` If action is null, then it will exit here and not even perform the contains operation. In our second example we basically run the contains operation x times, for every item in the array and have `valueOfarray.contains('null')` then. |
There was a problem hiding this comment.
This example confuses me. Why would you want to run the contains operation n times if you already know ctx.event.action is null and it's going to return false.
Co-authored-by: Colleen McGinnis <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
based on the discussions here: #1052
this is my first PR against the docs, and I am building a couple of new pages. I think it makes sense to split it out. I am putting it into that part of the docs. https://www.elastic.co/docs/manage-data/ingest/transform-enrich/ingest-pipelines The tips and tricks are generic and not specific to just o11y, or security.