Default ingest pipeline error handling #132093
Description
Description
When writing ingest pipelines, a best practice is to add a tag to every processor, so that in case of failure, the failure can refer to the exact processor where it occurred based on this unique tag.
This referral is done by using the on_failure block with a append processor, which appends certain information to the error.message field, providing an overview of which processors failed.
The existence of the on_failure block is equivalent to setting ignore_failure to true, meaning the rest of the pipeline will still be run.
Below is an example of a processor and failure handling that is common in integrations (although not everyone uses it).
- set:
field: tls.client.issuer
copy_from: _temp.cert
ignore_empty_value: true
tag: set_subject_organization
on_failure:
- set:
field: event.kind
value: pipeline_error
- append:
field: error.message
value: "Processor '{{{_ingest.on_failure_processor_type}}}' with tag '{{{_ingest.on_failure_processor_tag}}}' in pipeline '{{{_ingest.on_failure_pipeline}}}' failed with message '{{{_ingest.on_failure_message}}}'"
My proposal would be to implement a "default" error handling mechanism, which would make it easier to create ingest pipelines (less code) and make existing ingest pipelines more transparent when errors occur.
This could be an option added to each processor, where a new processor option default_error_handling: true would add the above on_failure block and would add the <processor_type>_<line_number> tag so the tag would always be unique.
Specific set values would take priority above the default error handling.