HTTP Filter Plugin - Request/Response Metadata #14368
Description
The HTTP Poller Input Plugin provides useful request/response metadata that can be used for troubleshooting and error handling further in the pipeline.
It would be very useful to have this data available for the HTTP Filter Plugin, as it will provide the ability to do meaningful error handling of events based on the HTTP Status code returned. For a scenario where an endpoint will return 400 for a bad query and 404 for no results, there is no ability for the customer to determine whether an error has occurred or a valid 'no records' response was received. IN the case of the 404 being received, a certain document should be created and sent to Elasticsearch to indicate this. In the case of an 400, no document should be sent.
Realise that this information is captured in the Logstash logs, however it is not useful in this scenario as the event and log entry cannot be tied together easily as part of the data flow.
We are unable to use the HTTP Poller for the solution due to the nature of the data flow. Note: This has been lodged on behalf of a customer.