Confusion around allowed settings for pipelines when using Elasticsearch sources #15123
Description
Currently, we maintain two separate white-listed settings of per-pipeline settings:
- A list used when reading pipelines.yml
- A list used when retrieving the pipeline from Elasticsearch
Historically, we have limited pipeline settings retrieved from Elasticsearch to those that are exposed via the Kibana UI, and what was documented in the Kibana API
However, it appears the Kibana API allows more settings than is documented - the elastic stack Terraform provider uses the Kibana API to create pipelines, and expects to be able to use the full set of pipeline settings, as does autocomplete for the Kibana API in the dev console.
We should either determine whether we need to maintain two lists of pipeline settings, and/or fix the disconnect between the Kibana API/Terraform provider and what Logstash allows from Elasticsearch based pipeline sources