Add support of Processors in all the Observability Integrations

[ishleenk17]

Processors are used to provide the ability to the user to enrich their data.
Currently, there is inconsistency in the Integrations wrt availability of Processors.

• Identify the Packages/datastreams missing the Processors
• Do bulk addition of Processors to the Integrations
• Test the changes

Expectation: Processors yaml box like below should be present in all Integrations

[Linu-Elias]

The list of packages with missing processor:

• Activemq
  https://github.com/elastic/integrations/blob/main/packages/activemq/data_stream/broker
  https://github.com/elastic/integrations/blob/main/packages/activemq/data_stream/queue
  https://github.com/elastic/integrations/blob/main/packages/activemq/data_stream/topic

• Apache
  https://github.com/elastic/integrations/blob/main/packages/apache/data_stream/status

• Apache Spark
  https://github.com/elastic/integrations/blob/main/packages/apache_spark/data_stream/application
  https://github.com/elastic/integrations/blob/main/packages/apache_spark/data_stream/driver
  https://github.com/elastic/integrations/blob/main/packages/apache_spark/data_stream/executor
  https://github.com/elastic/integrations/blob/main/packages/apache_spark/data_stream/node

• AWSFargate
  https://github.com/elastic/integrations/blob/main/packages/awsfargate/data_stream/task_stats

• Azure Application Insights
  https://github.com/elastic/integrations/blob/main/packages/azure_application_insights/data_stream/app_insights
  https://github.com/elastic/integrations/blob/main/packages/azure_application_insights/data_stream/app_state

• Azure Billing
  https://github.com/elastic/integrations/blob/main/packages/azure_billing/data_stream/billing

• Azure Functions
  https://github.com/elastic/integrations/blob/main/packages/azure_functions/data_stream/metrics

• Azure OpenAI
  https://github.com/elastic/integrations/blob/main/packages/azure_openai/data_stream/metrics

• Cassandra
  https://github.com/elastic/integrations/blob/main/packages/cassandra/data_stream/metrics

• CockroachDB
  https://github.com/elastic/integrations/blob/main/packages/cockroachdb/data_stream/status

• Etcd
  https://github.com/elastic/integrations/blob/main/packages/etcd/data_stream/leader
  https://github.com/elastic/integrations/blob/main/packages/etcd/data_stream/metrics
  https://github.com/elastic/integrations/blob/main/packages/etcd/data_stream/self
  https://github.com/elastic/integrations/blob/main/packages/etcd/data_stream/store

• HAProxy
  https://github.com/elastic/integrations/blob/main/packages/haproxy/data_stream/info
  https://github.com/elastic/integrations/blob/main/packages/haproxy/data_stream/stat

• InfluxDB
  https://github.com/elastic/integrations/blob/main/packages/influxdb/data_stream/advstatus
  https://github.com/elastic/integrations/blob/main/packages/influxdb/data_stream/status

• Jolokia Input
  https://github.com/elastic/integrations/blob/main/packages/jolokia_input/agent/

• Kafka
  https://github.com/elastic/integrations/blob/main/packages/kafka/data_stream/broker
  https://github.com/elastic/integrations/blob/main/packages/kafka/data_stream/consumergroup
  https://github.com/elastic/integrations/blob/main/packages/kafka/data_stream/partition

• Memcached
  https://github.com/elastic/integrations/blob/main/packages/memcached/data_stream/stats

• Microsoft SQLServer
  https://github.com/elastic/integrations/blob/main/packages/microsoft_sqlserver/data_stream/performance
  https://github.com/elastic/integrations/blob/main/packages/microsoft_sqlserver/data_stream/transaction_log

• MongoDB
  https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/collstats
  https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/dbstats
  https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/metrics
  https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/replstatus
  https://github.com/elastic/integrations/blob/main/packages/mongodb/data_stream/status

• MYSQL
  https://github.com/elastic/integrations/blob/main/packages/mysql/data_stream/galera_status
  https://github.com/elastic/integrations/blob/main/packages/mysql/data_stream/performance
  https://github.com/elastic/integrations/blob/main/packages/mysql/data_stream/status

• Nginx
  https://github.com/elastic/integrations/blob/main/packages/nginx/data_stream/stubstatus

• Oracle WebLogic
  https://github.com/elastic/integrations/blob/main/packages/oracle_weblogic/data_stream/deployed_application
  https://github.com/elastic/integrations/blob/main/packages/oracle_weblogic/data_stream/threadpool

• PostgreSQL
  https://github.com/elastic/integrations/blob/main/packages/postgresql/data_stream/activity
  https://github.com/elastic/integrations/blob/main/packages/postgresql/data_stream/bgwriter
  https://github.com/elastic/integrations/blob/main/packages/postgresql/data_stream/database
  https://github.com/elastic/integrations/blob/main/packages/postgresql/data_stream/statement

• Prometheus Input
  https://github.com/elastic/integrations/blob/main/packages/prometheus_input/agent/

• RabbitMQ
  https://github.com/elastic/integrations/blob/main/packages/rabbitmq/data_stream/connection
  https://github.com/elastic/integrations/blob/main/packages/rabbitmq/data_stream/exchange
  https://github.com/elastic/integrations/blob/main/packages/rabbitmq/data_stream/node
  https://github.com/elastic/integrations/blob/main/packages/rabbitmq/data_stream/queue

• Redis
  https://github.com/elastic/integrations/blob/main/packages/redis/data_stream/info
  https://github.com/elastic/integrations/blob/main/packages/redis/data_stream/key
  https://github.com/elastic/integrations/blob/main/packages/redis/data_stream/keyspace
  https://github.com/elastic/integrations/blob/main/packages/redis/data_stream/slowlog

• Redis Enterprise
  https://github.com/elastic/integrations/blob/main/packages/redisenterprise/data_stream/node
  https://github.com/elastic/integrations/blob/main/packages/redisenterprise/data_stream/proxy

• SpringBoot
  https://github.com/elastic/integrations/blob/main/packages/spring_boot/data_stream/gc
  https://github.com/elastic/integrations/blob/main/packages/spring_boot/data_stream/memory
  https://github.com/elastic/integrations/blob/main/packages/spring_boot/data_stream/threading

• Traefik
  https://github.com/elastic/integrations/blob/main/packages/traefik/data_stream/health

• VSphere
  https://github.com/elastic/integrations/blob/main/packages/vsphere/data_stream/datastore
  https://github.com/elastic/integrations/blob/main/packages/vsphere/data_stream/host
  https://github.com/elastic/integrations/blob/main/packages/vsphere/data_stream/virtualmachine

• Zookeeper
  https://github.com/elastic/integrations/blob/main/packages/zookeeper/data_stream/connection
  https://github.com/elastic/integrations/blob/main/packages/zookeeper/data_stream/mntr
  https://github.com/elastic/integrations/blob/main/packages/zookeeper/data_stream/server

[ishleenk17]

@Linu-Elias :Lets organize it at package level. I have done that for some in above comment

[ali786XI]

As I was reviewing some of the PRs related to processors raised by third party contributors, I just want to list them so that we can possibly reduce some of them in our ones.

• [Jolokia Input] Add processors option #10068
• Add support for processors on vsphere datastore, host and virtualmachine #10123

[shmsr]

@Linu-Elias Just curious. Are the changes done manually or you used any existing/new automation to do this change?

[Linu-Elias]

@shmsr I wrote a script to first fetch those packages that had missing processors and adding the processors by the same code, but updating changelog and manifest was done manually.

[shmsr]

@shmsr I wrote a script to first fetch those packages that had missing processors and adding the processors by the same code, but updating changelog and manifest was done manually.

Great! If the code is sufficient to perform the task on all packages safely and you believe it would be beneficial for others, you can open a pull request in this repository: https://github.com/elastic/obs-integrations-robots.

If not, that's okay too. However, it's great to have reliable automation tools that are generally useful, like this one: https://github.com/andrewkroh/go-examples/tree/main. We sometimes use tools created primarily by Andrew to execute extensive automation changes.

[ali786XI]

As we have reached the last PR on this should we include the vsphere package changes in the final PR or try to push the PR from the collaborator?

[ishleenk17]

As we have reached the last PR on this should we include the vsphere package changes in the final PR or try to push the PR from the collaborator?

The community PR has been blocked on CLA agreement, so I think it's ok to have vsphere as part of this meta issue.
You can link this issue in community PR.

[muthu-mps]

@ishleenk17 - Do we have plans to fix the processor link pointing to beats in the description as well as part of this issue?

From - https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html
To - https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html