[8.17](backport #6730) Add conditions to copy_fields processor

[mergify]

What does this PR do?

This commit adds conditions to the copy_fields processor from the monitoring Filebeat to prevent it from failing and spamming the event logger at debug level with:
target field xxx already exists, drop or rename this field first

Why is it important?

It makes the debug logs more useful by remove unnecessary entries from our monitoring Filebeat

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

## Disruptive User Impact

How to test this PR locally

1. Create a log file with more than 1kb: docker run -it --rm mingrammer/flog -n 20 > /tmp/flog.log

2. Package the Elastic-Agent from this PR

3. Start the Elastic-Agent with the following configuration

   elastic-agent.yml

   outputs:
     default:
       type: elasticsearch
       hosts:
         - https://localhost:9200
       protocol: https
       username: elastic
       password: changeme
       preset: latency
       ssl.verification_mode: none
   
   inputs:
     - type: filestream
       id: your-input-id
       log_level: debug
       streams:
         - id: your-filestream-stream-id
           data_stream:
             dataset: generic
           paths:
             - /tmp/flog.log
   
   agent.monitoring:
     enabled: true
     logs: true
     metrics: false
     pprof.enabled: false
     use_output: default
   
   # Needed if you already have an Elastic-Agent running on your machine
   # That's very helpful for running the tests locally
   agent.monitoring:
     http:
       enabled: false
       port: 7002
   
   agent.grpc:
     address: localhost
     port: 7001
   
   

4. Ensure the event logs (data/elastic-agent*/logs/events/*.ndjson) do not contain any messages from the copy_fields processor. The following command must return 0:

   cat data/elastic-agent*/logs/events/*.ndjson| grep copy_fields |wc -l
   

Related issues

• Closes Event log file gets flooded with copy_fields processor error #5299

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

---

This is an automatic backport of pull request #6730 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of acf1098 has failed:

On branch mergify/bp/8.17/pr-6730
Your branch is up to date with 'origin/8.17'.

You are currently cherry-picking commit acf109882.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1738855375-Add-conditions-to-copy_fields-processors.yaml
	modified:   internal/pkg/agent/application/monitoring/v1_monitor.go
	modified:   testing/integration/container_cmd_test.go
	modified:   testing/integration/event_logging_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   testing/integration/logs_ingestion_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: a5ea844

Failed CI Steps

• Unit tests - MacOS 13
• Merge coverage reports
• :sonarqube: Continuous Code Inspection
• Win2025:sudo:fleet

History

• 💔 Build #18854 failed 8c617a3
• 💔 Build #18558 failed 4a192da

cc @belimawr