Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[9.0](backport #6730) Add conditions to copy_fields processor #7331

Merged
merged 1 commit into from
Mar 12, 2025

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Mar 11, 2025

What does this PR do?

This commit adds conditions to the copy_fields processor from the monitoring Filebeat to prevent it from failing and spamming the event logger at debug level with:
target field xxx already exists, drop or rename this field first

Why is it important?

It makes the debug logs more useful by remove unnecessary entries from our monitoring Filebeat

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

## Disruptive User Impact

How to test this PR locally

  1. Create a log file with more than 1kb: docker run -it --rm mingrammer/flog -n 20 > /tmp/flog.log

  2. Package the Elastic-Agent from this PR

  3. Start the Elastic-Agent with the following configuration

    elastic-agent.yml

    outputs:
      default:
        type: elasticsearch
        hosts:
          - https://localhost:9200
        protocol: https
        username: elastic
        password: changeme
        preset: latency
        ssl.verification_mode: none
    
    inputs:
      - type: filestream
        id: your-input-id
        log_level: debug
        streams:
          - id: your-filestream-stream-id
            data_stream:
              dataset: generic
            paths:
              - /tmp/flog.log
    
    agent.monitoring:
      enabled: true
      logs: true
      metrics: false
      pprof.enabled: false
      use_output: default
    
    # Needed if you already have an Elastic-Agent running on your machine
    # That's very helpful for running the tests locally
    agent.monitoring:
      http:
        enabled: false
        port: 7002
    
    agent.grpc:
      address: localhost
      port: 7001
    
    

  4. Ensure the event logs (data/elastic-agent*/logs/events/*.ndjson) do not contain any messages from the copy_fields processor. The following command must return 0:

    cat data/elastic-agent*/logs/events/*.ndjson| grep copy_fields |wc -l
    

Related issues

Questions to ask yourself

  • How are we going to support this in production?
  • How are we going to measure its adoption?
  • How are we going to debug this?
  • What are the metrics I should take care of?
  • ...

This is an automatic backport of pull request #6730 done by [Mergify](https://mergify.com).

This commit adds conditions to the `copy_fields` processor from the
monitoring Filebeat to prevent it from failing and spamming the event
logger at debug level with:
`target field xxx already exists, drop or rename this field first`

---------

Co-authored-by: Pierre HILBERT <[email protected]>
(cherry picked from commit acf1098)
@mergify mergify bot added the backport label Mar 11, 2025
@mergify mergify bot requested a review from a team as a code owner March 11, 2025 21:59
@mergify mergify bot requested review from pkoutsovasilis and kaanyalti and removed request for a team March 11, 2025 21:59
@github-actions github-actions bot added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Mar 11, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

Copy link

Quality Gate failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube

@belimawr belimawr merged commit 4733843 into 9.0 Mar 12, 2025
14 of 15 checks passed
@belimawr belimawr deleted the mergify/bp/9.0/pr-6730 branch March 12, 2025 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants