Skip to content

Kubernetes.audit_logs: add support for cloud providers #14554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Kubernetes.audit_logs: add support for cloud providers #14554

wants to merge 3 commits into from
+1,038 −12

Conversation

chemamartinez
Copy link
Contributor

Proposed commit message

Extend the Kubernetes audit_logs data stream to support collecting audit logs from managed Kubernetes clusters in major cloud providers:

  • AWS EKS via CloudWatch Logs
  • Azure AKS via Event Hub
  • Google GKE via Pub/Sub

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Screenshots

Add-integration-Kubernetes-Integrations-Elastic-07-15-2025_06_49_PM Edit-integration-Elastic-Agent-elastic-package-Agent-policies-Fleet-Elastic-07-15-2025_06_52_PM Screenshot 2025-07-15 at 18 52 48 Screenshot 2025-07-15 at 18 53 07

@chemamartinez chemamartinez self-assigned this Jul 15, 2025
@chemamartinez chemamartinez added enhancement New feature or request Integration:kubernetes Kubernetes Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Jul 15, 2025
@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 15, 2025
@chemamartinez chemamartinez force-pushed the 5799-kubernetes-audit-cloud branch from c66a453 to c77254f Compare July 16, 2025 05:37
@chemamartinez chemamartinez marked this pull request as ready for review July 16, 2025 05:37
@chemamartinez chemamartinez requested a review from a team as a code owner July 16, 2025 05:37
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@elastic-vault-github-plugin-prod
Copy link

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@chemamartinez chemamartinez requested review from a team July 16, 2025 10:14
gizas
gizas reviewed Jul 21, 2025
View reviewed changes
packages/kubernetes/_dev/build/docs/audit-logs.md Outdated
@@ -1,8 +1,16 @@
# audit-logs

audit-logs integration collects and parses Kubernetes audit logs.
Audit logs integration collects and parses Kubernetes audit logs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Audit logs integration collects and parses Kubernetes audit logs.
Audit-logs integration collects and parses Kubernetes audit logs.

gizas
gizas reviewed Jul 21, 2025
View reviewed changes
packages/kubernetes/data_stream/audit_logs/agent/stream/aws-cloudwatch.yml.hbs
@@ -0,0 +1,103 @@
{{#unless log_group_name}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if those files should be placed under relevant aws-cloudwatch integration (and similarly under azure and gcp). I am thinking that it would be difficult for our users to figure out that we have audit logs support and to check on k8s when initially are on a CSP integration.

@zmoog wdyt on this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some discussions (see #5799 (comment) and further comments), that option was on the table but it shows some inconveniences in terms of maintenance mainly).

Happy to hear your thoughts on this point.

gizas
gizas reviewed Jul 21, 2025
View reviewed changes
packages/kubernetes/data_stream/audit_logs/elasticsearch/ingest_pipeline/default.yml Outdated
- append:
field: error.message
value: '{{{ _ingest.on_failure_message }}}'
value: >
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use this example: https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/cloudwatch_logs/elasticsearch/ingest_pipeline/default.yml#L25

gizas
gizas reviewed Jul 21, 2025
View reviewed changes
packages/kubernetes/data_stream/audit_logs/elasticsearch/ingest_pipeline/default.yml
@@ -1,6 +1,19 @@
---
description: Pipeline for processing Kubernetes audit logs.
processors:
- rename:
field: message
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should do it with set: https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/cloudwatch_logs/elasticsearch/ingest_pipeline/default.yml#L9

In order not to loose message field right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the default criteria for this in Obs integrations? We usually keep the original raw message in event.original and message gets removed so it is not duplicated.

gizas
gizas reviewed Jul 21, 2025
View reviewed changes
packages/kubernetes/data_stream/audit_logs/manifest.yml
@@ -6,10 +6,322 @@ elasticsearch:
mappings:
dynamic: false
streams:
- input: aws-cloudwatch
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar as above comment https://github.com/elastic/integrations/pull/14554/files#r2218628097

gizas
gizas reviewed Jul 21, 2025
View reviewed changes
packages/kubernetes/docs/audit-logs.md Outdated
@@ -1,8 +1,16 @@
# audit-logs

audit-logs integration collects and parses Kubernetes audit logs.
Audit logs integration collects and parses Kubernetes audit logs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Audit logs integration collects and parses Kubernetes audit logs.
Audit-logs integration collects and parses Kubernetes audit logs.

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate failed Quality Gate failed

Failed conditions
19.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @chemamartinez

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gizas gizas gizas left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@chemamartinez chemamartinez

Labels
documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:kubernetes Kubernetes Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Kubernetes Audit Logs] [AWS Cloudwatch] Create a new datastream for Ingesting EKS Audit Logs
4 participants
@chemamartinez @elasticmachine @gizas @andrewkroh