Skip to content

fix(ci): fix Splunk integration test flakiness and re-enable#25879

Merged
thomasqueirozb merged 3 commits into
masterfrom
fix/splunk-integration-tests
Jul 20, 2026
Merged

fix(ci): fix Splunk integration test flakiness and re-enable#25879
thomasqueirozb merged 3 commits into
masterfrom
fix/splunk-integration-tests

Conversation

@thomasqueirozb

@thomasqueirozb thomasqueirozb commented Jul 16, 2026

Copy link
Copy Markdown
Member

Summary

Fixes #23474 - Splunk integration tests were timing out due to two issues:

  1. Incorrect volume path in compose.yaml (introduced during integration test reorganization)
  2. Missing health check causing tests to start before Splunk was ready

Vector configuration

NA

How did you test this PR?

  • Verified the fix by running make check-clippy
  • The changes address root causes identified from the issue

/ci-run-integration-splunk
https://github.com/vectordotdev/vector/actions/runs/29761543841/job/88417689710?pr=25879

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

@github-actions github-actions Bot added domain: sinks Anything related to the Vector's sinks domain: ci Anything related to Vector's CI environment labels Jul 16, 2026
@thomasqueirozb thomasqueirozb added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Jul 16, 2026

@thomasqueirozb thomasqueirozb left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ci-run-integration-splunk

@datadog-vectordotdev

datadog-vectordotdev Bot commented Jul 16, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 56d2f9c | Docs | Give us feedback!

@thomasqueirozb thomasqueirozb left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ci-run-integration-splunk

@thomasqueirozb thomasqueirozb left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ci-run-integration-splunk

@thomasqueirozb
thomasqueirozb marked this pull request as ready for review July 20, 2026 17:28
@thomasqueirozb
thomasqueirozb requested a review from a team as a code owner July 20, 2026 17:28

@pront pront left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix 👍

@thomasqueirozb
thomasqueirozb added this pull request to the merge queue Jul 20, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56d2f9c4f7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

- 8088:8088
- 8089:8089
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8088/services/collector/health"]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for Splunk health before re-enabling the job

In the inspected CI path, scripts/run-integration-test.sh runs vdev ... start and then immediately vdev ... test, while Compose::start invokes Docker Compose as up --detach without --wait. Docker documents --wait as “Wait for services to be running|healthy”, so this new healthcheck is never used to block test execution; when Splunk is still booting past the helper's 60s token retry, the re-enabled Splunk integration job can still fail against a starting container. Please make the runner wait on health, or poll this endpoint before launching the tests.

Useful? React with 👍 / 👎.

- SPLUNK_HEC_TOKEN=abcd1234
volumes:
- ../data/splunk/default.yml:/tmp/defaults/default.yml
- ../data/default.yml:/tmp/defaults/default.yml

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep shared tests on the non-ack HEC token

With this corrected mount, data/default.yml is now applied and adds the ack-token input with useACK: true, but the shared log and metric test configs still call get_token() and take the first token returned by Splunk; this patch only pins the non-ack token in one test even though the new comment notes ACK_TOKEN can be returned first. When that happens, ordinary sink tests unexpectedly exercise indexer acknowledgements with the 10s default query interval, making the re-enabled Splunk suite slow or flaky. Pin the common test configs to DEFAULT_TOKEN or select the intended token by name.

Useful? React with 👍 / 👎.

Merged via the queue into master with commit a04ab31 Jul 20, 2026
168 checks passed
@thomasqueirozb
thomasqueirozb deleted the fix/splunk-integration-tests branch July 20, 2026 18:26
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

domain: ci Anything related to Vector's CI environment domain: sinks Anything related to the Vector's sinks no-changelog Changes in this PR do not need user-facing explanations in the release changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Splunk integration tests are failing

2 participants