Skip to content

fix(ci): fix Splunk integration test flakiness and re-enable #25879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thomasqueirozb merged 3 commits into from
Jul 20, 2026
Merged

fix(ci): fix Splunk integration test flakiness and re-enable #25879

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a2df088
fix(ci): fix Splunk integration test flakiness and re-enable
thomasqueirozb Jul 16, 2026
9fd59cf
fix(ci): add splunk to integration review trigger matrix
thomasqueirozb Jul 20, 2026
56d2f9c
fix(splunk hec sink): pin non-ack token for disabled_on_server ack test
thomasqueirozb Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci-integration-review.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ jobs:
"prometheus",
"pulsar",
"redis",
"splunk",
"webhdfs"
]
steps:
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,6 @@ jobs:
strategy:
fail-fast: false
matrix:
# TODO: Add "splunk" back once https://github.com/vectordotdev/vector/issues/23474 is fixed.
# If you modify this list, please also update the `int_tests` job in changes.yml.
service:
[
"amqp",
Expand All @@ -124,15 +122,16 @@ jobs:
"kafka",
"logstash",
"loki",
"mqtt",
"mongodb",
"mqtt",
"nats",
"nginx",
"opentelemetry",
"postgres",
"prometheus",
"pulsar",
"redis",
"splunk",
"webhdfs"
]
timeout-minutes: 90
Expand Down
2 changes: 1 addition & 1 deletion src/sinks/splunk_hec/common/util.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -495,7 +495,7 @@ pub mod integration_test_helpers {
.send()
},
Duration::from_millis(500),
Duration::from_secs(30),
Duration::from_secs(60),
)
.await;

Expand Down
10 changes: 9 additions & 1 deletion src/sinks/splunk_hec/logs/integration_tests.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ use crate::{
const USERNAME: &str = "admin";
const PASSWORD: &str = "password";
const ACK_TOKEN: &str = "ack-token";
// Matches `SPLUNK_HEC_TOKEN` in tests/integration/splunk/config/compose.yaml, which is not
// configured for indexer acknowledgements (unlike `ACK_TOKEN`).
const DEFAULT_TOKEN: &str = "abcd1234";

async fn recent_entries(index: Option<&str>) -> Vec<JsonValue> {
let client = reqwest::Client::builder()
Expand Down Expand Up @@ -462,7 +465,12 @@ async fn splunk_indexer_acknowledgements() {
async fn splunk_indexer_acknowledgements_disabled_on_server() {
let cx = SinkContext::default();

let config = config(JsonSerializerConfig::default().into(), vec!["asdf".into()]).await;
// `config()` fetches whatever token Splunk's inputs API happens to list first, which can
// be `ACK_TOKEN` and would defeat the point of this test. Force the non-ack token instead.
let config = HecLogsSinkConfig {
default_token: String::from(DEFAULT_TOKEN).into(),
..config(JsonSerializerConfig::default().into(), vec!["asdf".into()]).await
};
let (sink, _) = config.build(cx).await.unwrap();

let (tx, mut rx) = BatchNotifier::new_with_receiver();
Expand Down
8 changes: 7 additions & 1 deletion tests/integration/splunk/config/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,17 @@ services:
- SPLUNK_PASSWORD=password
- SPLUNK_HEC_TOKEN=abcd1234
volumes:
- ../data/splunk/default.yml:/tmp/defaults/default.yml
- ../data/default.yml:/tmp/defaults/default.yml

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep shared tests on the non-ack HEC token

With this corrected mount, data/default.yml is now applied and adds the ack-token input with useACK: true, but the shared log and metric test configs still call get_token() and take the first token returned by Splunk; this patch only pins the non-ack token in one test even though the new comment notes ACK_TOKEN can be returned first. When that happens, ordinary sink tests unexpectedly exercise indexer acknowledgements with the 10s default query interval, making the re-enabled Splunk suite slow or flaky. Pin the common test configs to DEFAULT_TOKEN or select the intended token by name.

Useful? React with 👍 / 👎.

ports:
- 8000:8000
- 8088:8088
- 8089:8089
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8088/services/collector/health"]

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for Splunk health before re-enabling the job

In the inspected CI path, scripts/run-integration-test.sh runs vdev ... start and then immediately vdev ... test, while Compose::start invokes Docker Compose as up --detach without --wait. Docker documents --wait as “Wait for services to be running|healthy”, so this new healthcheck is never used to block test execution; when Splunk is still booting past the helper's 60s token retry, the re-enabled Splunk integration job can still fail against a starting container. Please make the runner wait on health, or poll this endpoint before launching the tests.

Useful? React with 👍 / 👎.

interval: 10s
timeout: 5s
retries: 30
start_period: 60s

networks:
default:
Expand Down
Loading