Skip to content

kuma: preload Docker Hub images in E2E to avoid slow pulls - #24939

Draft
nubtron wants to merge 1 commit into
masterfrom
nubtron/kuma-e2e-preload-images
Draft

kuma: preload Docker Hub images in E2E to avoid slow pulls#24939
nubtron wants to merge 1 commit into
masterfrom
nubtron/kuma-e2e-preload-images

Conversation

@nubtron

@nubtron nubtron commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Speeds up the kuma E2E test on slow/constrained connections by preloading the Docker Hub images the Helm chart pulls, so the kind nodes don't have to fetch them from docker.io (slow / rate-limited) at install time.

Motivation

The kuma E2E test (kuma/tests/conftest.py) installs Kuma via helm upgrade --install kuma kuma/kuma. On a fresh install the chart pulls three images:

  • docker.io/kumahq/kuma-cp:<version> — control-plane deployment (~31 MB)
  • docker.io/kumahq/kumactl:<version> — CRD-install hook job (~42 MB)
  • registry.k8s.io/kubectl:v1.32.8@sha256:... — hook job (~30 MB)

The two kumahq/* images come from Docker Hub, which is the slow / rate-limited step on constrained connections (the chart tarball itself is only ~77 KB, so vendoring the chart would not help). The kubectl image comes from registry.k8s.io, which is fast and reliable, so it is left to pull normally.

The host's Docker image store is persistent across the disposable kind clusters, so docker pull is a cache hit from the second run onward — only the first run pays the download, and every subsequent run is hermetic.

Changes

  • Pull kumahq/kuma-cp and kumahq/kumactl on the host (single platform) before the cluster is created (a kind_run wrapper, mirroring velero/tests/conftest.py).
  • Load them into the kind cluster as a condition that runs before setup_kuma, so they are present in the nodes before helm install.
  • No imagePullPolicy change needed: the chart already defaults to IfNotPresent for the control plane, and the hook jobs use non-:latest tags (which default to IfNotPresent), so the side-loaded copies are used.

Why not datadog_checks.dev.kind.KindLoad?

KindLoad uses kind load docker-image, which pipes docker save into ctr images import --all-platforms inside the node. On Docker daemons using the containerd image store (features.containerd-snapshotter: true, the default on recent Docker / what CI uses), docker save of a multi-arch image emits an OCI index referencing blobs for every platform — including non-host-platform blobs that were never downloaded. kind load docker-image then fails with:

ERROR: failed to load image: command "docker exec ... ctr --namespace=k8s.io images import --all-platforms ..." failed
ctr: content digest sha256:...: not found

velero avoids this only because it docker builds a single-platform image. For upstream multi-arch images we instead use docker save --platform <os/arch> + kind load image-archive, which saves only the host-platform blobs and keeps the original image name so the chart's image references resolve to the side-loaded copies. Verified locally against a containerd-image-store Docker daemon where kind load docker-image reproduces the error above and the archive flow succeeds.

Testing

  • python -m py_compile kuma/tests/conftest.py
  • ddev test kuma -- -m "not e2e" → 27 passed ✅
  • Reproduced the kind load docker-image failure on a containerd-store daemon, then confirmed docker save --platform + kind load image-archive loads both kumahq/kuma-cp:2.10.6 and kumahq/kumactl:2.10.6 into a fresh kind cluster under their original names ✅

Notes

  • The kubectl hook image is intentionally not preloaded: it is digest-pinned (awkward to side-load) and served from the fast registry.k8s.io. If full hermeticity is later desired, it could be added to _kuma_images().
  • Test-only change; no user-facing impact.

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Aug 21, 2026

Copy link
Copy Markdown

Pipelines  Tests  Code Coverage

⚠️ Warnings

🚦 1 Pipeline job failed

Validate repository | Run Validations / Validate

View in Datadog · View in GitHub Actions

QA validation failed: PR #24939 is missing an Agent-release QA decision label. Must set either 'qa/required' or 'qa/skip-qa'.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 37.93%
Overall Coverage: 80.00% (-8.64%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ef84887 | Docs | View more details | Give us feedback!

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

evalya-impact-summary

evalya impact analysis
Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   1 (always emitted)
Diff (1 file):
  kuma/tests/conftest.py

Debug a specific task: evalya plan impact --path <path> --task <task>

Learn more about CI impact filtering

@nubtron
nubtron force-pushed the nubtron/kuma-e2e-preload-images branch from a18136b to ef84887 Compare August 21, 2026 10:10
@dd-octo-sts

dd-octo-sts Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Validation Report

Validation Description Status
qa-label Validate the pull request declares whether it needs QA for the next Agent release

Run ddev validate all changed --fix to attempt to auto-fix supported validations.

Passed validations (20)
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant