feat(eval): pin injected docker-agent image to host CLI version by default - #4176
Open
aheritier wants to merge 2 commits into
Open
feat(eval): pin injected docker-agent image to host CLI version by default#4176aheritier wants to merge 2 commits into
aheritier wants to merge 2 commits into
Conversation
…fault Previously the eval harness always injected the docker-agent binary from the rolling docker/docker-agent:edge tag, so `docker agent eval` tested against a moving main-HEAD binary regardless of the host CLI's own version. Default the injected image to docker/docker-agent:<version>, matching this CLI's release version, falling back to :edge for dev/main builds where version.Version isn't a valid semver. Add --agent-image to override the default explicitly, including a "none" value that skips binary injection entirely and trusts whatever /docker-agent is already present in the base image. Fixes part 2 of #4175.
RunOutputConfig.AgentImage previously used omitempty on the *resolved* image (ResolvedAgentImage), which is empty precisely when --agent-image none was passed to skip binary injection. That made an explicit opt-out indistinguishable from a run predating this field. Drop omitempty so the field is always present, with an empty string now unambiguously meaning injection was skipped. Also clarifies that binary injection applies to every eval run, not just --base-image ones, and adds save_test.go coverage for all three --agent-image outcomes.
aheritier
added
area/cli
aheritier
marked this pull request as ready for review
September 4, 2026 19:34
aheritier
added
the
kind/feat
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes part 2 of #4175 (full diagnosis: #4175 (comment)).
docker agent evalalways injected the docker-agent binary from the rollingdocker/docker-agent:edgetag, so eval results tested against a movingmain-HEAD binary regardless of the host CLI's own version or a supplied--base-image.Changes
docker/docker-agent:<version>, matching the host CLI's own release version (pkg/version.Version), falling back todocker/docker-agent:edgefor dev/main builds (non-semverversion.Version).--agent-image <ref>todocker agent evalto override the default explicitly.--agent-image noneskips binary injection entirely and trusts whatever/docker-agentis already present in the base image (addresses the issue's optional suggested fix too).config.agent_image), always present (even empty fornone) for reproducibility auditing.docs/features/evaluation/index.mdanddocs/features/cli/index.md.Testing
task build,go test ./pkg/evaluation/... ./cmd/root/...,task lint,go mod tidy --diffall pass.task testhas one pre-existing, unrelated failure inpkg/rag/treesitter(requiresCGO_ENABLED=1/ a C compiler unavailable in this sandbox); untouched by this change.Reviewed by an internal reviewer sub-agent (2 passes); all findings addressed except one non-blocking doc-placement nit (the
--agent-imageinjection note lives under "Custom Base Images" though it applies to every eval run — deliberately left as a very minor, low-risk follow-up rather than restructuring the doc section).