Skip to content

[shim] Fix flaky Docker tests - #4216

Merged
un-def merged 1 commit into
masterfrom
pr_shim_fix_docker_tests
Aug 26, 2026
Merged

[shim] Fix flaky Docker tests#4216
un-def merged 1 commit into
masterfrom
pr_shim_fix_docker_tests

Conversation

@un-def

@un-def un-def commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

TestDocker_SSHServer became flaky after #4203, which stopped Run() from waiting for the container: the tests now wait for the completion themselves, and the wait was capped at 60 seconds, while the container installs openssh-server over the network, which was measured to take from 24 to 52 seconds. #4203 also added three more Docker tests, all running in parallel with each other.

A timed out wait also leaves the container behind, as the deferred Remove() call rejects a task that is still running. The runner dir of such a container is deleted along with the test temporary dir, so the following test runs, which adopt every task container on the host, keep failing to save the state of a task whose dir is gone, reporting errors unrelated to the tests being run. That is, one flaky run poisons the runs that follow it.

  • The wait timeout is raised to 150 seconds, which is still below the timeout of the tests themselves.
  • The deferred cleanup call now terminates the task before removing it, so that a failed test does not leave its container behind.
  • Docker tests no longer run in parallel: they share the Docker daemon, and a DockerRunner adopts every task container on the host, including the containers of the other tests. Two runners processing the same task write over each other's task state files, and one of them may stop a container that the other still considers running.
  • The task info is no longer read inside the require.Eventually() condition, which is called in a goroutine that may outlive a timed out call, racing with the assertions that follow it.

Note that the Docker tests now take about 64 instead of 26 seconds, almost entirely due to TestDocker_SSHServer, which no longer overlaps with the others.

TestDocker_SSHServer became flaky after #4203, which stopped Run() from waiting
for the container: the tests now wait for the completion themselves, and the wait
was capped at 60 seconds, while the container installs openssh-server over the
network, which was measured to take from 24 to 52 seconds. #4203 also added three
more Docker tests, all running in parallel with each other.

A timed out wait also leaves the container behind, as the deferred Remove() call
rejects a task that is still running. The runner dir of such a container is
deleted along with the test temporary dir, so the following test runs, which
adopt every task container on the host, keep failing to save the state of a task
whose dir is gone, reporting errors unrelated to the tests being run. That is,
one flaky run poisons the runs that follow it.

* The wait timeout is raised to 150 seconds, which is still below the timeout of
  the tests themselves.
* The deferred cleanup call now terminates the task before removing it, so that a
  failed test does not leave its container behind.
* Docker tests no longer run in parallel: they share the Docker daemon, and a
  DockerRunner adopts every task container on the host, including the containers
  of the other tests. Two runners processing the same task write over each other's
  task state files, and one of them may stop a container that the other still
  considers running.
* The task info is no longer read inside the require.Eventually() condition, which
  is called in a goroutine that may outlive a timed out call, racing with the
  assertions that follow it.

Note that the Docker tests now take about 64 instead of 26 seconds, almost
entirely due to TestDocker_SSHServer, which no longer overlaps with the others.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@un-def
un-def merged commit 7cc59d5 into master Aug 26, 2026
27 checks passed
@un-def
un-def deleted the pr_shim_fix_docker_tests branch August 26, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant