[CI]: Investigate/fix cases of CI tests not running on latest code from PR branch

[njhill]

We've seen a couple of recent cases where PR changes fail on main after being merged, but the same test that ran on the PR's CI prior to merge passed. It appears that the code which the passing test ran on wasn't the latest code from the PR branch.

This is separate to the issue of test scoping. In these cases the test in question did run and passed.

It also shouldn't be a consequence of the use of USE_PRECOMPILED_WHEELS because the code differences in question were just changes to python files.

Examples:

• Tensorizer test: [Model loader]: support multi-thread model weight loading #23928 (comment)
• Basic models test: [BugFix][Spec Decode] Fix out-of-range index triggered by eagle3; re-enable test for LlamaForCausalLMEagle3 #24392

Slack threads: here, here.

This is obviously very high priority since it's causing frequent unnecessary breakages to main.

cc @dougbtv

[wwl2755]

After some investigation I found the test pipeline in those examples are different than the normal ones.

The CI didn't catch latest code:

The CI did catch the latest code:

It is because that the docker image build was skipped in those cases. So the root cause, I'm wondering, could be related to the logic of in what condition this "SKIP_IMAGE_BUILD" is set and how it operates. cc: @dougbtv

[njhill]

Thanks @wwl2755 this is a useful observation I think!

[dougbtv]

Big thanks Wenlong

That SKIP_IMAGE_BUILD is from a reverted PR in ci-infra: https://buildkite.com/vllm/ci/builds/29873#01992bd1-e3b8-4e6d-ab26-8dcb4b7ea2d1/131-135

vllm-project/ci-infra@409e601
vllm-project/ci-infra@f85bf9c

Nick noted that we need to mark the code as editable during pip install / running setup.py, I will follow up on this.

The revert happened at...

Date:   Mon Sep 8 18:41:47 2025 -0700

The error occurred at...

2025-09-09T00:14:20Z

Which is about 90 minutes before the revert.

I will use this as input for working on: vllm-project/ci-infra#161

[wwl2755]

Glad it helped. Thanks for the nice work on reducing the CI time!