KAFKA-18500 Build PRs at HEAD commit

[mumrah]

The default checkout behavior for GitHub Actions is to use a special
merge ref which is equivalent to the base branch with the PR merged into
it. While this is crucial for checking compilation issues against trunk,
it significantly diminishes our ability to use any build caching.

This patch changes the JUnit test jobs to checkout the HEAD commit of the PR
when building. The "Compile and Check" step still checks out the merge commit
so we can keep that level of validation.

Reviewers: Ismael Juma [email protected], Chia-Ping Tsai [email protected]

[mumrah]

Ok, there is a fundamental problem here. The pull_request target is building the merge commit of this PR against the base rather than just the PR contents. This means, the build will include changes on trunk which have not yet been cached.

When trunk is moving quickly, our PRs will have little hope to benefit from much caching.

For example:

(trunk) HEAD --- A --- B --- C
(PR) HEAD --- X --- Y --- Z --- C

If commit C was the last trunk commit to be built, there will be Gradle cache files for that commit. Commits A and B are still building. If the PR was simply building X, this would be fine and we would expect cache hits for anything not changed by X, Y, Z. However, the pull_request event will result in a build of something totally different:

(merge) HEAD --- A --- B --- C
            `X --- Y --- Z --- C

So when the PR is built, it will be fetching the latest cache (C), but will include file changes from A and B in addition to the PR changes. This greatly increases cache misses.

---

I think the merge queue might be a solution to this. If we do a full build as part of the merge queue, then no code will land on trunk that has not been built, tested, and cached. The risk with this approach is that flaky builds will prevent things from getting into trunk.

@ijuma @dajac thoughts?

[ijuma]

One way to workaround the flaky test issue would be to increase the number of retries for failed tests when running the build via the merge queue.

That said, we'd want to measure how much time we're taking with this extra time versus time saved in PRs themselves.

[mumrah]

Ok, I was able to modify the build to check out the PR at HEAD instead of the merge commit. This increase our cache hits a lot.

The recent job finished in around 1 hour because it only had to run :core:test and :streams:test (since those had failures in the trunk job, and so were not cached)

[mumrah]

I validated the Build Scan status checks on my fork

That shows all four build scans being reported to the commit (which will appear in PRs)

[mumrah]

@ijuma WDYT about this approach?

[ijuma]

I haven't reviewed the code changes - the approach looks good.

[chia7712]

I was able to modify the build to check out the PR at HEAD instead of the merge commit. This increase our cache hits a lot.

If we run tests against HEAD, can CI check for new tests or compilation issues in the PR? Alternatively, to maximize cache usage, could we automatically run the PR against the latest cached commit as a base?

[ijuma]

@chia7712 compilation isn't affected by this PR, just the actual test execution.

The "Compile and Check" step still checks out the merge commit so we can keep that level of validation.

[chia7712]

compilation isn't affected by this PR, just the actual test execution.

thanks - I overlooked the description :(

[mumrah]

Alternatively, to maximize cache usage, could we automatically run the PR against the latest cached commit as a base?

This would require doing something similar to the merge ref, but for the latest cache SHA (instead of trunk HEAD). I think it's probably achievable, but maybe a bit tricky to do correctly. I imagine there are probably cases where the PR can't be merged into the cache SHA. For example, maybe the PR has already merged in trunk beyond the cached SHA and made some additional commits.

It still might be worth exploring though.

For now, I think this approach is easy to understand ("build the code in the PR branch"), should give us some caching boost, and hopefully causes few surprises.

[chia7712]

LGTM

[chia7712]

I noticed that this script requires us to login Github (or access key) - is it possible to use curl + jq to parse https://api.github.com/repos/apache/kafka/actions/caches?key=gradle-home-v1&&ref=refs/heads/trunk?

[mumrah]

Wow, I'm surprised that works! I would have assumed that even public APIs would need some kind of auth token.

I filed https://issues.apache.org/jira/browse/KAFKA-18903 for this