Adjust CI to enable cache sharing with forked PRs #2353
Conversation
by leting codecov script pickup on circleci env on it own to corroborate identity
|
So to be clear, the github action is just for triggering dockerhub to re-build the base image? Maybe I'm missing some small detail, but if this is just on a daily cron job, doesn't dockerhub have that option itself? I see an autobuild option there. |
Well, Dockerhub can trigger image rebuilds when autobuild is enabled, but it's rather limited. First, it can only trigger on push events to github branches. While the GH action replicates this, the action is a lot smarter or selective in that it only triggers a rebuild if certain files are changed. E.g why rebuild and break all the existing CI caches if the only thing a merge commit changed is the README file? The second thing that Docker Hub can't do on it's own is scheduled builds, e.g. time based triggers irregardless of commit events. This is the same logic we had before in CircleCI, and checks if any ros2 package are out of date. Ideally we could use linked repos feature in Dockerhub to trigger image rebuilds when the ros:rolling parent image updates itself, but linking to official library repos has since been disabled by Dockerhub because of all churn that causes when library images do update. Checking apt directly is also a little better and that we can detect when to rebuild the image even if the packages to update were not originally installed from the parent image. |
|
OK got it, thanks for the detailed explanation |
* Use tokenless codecov upload by leting codecov script pickup on circleci env on it own to corroborate identity * Remove dockerhub cron job from circleci * Add dockerhub cron job from gh actions
So for a recent PR #2349 after merging #2343 , I noticed that it's cache restore step was not finding the matching cache key provide by the main branch:
Prior existing cache stored by main branch CI job
Later failed cache restore from forked PR CI job
This was because CircleCI does not share caches with forked repos, unless sharing of environment variables is enabled:
We can simply enable sharing of environment variables in the repo's circleci settings, but we currently have two secret tokens stored in the repo's circleci environment. The codecov token and the DockerHub build trigger token. Turns out codecov token should be unnecessary for public repos when using major CI's like CircleCI or GH actions, so we could delete that variable from the current CircleCI environment.
As for the DockerHub build trigger token, given GH actions has a slightly more nuance control for protecting repo secrets:
I've ported the cron job config from #2344 into a GH action workflow so we can move the DockerHub secret into GH actions. This also has the added benefit of allowing the use of GH action event triggers to rebuild the CI image if any of the
package.xmlor.reposfiles are changed, given they're subsequently used to pre-install depencies in the CI image.