Rewrite CI to only test languages that have actually changed

[mgeisler]

We currently build and test all translations — resulting in some 32 GitHub workflows being executed per PR. This is often unnecessary when only the original English text changes.

I'm considering if it would be worth it to use a big build system such as Bazel. This would allow us to get very fine-grained dependency tracking and this would allow us to skip unnecessary steps. However, I believe it will be significantly more complex to maintain the build system files that way.

Thoughts or ideas?

[djmitche]

Is the issue saving the CPU time to run those extra jobs, or the visibility of the jobs in the PR?

One issue with moving this into a build system is, then everything runs in one GH check that runs bzl test or whatever, so you lose some resolution in glancing at a PR to see what has succeeded or failed.

If the issue is CPU time, I suspect it's the checkout/update that takes most of the time, with the actual mdbook invocation taking only a few hundred ms. In that case, it might make sense to have a single "test all translations" job that tests all of the translations in a single go. If that succeeds, great, job done. If it fails, it can use workflow_run to run per-translation workflows -- maybe even only the failing translations?

[mgeisler]

Is the issue saving the CPU time to run those extra jobs, or the visibility of the jobs in the PR?

Honestly, it's mostly the feeling of wasting resources but also the feeling that we could spend less wall clock time if we had fewer jobs with better caching.

One issue with moving this into a build system is, then everything runs in one GH check that runs bzl test or whatever, so you lose some resolution in glancing at a PR to see what has succeeded or failed.

Yes, that is an important tradeoff. The workers start quite quickly, it normally only takes ~5 seconds for them to start. So even if we had a more unified setup, we could still decide to test individual languages in their own job: most jobs would finish in a few seconds then.

If the issue is CPU time, I suspect it's the checkout/update that takes most of the time, with the actual mdbook invocation taking only a few hundred ms.

It is mdbook test part that ends up taking time, around 1 minute per language. Way back, we had problems with some translations accidentally using fancy curly quotes “ instead of " in the code blocks — which of course didn't work so well 🙂 That is why I originally introduced the mdbook test run for each translation.

• Pros of a unified build system:
  • I think it would be very nice to run a single bazel test ... command locally to test everything.
  • Better caching would give us faster builds.
• Cons of a unified build system:
  • The tooling would be foreign to most Rust developers.
  • More complex setup which someone (likely me) would need to maintain.
  • Bazel support for Rust is still new, we could run into corner-cases which we don't know how to fix.

Having written this, I realize that the "pros" are somewhat weak: it's not like I constantly find myself scratching my head wondering how I can run the relevant test or build commands when I do a change 😄 Faster builds is also relative: because we fan out to 18 jobs for the translations, it only takes 3-4 minutes in total to run everything on GitHub.