example/bzlmod: recommend extra_toolchains flag

[sluongng]

In the latest Bazel versions, rules_cc toolchains will take priority
over our own toolchains during automatic resolution.

Recommend users to override the priority by using explicit
--extra_toolchains flag.

[fmeum]

The way toolchains are ordered hasn't changed, so I think this should still only depend on the relative order between the dep on rules_cc (if present) and our module.

Ideally we wouldn't mix this way of registering the toolchain with --extra_toolchains - multiple registrations of the same toolchain are confusing. I see a few options:

1. Document the required order of bazel_deps (not a great experience at this time, but rules_cc needs to come up with something better anyway that we could then adopt)
2. Drop the register_toolchains call from our module file and ask users to register it themselves via register_toolchains (if they always want it) or --extra_toolchains if the extra flexibility is actually needed. A breaking change of sorts, but once migrated, users will have explicit control over the order.

[sluongng]

I think we can do (2), except still keep the register_toolchains in our module.
Could you give the PR another read? 🤔

[dan-stowell]

I was able to get a slimmed-down version of this example to build with RBE from Darwin with the following:

bazel build //:main_cc \
    --bes_results_url=https://app.buildbuddy.io/invocation/ \
    --bes_backend=grpcs://remote.buildbuddy.io \
    --remote_executor=grpcs://remote.buildbuddy.io \
    --platforms=@toolchains_buildbuddy//platforms:linux_x86_64 \
    --extra_execution_platforms=@toolchains_buildbuddy//platforms:linux_x86_64 \
    --experimental_platform_in_output_dir

I do call register_toolchains(...) in the MODULE.bazel of that example.

[fmeum]

This is still misleading: The individual toolchains we register don't compete with each other since their target constraints differ, so users never have to order them explicitly. Instead, they could just drop some of the arguments to this call.

The case in which users need --extra_toolchains is if they only want our toolchain(s) to win sometimes. I guess we don't have to tell them about --extra_toolchains in that case since there is nothing special about this case (no ordering issues, nothing specific to Bzlmod or BuildBuddy).

We could still call out the flag as an alternative to the call, but not for the purpose mentioned in the comment.