Support Bzlmod and add rules_scala to bazel-central-registry

[sfc-gh-pbennes]

Greetings!

It looks like rules_scala isn't part of the Bzlmod effort or added to bazel-central-registry yet. I've opened an issue both here and in https://github.com/bazelbuild/bazel-central-registry to request it: bazelbuild/bazel-central-registry#522

[chrislovecnm]

Just a note to anyone that starts the work native.register_toolchains is not supported when using bzlmod.

[sluongng]

@chrislovecnm it's still available here no? https://bazel.build/rules/lib/globals/module#register_toolchains

[chrislovecnm]

@sluongng When I last tried it, it did not work when using it inside an extension. In rules_python we are skipping the native call:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/repositories.bzl#L569

And build the toolchains here:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/extensions/private/pythons_hub.bzl#L56

We build the toolchains and use the following:

https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/MODULE.bazel#L49

[sluongng]

Yeah you would only want to prepare the toolchain (as a new repository) inside the extension I think. Registering it must be called from the MODULE.bazel file.

So native.register_toolchains call from the extension's starlark does not work, but having the extension create @pythons_hub//:all and then MODULE.bazel calling register_toolchains("@pythons_hub//:all") worked. 👍

[mateuszkuta256]

I prepared a minimal bzlmod support here and gonna split it into a few PRs
TODOs after merging it:

• write documentation on how to use rules_scala with bzlmod (some properties like SCALA_VERSION will probably be configurable via repo-env)
• prepare release notes
• contribute repo to the Bazel Central Registry
• add tests that the project builds using both WORKSPACE and MODULE.bazel
• migrate test/external workspaces to blzmod too

[pcj]

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

[mateuszkuta256]

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

hi, unfortunately I'm working on other things now and won't continue with this PR in the foreseeable future
I remember this migration was on hold because interfered with 'cross compilation support'
It looks like great progress was done it this area, so maybe someone can already take over the PR

[mbland]

I'd like to take on the task of Bzlmodifying this repo through a series of pull requests.

I've already created a (mostly) working branch in my own fork. Though I saw a couple draft pull requests here, I ended up taking a different approach and got it mostly working. In fact, I did exactly what @sluongng suggested in #1482 (comment). (I didn't notice this comment before just now—I might've read it, but not understood it at the time—but I did study rules_python and rules_go, and ended up doing exactly that.)

There are still issues I need help to address (recorded in some of the commit messages), I didn't strictly maintain WORKSPACE compatibility (fixable), and I only tested against Scala 2.13. But I can start teasing chunks out of it in a methodical fashion, to ensure that I maintain WORKSPACE and cross-version compatibility.

For an example of what it looks like from a client perspective, here's what the MODULE.bazel stanza from my local EngFlow/example repo looks like (as opposed to the current stanza, explanatory comment notwithstanding):

bazel_dep(name = "rules_scala", repo_name = "io_bazel_rules_scala")
local_path_override(
    module_name = "rules_scala",
    path = "../../bazelbuild/rules_scala"
)   

scala_dev_deps = use_extension(
    "@io_bazel_rules_scala//scala/extensions:deps.bzl",
    "scala_deps",
    dev_dependency = True,
)   
scala_dev_deps.toolchains(
    scalatest = True,
)

So if folks are game for me to do this, I'll start carving off pieces as separate pull requests, and we can resolve any outstanding problems in the process.

cc: @BillyAutrey @jayconrod @benjaminp @TheGrizzlyDev

[mbland]

A quick update for visibility: I'm very close to having the Bzlmodified rules_scala passing 100% of the tests, down to the last couple of failures:

• I need to replace the bind() calls from twitter_scrooge with alias() or some such to get test_version.sh to pass.
• I need to fix the Scala 3.1.2 and Scala 3.3.3 failures for the Scalafmt targets from test/shell/test_cross_build.sh (corresponding to the test3 and library3 test cases in test_scalafmt()).

I also need to get test_lint.sh working again, and update the dt_patches repos for Bzlmod. But the following all pass:

• test_rules_scala.sh
• test_reproducibility.sh
• test_examples.sh
• test_coverage.sh

Hopefully I can get these fixed up today, and I'll start peeling off the next pull request or two. And thanks to @simuons and @liucijus for reviewing and merging #1619 and #1620 already.

[mbland]

Just wanted to take a moment and thank @mbland for the herculean efforts here. Seriously, this is incredible work. Not only is this a massive amount of work, but the regular comms and clear status updates have been a pleasure to keep up with. I remember suffering from a misplaced sense of self-confidence sometime last year and starting my own bzlmod branch to start hacking on this, only to quickly abandon my efforts. I'm sure I couldn't have been the only one. :) Thank you, Mike, for doing what others could not!

@rockpunk I appreciates the appreciation. I'm fortunate to have interest in the problem, my employer's encouragement to help solve it, and the support of the maintainers and everyone here. I'm certain others could've done the work, but I'm grateful to have had the support of my employer that perhaps others didn't.

Of course, the work, ever mysterious and important, isn't quite over yet, which brings me to the next...

Weekly update! (And I'm glad to hear someone other than me gets value out of my weekly brain dumps.)

Overview

#1722 has changed a little bit, as described in the sections below.

All pending change branches are up to date with the latest in #1722, as are the bzlmod and bzlmod-bazel-8 branches.

Open

• Enable Bzlmod #1722

Repo names blog post

As promised, I published Migrating to Bazel Modules (a.k.a. Bzlmod) - Repo Names, Again..., using pull requests from this effort as examples. I also posted an announcement of this post in my blog, which contains a suitable reference that I'd missed in the original post.

Removing unnecessary artifacts

@simuons asked in #1722 (comment) whether there was a better way to reuse protobuf.patch. I'd originally created copies due to paranoia over breaking Windows builds. But of course, after I switched them from copies to symlinks, everything was fine. (Plus, I'm not sure our only Windows CI job uses any of the nested modules anyway.)

While I was at it, I decided that empty WORKSPACE.bzlmod files weren't necessary anymore, given the --noenable_workspace flag in .bazelrc. After removing all of those, #1722 is now only 70 files, of which:

• 13 are symlinks to protoc/0001-protobuf-19679-rm-protoc-dep.patch
• 17 are MODULE.bazel files mirroring existing WORKSPACE files

Updated scala_deps module extension API

@simuons asked in #1722 (comment) to replace the boolean flags in scala_deps.toolchains() with individual tag classes. The thought being, each toolchain can potentially define its own set of attributes via its own tag class.

I've since changed the API from this (example from test_version/MODULE.bazel.template):

scala_deps.toolchains(
    scala_proto = True,
    scala_proto_options = ["grpc"],
    scalatest = True,
    specs2 = True,
)

to this:

scala_deps.scala()
scala_deps.scala_proto(
    options = ["grpc"],
)
scala_deps.scalatest()
scala_deps.specs2()

Note that:

• The instantiation of a tag class, even without arguments, amounts to scala_deps.toolchains(toolchain = True) in the previous API.
• The builtin Scala toolchain now requires scala_deps.scala(), whereas the previous API defaulted to scala_deps.toolchains(scala = True).
• scala_deps.scala_proto(options = [...]) replaces scala_deps.toolchains(scala_proto_options = [...]).

All other tag class definitions remain the same, specifically:

• settings
• overridden_artifact
• compiler_srcjar
• scalafmt
• twitter_scrooge

Fixing test_bzlmod_macros.sh on Linux and Windows

I noticed that the new test/shell/test_bzlmod_macros.sh tests from #1722 were only running on macOS in CI. I learned that one does not have to escape { in Bash regular expressions on macOS, but one must on Linux and Windows (i.e., \{).

The test then ran and failed under the test_rules_scala_jdk21 and test_rules_scala_win CI jobs. The test_rules_scala_jdk21 fix was trivial. I then updated _print_error_message from test/shell/test_helper.sh to fix the mangled error message on Windows.

That left the problem with local_path_override breaking on Windows. test_bzlmod_macros.sh generated test modules containing local_path_override calls with absolute paths. When Bazel tried to create a symlink from the MSYS2 encoded absolute path, the Windows API threw an error for want of a drive letter. The solution was to pass a relative path instead.

For more details, see the commit messages from #1722.

Reproducing and fixing Linux CI issues

In case this is useful to anyone: I develop on macOS. Until now, most of my changes have been compatible with the Linux CI build. But to really reproduce and validate the above breakage, I wanted to use the same Docker image as the CI build.

It's not available on Docker Hub, so here's how I built the image locally:

git clone [email protected]:bazelbuild/continuous-integration.git
cd continuous-integration/buildkite/docker/ubuntu2004
patch -p1 <ubuntu2004.patch
docker build --target ubuntu2004 -t bazelci/ubuntu2004 .

Here's ubuntu2004.patch:

diff --git i/buildkite/docker/ubuntu2004/Dockerfile w/buildkite/docker/ubuntu2004/Dockerfile
index da1f8845..a1a14b7d 100644
--- i/buildkite/docker/ubuntu2004/Dockerfile
+++ w/buildkite/docker/ubuntu2004/Dockerfile
@@ -55,13 +55,6 @@ RUN apt-get -y update && \
     apt-get -y purge apport && \
     rm -rf /var/lib/apt/lists/*
 
-# Workaround https://bugs.launchpad.net/ubuntu/+source/gcc-9/+bug/2029910
-# TODO: remove when the latest ubuntu:20.04 image has the fix.
-WORKDIR /tempdir
-RUN apt-get -y update && apt-get download libgcc-10-dev
-RUN dpkg -x libgcc-10-dev_10.5.0-1ubuntu1~20.04_amd64.deb .
-RUN cp usr/lib/gcc/x86_64-linux-gnu/10/libtsan_preinit.o /usr/lib/gcc/x86_64-linux-gnu/9/
-RUN rm -rf /tempdir
 WORKDIR /
 
 # Allow using sudo inside the container.
@@ -122,6 +115,11 @@ RUN apt-get -y update && \
     apt-get -y install openjdk-21-jdk-headless && \
     rm -rf /var/lib/apt/lists/*
 
+# https://www.docker.com/blog/understanding-the-docker-user-instruction/
+RUN useradd -m -u 501 -g 20 mbland
+USER mbland
+WORKDIR /home/mbland
+
 ENV JAVA_HOME /usr/lib/jvm/java-21-openjdk-${BUILDARCH}
 
 FROM ubuntu2004 AS ubuntu2004-kythe

Then I would run the following script within the rules_scala directory:

#!/usr/bin/env bash

exec docker run -it --rm --init \
  -v ".:/home/$USER/$(basename "$PWD")" \
  bazelci/ubuntu2004 /bin/bash

Within the container, I could then cd rules_scala and do my business.

BTW, if the rules_scala working directory was created with git-new-workdir, the .git directory will contain symlinks to absolute paths. These are valid on the host, but not inside the container, so git commands will then fail with:

mbland@bbc5d99432fc:~/rules_scala-bzlmod$ git ls-files
fatal: not a git repository (or any parent up to mount point /home/mbland)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

mbland@bbc5d99432fc:~/rules_scala-bzlmod$ ls -l .git/config
lrwxr-xr-x 1 mbland dialout 52 Oct  2  2024 .git/config -> /Users/mbland/src/bazelbuild/rules_scala/.git/config

(Hopefully this'll turn up in some poor, confused person's web searches one day.)

Reproducing and fixing Windows CI issues

I have Windows 11 installed on a personal machine with Parallels Desktop, which I used to resolve the Windows breakages via Git for Windows. I'm not sure which MSYS2 environment the CI build uses, but the Git Bash program from Git for Windows seems close enough.

Side quest: Building on Windows 11 ARM64

While I could run test_bzlmod_macros.sh on my Windows ARM64 build, other tests broke.

First, _default_platform() from protoc/private/protoc_toolchains.bzl failed. No protoc/private/protoc_integrity entry contains an exec_compat matching ["@platforms//os:windows", "@platforms//cpu:aarch64"], since no Windows ARM64 protobuf build currently exists. However, Windows ARM can run x86 binaries. So at some point, I may try to commit the following update to _default_platform():

diff --git a/protoc/private/protoc_toolchains.bzl b/protoc/private/protoc_toolchains.bzl
index 389d88c4..efbe8bf6 100644
--- a/protoc/private/protoc_toolchains.bzl
+++ b/protoc/private/protoc_toolchains.bzl
@@ -17,6 +17,11 @@ def _default_platform():
     for platform, info in PROTOC_BUILDS.items():
         if sorted(info["exec_compat"]) == host_platform:
             return platform
+
+    # Temporary measure until native Windows ARM64 builds exist.
+    if host_platform == ["@platforms//cpu:aarch64", "@platforms//os:windows"]:
+        return "win64"
+
     fail(
         "no protoc build found for host platform with constraints: " +
         ", ".join(HOST_CONSTRAINTS),

(Note: The string.join in the last line is a bug fix from minimum-dependency-versions.)

The trickier problem is rules_python:

$ bazel build //src/...

ERROR: C:/users/msb/_bazel_msb/flxik4tq/external/bazel_tools/tools/jdk/BUILD:134:10:
  While resolving toolchains for target @@bazel_tools//tools/jdk:proguard_whitelister (ada0281):
  No matching toolchains found for types:
  @@bazel_tools//tools/python:toolchain_type

Note that @rules_python//python:toolchain_type is an alias to @bazel_tools//tools/python:toolchain_type.

However, Windows ARM64 support may be forthcoming, given recent movement on:

• Support Windows on ARM64 rules_python#2276
• Add aarch64 windows releases astral-sh/python-build-standalone#386
• Add ARM64 Windows builds for Python 3.11+ astral-sh/python-build-standalone#387

Pending branches

These are branches containing changes I'll submit as pull requests after #1722 lands. All of the following are orthogonal to one another and can land in any order:

• bzlmod-bumps: Updates to Bazel 7.6.1, rules_python 1.3.0, Guava 33.4.8-jre, Go 1.24.2, etc.

• test-src: Adds src/... to the bazel build and bazel test commands in test_rules_scala.sh to follow up on Add Java 24 compatibility #1719 and Cleanup WorkerTest after SecurityManager was removed #1721. Joins the contents lines in WorkerTest.testPersistentWorkerArgsfile using the line.separator system property to fix a test failure on Windows.

• bzlmod-workflows: Uses the [Publish to BCR][] reusable GitHub workflow. Updates .github/workflows/release.yml and adds publish-to-bcr.yml for publishing to the [Bazel Central Registry][].

• replace-scalafmt-default-repo: Removes the scalafmt_config() macro and replaces it with the new @rules_scala_toolchains//scalafmt:config target. Fixes a bug discovered while writing test_dependency_versions.sh (added in the test-dependency-versions branch described below).

• minimum-dependency-versions: Separates the latest dependency versions that we test against from the minimum required versions for users. Documents these minimum version requirements in README.md.

• remove-label-wrappers: Removes Label wrappers around target string literals where they're not strictly necessary. See "More repo name insights, plus Label usage" below.

The following branch depends on #1722, replace-scalafmt-default-repo, and minimum-dependency-versions:

• test-dependency-versions: Adds test_dependency_versions.sh to validate the minimum Bazel and dependency versions required by rules_scala, with and without the precompiled protocol compiler toolchain. Extracts setup, run, and teardown helpers from test/shell/test_bzlmod_helpers.sh into test_runner.sh for reuse by test_dependency_versions.sh. Adds a mechanism to skip tests by prefixing their name with _ to run_test_local and run_test_ci. Adds the RULES_SCALA_TEST_REGEX environment variable to test_runner.sh. Adds documentation for RULES_SCALA_TEST_{ONLY,REGEX,VERBOSE} to the header comment of test_runner.sh. Adds ./test_dependency_versions jobs to .bazelci/presubmit.yml.

Preview branches

As always, if you want to try the functionality before #1722 lands, please only use either my bzlmod or bzlmod-bazel-8 branches. I'll keep merging all of the Pending branches into them, so they remain stable even when the pending branches change. Both contain all the changes described above.

[mbland]

D'oh! Didn't mean for #1722 to close this yet. I guess my commit/PR message must've triggered that.

@simuons @liucijus Do you want to reopen this until we publish the module, or open a new issue?

[mbland]

Weekly update!

Overview

#1722 is in, and MODULE.bazel exists on master. Praise Kier!

No, actually: Praise @simuons and @liucijus! 🙂 Not just for landing #1722, but for following up so quickly on all the refinements I've sent after that.

Now we're down to just three pull requests, all orthogonal to one another, and no pending branches! As noted below:

• Add toolchain options API to WORKSPACE and Bzlmod #1730 contains a minor API change affecting both WORKSPACE and Bzlmod.
• Use the Publish to BCR reusable GitHub workflow #1731 adds the Publish to BCR workflow, now that it's ready for prime time.

Once those two land (plus maybe #1729), we're ready to release and publish this thing and close out here.

Merged

• Enable Bzlmod #1722
• Add src/... to test_rules_scala, fix Windows break #1724
• Replace @scalafmt_default with toolchains target #1725
• Establish minimum compatible Bazel, dep versions #1726
• Bump Bazel, rules_*, Guava, Go, golang tools, etc. #1727
• Remove Label wrappers for target string literals #1728

Open

• Add test_dependency_versions, update test_runner #1729
• Add toolchain options API to WORKSPACE and Bzlmod #1730
• Use the Publish to BCR reusable GitHub workflow #1731

Pending branches

None! That's it! No more!

Preview branches

As always, if you want to try everything out before the next release, please only use either my bzlmod or bzlmod-bazel-8 branches. Both contain all the changes from the open pull requests.

Updated scala_toolchains() and scala_deps module extension APIs

I've added what I call "toolchains options support" in #1730. This means that scala_toolchains() accepts either boolean or dict arguments for specific toolchains, and the scala_deps module extension generates them from tag classes. This yields similar changes in both the WORKSPACE and Bzlmod APIs, specifically:

• Add toolchain options API to WORKSPACE and Bzlmod #1730 adds toolchain options support to the scala_toolchains() parameters scalafmt, scala_proto, and twitter_scrooge, and to the scalafmt tag class.

• Add toolchain options API to WORKSPACE and Bzlmod #1730 eliminates the scalafmt_default_config, scala_proto_options, and twitter_scrooge_deps option parameters from scala_toolchains().

• The scala_proto and twitter_scrooge tag class APIs haven't changed, but now they and the scalafmt tag class share a common implementation.

• All these toolchains options dicts are aligned with the underlying toolchain macro arguments, making the relationship between them clear.

• Thanks to the shared implementation, it will be easy to add options support to additional toolchains over time. These updates will require only a minor release, not a major one.

For example, under WORKSPACE, enabling scalafmt and setting a custom config changed like this:

# Before
scala_toolchains(
    # Other toolchains settings...
    scalafmt = True,
    scalafmt_default_config = "//path/to/my/custom:scalafmt.conf",
)

# After
scala_toolchains(
    # Other toolchains settings...
    scalafmt = {"default_config": "//path/to/my/custom:scalafmt.conf"},
)

scalafmt = True will still work if you want to use the default options, but it's implied if you supply your own options object.

Under Bzlmod, the scala_proto tag class changed thus to align the interface with setup_scala_proto_toolchains from //scala_proto:toolchains.bzl:

# Before
scala_deps.scala_proto(
    options = ["grpc"],
)

# After
scala_deps.scala_proto(
    default_gen_opts = ["grpc"],
)

scala_deps.scala_proto() will work in this case, since under the hood, {"default_gen_opts": ["grpc"]} is the default.

Please see the description of #1730 for further details on all the good things.

This came about after @simuons asked in #1722 (comment) to replace the boolean flags in scala_deps.toolchains() with individual tag classes. The thought being, each toolchain can potentially define its own set of attributes via its own tag class. (I mentioned all this in last week's update.)

In the same comment, he also asked whether we could replace toolchain macros (invoked by users in BUILD files) entirely with a module extension. Though I noted this won't be trivial to do, and shouldn't hold up the release, I later experimented with the toolchain macro replacement idea. I'm not yet near achieving that goal, but my experimentation yielded the immediate improvements described above and in #1730.