Add toolchain options API to WORKSPACE and Bzlmod

Updates `scala_toolchains()` to accept either boolean or dict arguments
for specific toolchains, and updates `//scala/extensions:deps.bzl` to
generate them from tag classes. Part of bazel-contrib#1482.

Notable qualities:

- Adds toolchain options support to the `scala_toolchains()` parameters
  `scalafmt`, `scala_proto`, and `twitter_scrooge`, and to the
  `scalafmt` tag class.

- Eliminates the `scalafmt_default_config`, `scala_proto_options`, and
  `twitter_scrooge_deps` option parameters from `scala_toolchains()`.

- Provides uniform, strict evaluation and validation of toolchain
  options passed to `scala_toolchains()`.

- Configures enabled toolchains using root module settings or the
  default toolchain settings only.

- Introduces the shared `TOOLCHAIN_DEFAULTS` dict in
  `//scala/private:toolchains_defaults.bzl` to aggregate individual
  `TOOLCHAIN_DEFAULTS` macro parameter dicts.

This change also:

- Replaces the non-dev dependency `scala_deps.scala()` tag instantiation
  in `MODULE.bazel` with `dev_deps.scala()`.

- Renames the `options` parameter of the `scala_deps.scala_proto` tag
  class to `default_gen_opts` to match `setup_scala_proto_toolchain()`.

- Introduces `_stringify_args()` to easily pass all toolchain macro args
  compiled from `scala_toolchains_repo()` attributes through to the
  generated `BUILD` file.

- Extracts `_DEFAULT_TOOLCHAINS_REPO_NAME` and removes the
  `scala_toolchains_repo()` macro.

- Includes docstrings for the new private implementation functions, and
  updates all other docstrings, `README.md`, and other relevant
  documentation accordingly.

---

Inspired by @simuons's suggestion to replace toolchain macros with a
module extension in:

- bazel-contrib#1722 (comment)

Though a full replacement is a ways off, this is a step in that
direction that surfaced several immediate improvements.

First, extensibility and maintainability:

- The new implementation enables adding options support for other
  toolchains in the future while maintaining backward compatibility, for
  both the `WORKSPACE` and Bzlmod APIs. Adding this support will only
  require a minor release, not a major one.

- The `scala_deps` module extension implementation is easier to read,
  since all toolchains now share the `_toolchain_settings` mechanism.

Next, improved consistency of the API and implementation:

- Toolchain options parameters should present all the same parameters as
  the macros to which they correspond, ensured by the
  `TOOLCHAIN_DEFAULTS` mechanism. This is to make it easier for users
  and maintainers to see the direct relationship between these separate
  sets of parameters. (They can also define additional parameters to
  those required by the macro, like `default_config` from the `scalafmt`
  options.)

  This principle drove the renaming of the `scala_deps.scala_proto` tag
  class parameter from `options` to `default_gen_opts`. It also inspired
  updating `scala_toolchains_repo()` to pass toolchain options through
  `_stringify_args()` to generate `BUILD` macro arguments.

- The consolidated `TOOLCHAIN_DEFAULTS` dict reduces duplication between
  the `scala/extensions/deps.bzl` and `scala/toolchains.bzl` files. It
  ensures consistency between tag class `attr` default values for Bzlmod
  users and the `scala_toolchains()` default parameter values for
  `WORKSPACE` users.

  The `TOOLCHAINS_DEFAULTS` dicts corresponding to each toolchain macro
  do duplicate the information in the macro argument lists. However, the
  duplicated values in this case are physically adjacent to one another,
  minimizing the risk of drift.

- Extracting `_DEFAULT_TOOLCHAINS_REPO_NAME` is a step towards enabling
  customized repositories based on the builtin toolchains, while
  specifying different options. This extraction revealed the fact that
  the `scala_toolchains_repo()` macro was no longer necessary, since
  only `scala_toolchains()` ever called it.

Finally, fixes for the following design bugs:

- Previously, `scala_deps.scala_proto(options = [...])` compiled the
  list of `default_gen_opts` from all tag instances in the module graph.
  This might've been convenient, but didn't generalize to other options
  for other toolchains. In particular, it differed from the previous
  `toolchains`, `scalafmt`, and `twitter_scrooge` tag class behavior.

  The new semantics are unambiguous, consistent, and apply to all
  toolchains equally; they do not show a preference for any one
  toolchain over the others. They also maintain the existing `scalafmt`
  and `twitter_scrooge` tag class semantics, but now using a generic
  mechanism, simplifying the implementation.

- Instantating `scala_deps.scala()` was a bug left over from the
  decision in bazel-contrib#1722 _not_ to enable the builtin Scala toolchain by
  default under Bzlmod.

  The previous `scala_deps.toolchains()` tag class had a default `scala
  = True` parameter. The user could set `scala = False` to disable the
  builtin Scala toolchain. After replacing `toolchains()` with
  individual tag classes, the documented behavior was that the user must
  enable the builtin Scala toolchain by instantiating
  `scala_deps.scala()`.

  By instantiating `scala_deps.scala()` in our own `MODULE.bazel` file,
  we ensured that `rules_scala` would always instantiate the builtin Scala
  toolchain. While relatively harmless, it violated the intention of
  allowing the user to avoid instantiating the toolchain altogether.

Author: mbland

MODULE.bazel

@@ -77,7 +77,6 @@ use_repo(
     "rules_scala_toolchains",
     "scala_compiler_sources",
 )
-scala_deps.scala()
 
 # Register some of our testing toolchains first when building our repo.
 register_toolchains(
@@ -101,6 +100,7 @@ dev_deps = use_extension(
     "scala_deps",
     dev_dependency = True,
 )
+dev_deps.scala()
 dev_deps.jmh()
 dev_deps.junit()
 dev_deps.scala_proto()

docs/phase_scalafmt.md

@@ -86,8 +86,7 @@ a custom configuration:
     # WORKSPACE
     scala_toolchains(
         # Other toolchains settings...
-        scalafmt = True,
-        scalafmt_default_config = "//path/to/my/custom:scalafmt.conf",
+        scalafmt = {"default_config": "//path/to/my/custom:scalafmt.conf"},
     )
     ```
 

scala/extensions/deps.bzl

@@ -14,8 +14,8 @@ Provides the `scala_deps` module extension with the following tag classes:
 - `twitter_scrooge`
 - `jmh`
 
-For documentation, see the `_tag_classes` dict, and the `_<TAG>_attrs` dict
-corresponding to each `<TAG>` listed above.
+For documentation, see the `_{general,toolchain}_tag_classes` dicts and the
+`_<TAG>_attrs` dict corresponding to each `<TAG>` listed above.
 
 See the `scala/private/macros/bzlmod.bzl` docstring for a description of
 the defaults, attrs, and tag class dictionaries pattern employed here.
@@ -27,6 +27,7 @@ load(
     "root_module_tags",
     "single_tag_values",
 )
+load("//scala/private:toolchain_defaults.bzl", "TOOLCHAIN_DEFAULTS")
 load("//scala:scala_cross_version.bzl", "default_maven_server_urls")
 load("//scala:toolchains.bzl", "scala_toolchains")
 
@@ -89,9 +90,7 @@ _compiler_srcjar_attrs = {
     "integrity": attr.string(),
 }
 
-_scalafmt_defaults = {
-    "default_config": "//:.scalafmt.conf",
-}
+_scalafmt_defaults = TOOLCHAIN_DEFAULTS["scalafmt"]
 
 _scalafmt_attrs = {
     "default_config": attr.label(
@@ -100,24 +99,16 @@ _scalafmt_attrs = {
     ),
 }
 
-_scala_proto_defaults = {
-    "options": [],
-}
+_scala_proto_defaults = TOOLCHAIN_DEFAULTS["scala_proto"]
 
 _scala_proto_attrs = {
-    "options": attr.string_list(
-        default = _scala_proto_defaults["options"],
+    "default_gen_opts": attr.string_list(
+        default = _scala_proto_defaults["default_gen_opts"],
         doc = "Protobuf options, like 'scala3_sources' or 'grpc'",
     ),
 }
 
-_twitter_scrooge_defaults = {
-    "libthrift": None,
-    "scrooge_core": None,
-    "scrooge_generator": None,
-    "util_core": None,
-    "util_logging": None,
-}
+_twitter_scrooge_defaults = TOOLCHAIN_DEFAULTS["twitter_scrooge"]
 
 _twitter_scrooge_attrs = {
     k: attr.label(default = v)
@@ -186,39 +177,51 @@ _toolchain_tag_classes = {
     ),
 }
 
-_tag_classes = _general_tag_classes | _toolchain_tag_classes
+def _toolchain_settings(module_ctx, tags, tc_names, toolchain_defaults):
+    """Configures all builtin toolchains enabled throughout the module graph.
+
+    Configures toolchain options for enabled toolchains that support them based
+    on the root module's settings for each toolchain. In other words, it uses:
 
-def _toolchains(mctx):
-    result = {k: False for k in _toolchain_tag_classes}
+    - the root module's tag class settings, if present; and
+    - the default tag class settings otherwise.
 
-    for mod in mctx.modules:
-        values = {tc: len(getattr(mod.tags, tc)) != 0 for tc in result}
+    This avoids trying to reconcile different toolchain settings across the
+    module graph. Non root modules that require specific settings should either:
 
-        if mod.is_root:
-            return values
+    - publish their required toolchain settings, or
+    - define and register a custom toolchain instead.
 
-        # Don't overwrite `True` values with `False` from another tag.
-        result.update({k: v for k, v in values.items() if v})
+    Args:
+        module_ctx: the module context object
+        tags: a tags object, presumably the result of `root_module_tags()`
+        tc_names: names of all supported toolchains
+        toolchain_defaults: a dict of `{toolchain_name: default options dict}`
 
-    return result
+    Returns:
+        a dict of `{toolchain_name: bool or options dict}` to pass as keyword
+            arguments to `scala_toolchains()`
+    """
+    toolchains = {k: False for k in tc_names}
 
-def _scala_proto_options(mctx):
-    result = {}
+    for mod in module_ctx.modules:
+        values = {tc: len(getattr(mod.tags, tc)) != 0 for tc in toolchains}
 
-    for mod in mctx.modules:
-        for tag in mod.tags.scala_proto:
-            result.update({opt: True for opt in tag.options})
+        # Don't overwrite True values with False from another tag.
+        toolchains.update({k: v for k, v in values.items() if v})
 
-    return sorted(result.keys())
+    for tc, defaults in toolchain_defaults.items():
+        if toolchains[tc]:
+            values = single_tag_values(module_ctx, getattr(tags, tc), defaults)
+            toolchains[tc] = {k: v for k, v in values.items() if v != None}
+
+    return toolchains
+
+_tag_classes = _general_tag_classes | _toolchain_tag_classes
 
 def _scala_deps_impl(module_ctx):
     tags = root_module_tags(module_ctx, _tag_classes.keys())
-    scalafmt = single_tag_values(module_ctx, tags.scalafmt, _scalafmt_defaults)
-    scrooge_deps = single_tag_values(
-        module_ctx,
-        tags.twitter_scrooge,
-        _twitter_scrooge_defaults,
-    )
+    tc_names = [tc for tc in _toolchain_tag_classes]
 
     scala_toolchains(
         overridden_artifacts = repeated_tag_values(
@@ -229,13 +232,9 @@ def _scala_deps_impl(module_ctx):
             tags.compiler_srcjar,
             _compiler_srcjar_attrs,
         ),
-        scala_proto_options = _scala_proto_options(module_ctx),
-        # `None` breaks the `attr.string_dict` in `scala_toolchains_repo`.
-        twitter_scrooge_deps = {k: v for k, v in scrooge_deps.items() if v},
         **(
             single_tag_values(module_ctx, tags.settings, _settings_defaults) |
-            {"scalafmt_%s" % k: v for k, v in scalafmt.items()} |
-            _toolchains(module_ctx)
+            _toolchain_settings(module_ctx, tags, tc_names, TOOLCHAIN_DEFAULTS)
         )
     )
 

scala/private/toolchain_defaults.bzl

@@ -0,0 +1,20 @@
+"""Gathers defaults for toolchain macros in one place.
+
+Used by both //scala:toolchains.bzl and //scala/extensions:deps.bzl.
+"""
+
+load(
+    "//scala/scalafmt/toolchain:setup_scalafmt_toolchain.bzl",
+    _scalafmt = "TOOLCHAIN_DEFAULTS",
+)
+load("//scala_proto:toolchains.bzl", _scala_proto = "TOOLCHAIN_DEFAULTS")
+load(
+    "//twitter_scrooge/toolchain:toolchain.bzl",
+    _twitter_scrooge = "TOOLCHAIN_DEFAULTS",
+)
+
+TOOLCHAIN_DEFAULTS = {
+    "scalafmt": _scalafmt,
+    "scala_proto": _scala_proto,
+    "twitter_scrooge": _twitter_scrooge,
+}

scala/scalafmt/toolchain/setup_scalafmt_toolchain.bzl

@@ -8,6 +8,13 @@ load("//scala:providers.bzl", "declare_deps_provider")
 load("//scala:scala_cross_version.bzl", "version_suffix")
 load("@rules_scala_config//:config.bzl", "SCALA_VERSIONS")
 
+TOOLCHAIN_DEFAULTS = {
+    # Used by `scala_toolchains{,_repo}` to generate
+    # `@rules_scala_toolchains//scalafmt:config`, the default config for
+    # `ext_scalafmt` from `phase_scalafmt_ext.bzl`.
+    "default_config": Label("//:.scalafmt.conf"),
+}
+
 def setup_scalafmt_toolchain(
         name,
         scalafmt_classpath,

scala/toolchains.bzl

@@ -7,6 +7,7 @@ load(
     "scala_version_artifact_ids",
     "setup_scala_compiler_sources",
 )
+load("//scala/private:toolchain_defaults.bzl", "TOOLCHAIN_DEFAULTS")
 load("//scala/scalafmt:scalafmt_repositories.bzl", "scalafmt_artifact_ids")
 load("//scala:scala_cross_version.bzl", "default_maven_server_urls")
 load("//scala:toolchains_repo.bzl", "scala_toolchains_repo")
@@ -18,14 +19,69 @@ load("//third_party/repositories:repositories.bzl", "repositories")
 load(
     "//twitter_scrooge/toolchain:toolchain.bzl",
     "twitter_scrooge_artifact_ids",
-    _TWITTER_SCROOGE_DEPS = "TOOLCHAIN_DEPS",
 )
 load("@rules_scala_config//:config.bzl", "SCALA_VERSIONS")
 
-def _get_unknown_entries(entries, allowed_entries):
-    return [e for e in entries if e not in allowed_entries]
+_DEFAULT_TOOLCHAINS_REPO_NAME = "rules_scala_toolchains"
+
+def _toolchain_opts(tc_arg):
+    """Converts a toolchain parameter to a (bool, dict of options).
+
+    Used by `scala_toolchains` to parse toolchain arguments as True, False,
+    None, or a dict of options.
+
+    Args:
+        tc_arg: a bool, dict, or None
+
+    Returns:
+        a bool indicating whether the toolchain is enabled, and a dict
+            containing any provided toolchain options
+    """
+    if tc_arg == False or tc_arg == None:
+        return False, {}
+    return True, ({} if tc_arg == True else tc_arg)
+
+def _process_toolchain_options(toolchain_defaults, **kwargs):
+    """Checks the validity of toolchain options and provides defaults.
+
+    Updates each toolchain option dictionary with defaults for every missing
+    entry.
+
+    Args:
+        toolchain_defaults: a dict of `{toolchain_name: default options dict}`
+        **kwargs: keyword arguments of the form `toolchain_name = options_dict`
+
+    Returns:
+        a list of error messages for invalid toolchains or options
+    """
+    errors = []
+
+    for tc, options in kwargs.items():
+        defaults = toolchain_defaults.get(tc, None)
+
+        if defaults == None:
+            errors.append("unknown toolchain or doesn't have defaults: " + tc)
+            continue
+
+        unexpected = [a for a in options if a not in defaults]
+
+        if unexpected:
+            plural = "s" if len(unexpected) != 1 else ""
+            errors.append(
+                "unexpected %s toolchain attribute%s: " % (tc, plural) +
+                ", ".join(unexpected),
+            )
+
+        options.update({
+            k: v
+            for k, v in defaults.items()
+            if k not in options and v != None
+        })
+
+    return errors
 
 def scala_toolchains(
+        name = _DEFAULT_TOOLCHAINS_REPO_NAME,
         maven_servers = default_maven_server_urls(),
         overridden_artifacts = {},
         fetch_sources = False,
@@ -36,12 +92,9 @@ def scala_toolchains(
         junit = False,
         specs2 = False,
         scalafmt = False,
-        scalafmt_default_config = Label("//:.scalafmt.conf"),
         scala_proto = False,
-        scala_proto_options = [],
         jmh = False,
-        twitter_scrooge = False,
-        twitter_scrooge_deps = {}):
+        twitter_scrooge = False):
     """Instantiates rules_scala toolchains and all their dependencies.
 
     Provides a unified interface to configuring `rules_scala` both directly in a
@@ -54,6 +107,7 @@ def scala_toolchains(
     All arguments are optional.
 
     Args:
+        name: Name of the generated toolchains repository
         maven_servers: Maven servers used to fetch dependency jar files
         overridden_artifacts: artifacts overriding the defaults for the
             configured Scala version, in the format:
@@ -82,28 +136,26 @@ def scala_toolchains(
         scalatest: whether to instantiate the ScalaTest toolchain
         junit: whether to instantiate the JUnit toolchain
         specs2: whether to instantiate the Specs2 JUnit toolchain
-        scalafmt: whether to instantiate the Scalafmt toolchain
-        scalafmt_default_config: the default config file for Scalafmt targets
-        scala_proto: whether to instantiate the scala_proto toolchain
-        scala_proto_options: protobuf options, like 'scala3_sources' or 'grpc';
-            `scala_proto` must also be `True` for this to take effect
+        scalafmt: boolean or dictionary of Scalafmt options:
+            - default_config: default Scalafmt config file target
+        scala_proto: boolean or dictionary of `setup_scala_proto_toolchain()`
+            options
         jmh: whether to instantiate the Java Microbenchmarks Harness toolchain
-        twitter_scrooge: whether to instantiate the twitter_scrooge toolchain
-        twitter_scrooge_deps: dictionary of string to Label containing overrides
-            for twitter_scrooge toolchain dependency providers with keys:
-                libthrift
-                scrooge_core
-                scrooge_generator
-                util_core
-                util_logging
+        twitter_scrooge: bool or dictionary of `setup_scrooge_toolchain()`
+            options
     """
-    unknown_ts_deps = _get_unknown_entries(
-        twitter_scrooge_deps,
-        _TWITTER_SCROOGE_DEPS,
+    scalafmt, scalafmt_options = _toolchain_opts(scalafmt)
+    scala_proto, scala_proto_options = _toolchain_opts(scala_proto)
+    twitter_scrooge, twitter_scrooge_options = _toolchain_opts(twitter_scrooge)
+
+    errors = _process_toolchain_options(
+        TOOLCHAIN_DEFAULTS,
+        scalafmt = scalafmt_options,
+        scala_proto = scala_proto_options,
+        twitter_scrooge = twitter_scrooge_options,
     )
-
-    if unknown_ts_deps:
-        fail("unknown twitter_scrooge_deps:", ", ".join(unknown_ts_deps))
+    if errors:
+        fail("\n".join(errors))
 
     setup_scala_compiler_sources(scala_compiler_srcjars)
 
@@ -135,7 +187,7 @@ def scala_toolchains(
     if twitter_scrooge:
         artifact_ids_to_fetch_sources.update({
             id: False
-            for id in twitter_scrooge_artifact_ids(**twitter_scrooge_deps)
+            for id in twitter_scrooge_artifact_ids(**twitter_scrooge_options)
         })
 
     for scala_version in SCALA_VERSIONS:
@@ -174,20 +226,21 @@ def scala_toolchains(
         )
 
     scala_toolchains_repo(
+        name = name,
         scalatest = scalatest,
         junit = junit,
         specs2 = specs2,
         scalafmt = scalafmt,
-        scalafmt_default_config = scalafmt_default_config,
+        scalafmt_default_config = scalafmt_options["default_config"],
         scala_proto = scala_proto,
-        scala_proto_options = scala_proto_options,
+        scala_proto_options = scala_proto_options["default_gen_opts"],
         jmh = jmh,
         twitter_scrooge = twitter_scrooge,
-        twitter_scrooge_deps = twitter_scrooge_deps,
+        twitter_scrooge_deps = twitter_scrooge_options,
     )
 
-def scala_register_toolchains():
-    native.register_toolchains("@rules_scala_toolchains//...:all")
+def scala_register_toolchains(name = _DEFAULT_TOOLCHAINS_REPO_NAME):
+    native.register_toolchains("@%s//...:all" % name)
 
 def scala_register_unused_deps_toolchains():
     native.register_toolchains(