fix(precompile): autodetecting toolchain and feature flags

With this we can finally be correct with how we deal with the precompile
toolchains. This is done in two ways:
1. Add a config_setting that explicitly enableds/disables the precompile
   toolchains.
2. Add an autodetecting toolchain for precompile in cases where users
   may want to use the autodetecting toolchain and we would like to
   ensure that the hermetic toolchain is not pulled in because it exists
   and is used in the target.

To keep the behaviour backwards compatible, add an alias to the
autodetecting toolchain with a deprecation notice. The API is new in
0.33.0 and it is unfortunate that we have to do this, but the users
should still be able to use the code as is.

Fixes #1967.

Author: aignas

CHANGELOG.md

@@ -28,7 +28,14 @@ A brief description of the categories of changes:
 * `protobuf`/`com_google_protobuf` dependency bumped to `v24.4`
 
 ### Fixed
-* Nothing yet
+* (precompile) The toolchains are no longer automatically matched if the
+  precompiling feature is disabled. It can be enabled as described in the
+  updated docs. What is more, the autodetecting toolchain should be registered
+  via `--extra_toolchains=@rules_python//python/autodetecting_toolchain:all`
+  instead of
+  `--extra_toolchains=@rules_python//python:autodetecting_toolchain`. The old
+  target is going to work, but may be removed in the future release.
+  Fixes [1967](https://github.com/bazelbuild/rules_python/issues/1967).
 
 ### Removed
 * Nothing yet

docs/sphinx/api/python/autodetecting_toolchain/index.md

@@ -0,0 +1,24 @@
+:::{default-domain} bzl
+:::
+:::{bzl:currentfile} //python/autodetecting_toolchain:BUILD.bazel
+:::
+
+# //python/autodetecting_toolchain
+
+::::{target} all
+
+A simple set of toolchains that simply uses `python3` from the runtime environment.
+
+Note that this toolchain provides no build-time information, which makes it of
+limited utility.
+
+This is only provided to aid migration off the builtin Bazel toolchain 
+(`@bazel_tools//python:autodetecting_toolchain`), and is largely only applicable
+to WORKSPACE builds.
+
+:::{deprecated} unspecified
+
+Switch to using a hermetic toolchain or manual toolchain configuration instead.
+:::
+
+::::

docs/sphinx/api/python/config_settings/index.md

@@ -10,6 +10,17 @@ Determines the default hermetic Python toolchain version. This can be set to
 one of the values that `rules_python` maintains.
 :::
 
+::::{bzl:flag} precompile_toolchain
+Determines if the precompile toolchains are used.
+
+Values:
+
+* `enabled`: Allow matching of the registered toolchains at build time.
+* `disabled`: Don't match registered precompile toolchains at build time.
+:::{versionadded} 0.33.2
+:::
+::::
+
 ::::{bzl:flag} precompile
 Determines if Python source files should be compiled at build time.
 
@@ -34,6 +45,8 @@ Values:
 * `force_disabled`: Like `disabled`, except overrides target-level setting. This
   is useful useful for development, testing enabling precompilation more
   broadly, or as an escape hatch if build-time compiling is not available.
+:::{versionadded} 0.33.0
+:::
 ::::
 
 ::::{bzl:flag} precompile_source_retention
@@ -51,9 +64,11 @@ Values:
 * `omit_source`: Don't include the orignal py source.
 * `omit_if_generated_source`: Keep the original source if it's a regular source
   file, but omit it if it's a generated file.
+:::{versionadded} 0.33.0
+:::
 ::::
 
-:::{bzl:flag} precompile_add_to_runfiles
+::::{bzl:flag} precompile_add_to_runfiles
 Determines if a target adds its compiled files to its runfiles.
 
 When a target compiles its files, but doesn't add them to its own runfiles, it
@@ -66,7 +81,9 @@ Values:
   runfiles; they are still added to {bzl:obj}`PyInfo.transitive_pyc_files`. See
   also: {bzl:obj}`py_binary.pyc_collection` attribute. This is useful for allowing
   incrementally enabling precompilation on a per-binary basis.
+:::{versionadded} 0.33.0
 :::
+::::
 
 ::::{bzl:flag} pyc_collection
 Determine if `py_binary` collects transitive pyc files.
@@ -78,6 +95,8 @@ This flag is overridden by the target level `pyc_collection` attribute.
 Values:
 * `include_pyc`: Include `PyInfo.transitive_pyc_files` as part of the binary.
 * `disabled`: Don't include `PyInfo.transitive_pyc_files` as part of the binary.
+:::{versionadded} 0.33.0
+:::
 ::::
 
 ::::{bzl:flag} py_linux_libc

docs/sphinx/api/python/index.md

@@ -26,18 +26,4 @@ provides:
 
 ::::{target} autodetecting_toolchain
 
-A simple toolchain that simply uses `python3` from the runtime environment.
-
-Note that this toolchain provides no build-time information, which makes it of
-limited utility.
-
-This is only provided to aid migration off the builtin Bazel toolchain 
-(`@bazel_tools//python:autodetecting_toolchain`), and is largely only applicable
-to WORKSPACE builds.
-
-:::{deprecated} unspecified
-
-Switch to using a hermetic toolchain or manual toolchain configuration instead.
-:::
-
-::::
+Deprecated, see `{bzl:target}//python/autodetecting_toolchain:all`.

docs/sphinx/toolchains.md

@@ -233,12 +233,12 @@ use `python3` from the environment a binary runs within. This provides extremely
 limited functionality to the rules (at build time, nothing is knowable about
 the Python runtime).
 
-Bazel itself automatically registers `@bazel_tools//python:autodetecting_toolchain`
+Bazel itself automatically registers `@bazel_tools//python/autodetecting_toolchain:all`
 as the lowest priority toolchain. For WORKSPACE builds, if no other toolchain
 is registered, that toolchain will be used. For bzlmod builds, rules_python
 automatically registers a higher-priority toolchain; it won't be used unless
 there is a toolchain misconfiguration somewhere.
 
 To aid migration off the Bazel-builtin toolchain, rules_python provides
-{obj}`@rules_python//python:autodetecting_toolchain`. This is an equivalent
+{obj}`@rules_python//python/autodetecting_toolchain:all`. This is an equivalent
 toolchain, but is implemented using rules_python's objects.

python/BUILD.bazel

@@ -24,7 +24,6 @@ that @rules_python//python is only concerned with the core rules.
 """
 
 load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
-load("//python/private:autodetecting_toolchain.bzl", "define_autodetecting_toolchain")
 load(":current_py_toolchain.bzl", "current_py_toolchain")
 
 package(default_visibility = ["//visibility:public"])
@@ -319,11 +318,15 @@ toolchain_type(
 # safe if you know for a fact that your build is completely compatible with the
 # version of the `python` command installed on the target platform.
 
-define_autodetecting_toolchain(name = "autodetecting_toolchain")
+alias(
+    name = "autodetecting_toolchain",
+    actual = "//python/autodetecting_toolchain",
+    deprecation = "Use //python/autodetecting_toolchain:all instead; targets under //docs are internal.",
+)
 
 alias(
     name = "autodetecting_toolchain_nonstrict",
-    actual = ":autodetecting_toolchain",
+    actual = "//python/autodetecting_toolchain",
 )
 
 # ========= Packaging rules =========

python/autodetecting_toolchain/BUILD.bazel

@@ -0,0 +1,19 @@
+# Copyright 2024 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load("//python/private:autodetecting_toolchain.bzl", "define_autodetecting_toolchain")
+
+package(default_visibility = ["//:__subpackages__"])
+
+define_autodetecting_toolchain(name = "autodetecting_toolchain")

python/config_settings/BUILD.bazel

@@ -5,6 +5,7 @@ load(
     "PrecompileAddToRunfilesFlag",
     "PrecompileFlag",
     "PrecompileSourceRetentionFlag",
+    "PrecompileToolchainFlag",
     "PycCollectionFlag",
 )
 load(
@@ -29,6 +30,23 @@ construct_config_settings(
     name = "construct_config_settings",
 )
 
+string_flag(
+    name = "precompile_toolchain",
+    build_setting_default = PrecompileToolchainFlag.DISABLED,
+    values = sorted(PrecompileToolchainFlag.__members__.values()),
+    visibility = ["//visibility:private"],
+)
+
+config_setting(
+    name = "is_precompile_toolchain_enabled",
+    flag_values = {
+        "precompile_toolchain": PrecompileToolchainFlag.ENABLED,
+    },
+    # NOTE: Only public because it is used in py_toolchain_suite from toolchain
+    # repositories
+    visibility = ["//visibility:public"],
+)
+
 string_flag(
     name = "precompile",
     build_setting_default = PrecompileFlag.AUTO,

python/private/BUILD.bazel

@@ -59,6 +59,8 @@ bzl_library(
     name = "autodetecting_toolchain_bzl",
     srcs = ["autodetecting_toolchain.bzl"],
     deps = [
+        ":py_exec_tools_toolchain_bzl",
+        ":toolchain_types_bzl",
         "//python:py_runtime_bzl",
         "//python:py_runtime_pair_bzl",
     ],

python/private/autodetecting_toolchain.bzl

@@ -16,6 +16,12 @@
 
 load("//python:py_runtime.bzl", "py_runtime")
 load("//python:py_runtime_pair.bzl", "py_runtime_pair")
+load(":py_exec_tools_toolchain.bzl", "py_exec_tools_toolchain")
+load(
+    ":toolchain_types.bzl",
+    "EXEC_TOOLS_TOOLCHAIN_TYPE",
+    "TARGET_TOOLCHAIN_TYPE",
+)
 
 def define_autodetecting_toolchain(name):
     """Defines the autodetecting Python toolchain.
@@ -65,6 +71,22 @@ def define_autodetecting_toolchain(name):
     native.toolchain(
         name = name,
         toolchain = ":_autodetecting_py_runtime_pair",
-        toolchain_type = ":toolchain_type",
+        toolchain_type = TARGET_TOOLCHAIN_TYPE,
+        visibility = ["//visibility:public"],
+    )
+
+    is_precompile_enabled = Label("//python/config_settings:is_precompile_toolchain_enabled")
+    py_exec_tools_toolchain(
+        name = "_autodetecting_py3_exec_toolchain",
+        exec_interpreter = "//python/private:autodetecting_toolchain_interpreter.sh",
+        exec_interpreter_version_info = None,
+        precompiler = Label("//tools/precompiler:precompiler"),
+        visibility = ["//visibility:private"],
+    )
+    native.toolchain(
+        name = name + "exec",
+        toolchain = ":_autodetecting_py3_exec_toolchain",
+        toolchain_type = EXEC_TOOLS_TOOLCHAIN_TYPE,
+        target_settings = [is_precompile_enabled],
         visibility = ["//visibility:public"],
     )

python/private/flags.bzl

@@ -37,6 +37,15 @@ def _precompile_flag_get_effective_value(ctx):
         value = PrecompileFlag.DISABLED
     return value
 
+# Determines if the Python source precompilation toolchain should be registered.
+# buildifier: disable=name-conventions
+PrecompileToolchainFlag = enum(
+    # Enable registering the precompilation toolchains using the hermetic toolchain.
+    ENABLED = "enabled",
+    # Disable registering the precompilation toolchains using the hermetic toolchain.
+    DISABLED = "disabled",
+)
+
 # Determines if Python source files should be compiled at build time.
 #
 # NOTE: The flag value is overridden by the target-level attribute, except

python/private/py_toolchain_suite.bzl

@@ -100,14 +100,19 @@ def py_toolchain_suite(*, prefix, user_repository_name, python_version, set_pyth
         **kwargs
     )
 
+    is_precompile_enabled = Label("//python/config_settings:is_precompile_toolchain_enabled")
     native.toolchain(
         name = "{prefix}_py_exec_tools_toolchain".format(prefix = prefix),
         toolchain = "@{user_repository_name}//:py_exec_tools_toolchain".format(
             user_repository_name = user_repository_name,
         ),
         toolchain_type = EXEC_TOOLS_TOOLCHAIN_TYPE,
-        # The target settings capture the Python version
-        target_settings = target_settings,
+        # The target settings capture the Python version, the select works like a
+        # (target_settings AND is_precompile_enabled)
+        target_settings = select({
+            is_precompile_enabled: target_settings,
+            "//conditions:default": [is_precompile_enabled],
+        }),
         exec_compatible_with = kwargs.get("target_compatible_with"),
     )