feat(rules): allow deriving custom rules from core rules (#2666)

This exposes public functions for creating builders for py_binary,
py_test, and py_library.
It also adds some docs and examples for how to use them.

I'm calling this a "volatile" API -- it's public, but the pieces that
comprise
it (e.g. all the rule args, attributes, the attribute args, etc) are
likely to change
in various ways, and not all modifications to them can be supported in a
backward
compatible way. Hence the "volatile" term:
* hold it gently and its fine
* shake it a bit and its probably fine
* shake it moderately and something may or may not blow up
* shake it a lot and something will certainly blow up.

Work towards #1647

---------

Co-authored-by: Ignas Anikevicius <[email protected]>

Author: rickeylev

CHANGELOG.md

@@ -91,6 +91,9 @@ Unreleased changes template.
 * (pypi) Direct HTTP urls for wheels and sdists are now supported when using
   {obj}`experimental_index_url` (bazel downloader).
   Partially fixes [#2363](https://github.com/bazelbuild/rules_python/issues/2363).
+* (rules) APIs for creating custom rules based on the core py_binary, py_test,
+  and py_library rules
+  ([#1647](https://github.com/bazelbuild/rules_python/issues/1647))
 
 {#v0-0-0-removed}
 ### Removed

docs/BUILD.bazel

@@ -100,6 +100,8 @@ sphinx_stardocs(
         "//python:py_test_bzl",
         "//python:repositories_bzl",
         "//python/api:api_bzl",
+        "//python/api:executables_bzl",
+        "//python/api:libraries_bzl",
         "//python/cc:py_cc_toolchain_bzl",
         "//python/cc:py_cc_toolchain_info_bzl",
         "//python/entry_points:py_console_script_binary_bzl",

docs/_includes/volatile_api.md

@@ -0,0 +1,5 @@
+:::{important}
+
+**Public, but volatile, API.** Some parts are stable, while others are
+implementation details and may change more frequently.
+:::

docs/extending.md

@@ -0,0 +1,143 @@
+# Extending the rules
+
+:::{important}
+**This is public, but volatile, functionality.**
+
+Extending and customizing the rules is supported functionality, but with weaker
+backwards compatibility guarantees, and is not fully subject to the normal
+backwards compatibility procedures and policies. It's simply not feasible to
+support every possible customization with strong backwards compatibility
+guarantees.
+:::
+
+Because of the rich ecosystem of tools and variety of use cases, APIs are
+provided to make it easy to create custom rules using the existing rules as a
+basis. This allows implementing behaviors that aren't possible using
+wrapper macros around the core rules, and can make certain types of changes
+much easier and transparent to implement.
+
+:::{note}
+It is not required to extend a core rule. The minimum requirement for a custom
+rule is to return the appropriate provider (e.g. {bzl:obj}`PyInfo` etc).
+Extending the core rules is most useful when you want all or most of the
+behavior of a core rule.
+:::
+
+Follow or comment on https://github.com/bazelbuild/rules_python/issues/1647
+for the development of APIs to support custom derived rules.
+
+## Creating custom rules
+
+Custom rules can be created using the core rules as a basis by using their rule
+builder APIs.
+
+* [`//python/apis:executables.bzl`](#python-apis-executables-bzl): builders for
+  executables.
+* [`//python/apis:libraries.bzl`](#python-apis-libraries-bzl): builders for
+  libraries.
+
+These builders create {bzl:obj}`ruleb.Rule` objects, which are thin
+wrappers around the keyword arguments eventually passed to the `rule()`
+function. These builder APIs give access to the _entire_ rule definition and
+allow arbitrary modifications.
+
+This is level of control is powerful, but also volatile. A rule definition
+contains many details that _must_ change as the implementation changes. What
+is more or less likely to change isn't known in advance, but some general
+rules are:
+
+* Additive behavior to public attributes will be less prone to breaking.
+* Internal attributes that directly support a public attribute are likely
+  reliable.
+* Internal attributes that support an action are more likely to change.
+* Rule toolchains are moderately stable (toolchains are mostly internal to
+  how a rule works, but custom toolchains are supported).
+
+## Example: validating a source file
+
+In this example, we derive from `py_library` a custom rule that verifies source
+code contains the word "snakes". It does this by:
+
+* Adding an implicit dependency on a checker program
+* Calling the base implementation function
+* Running the checker on the srcs files
+* Adding the result to the `_validation` output group (a special output
+  group for validation behaviors).
+
+To users, they can use `has_snakes_library` the same as `py_library`. The same
+is true for other targets that might consume the rule.
+
+```
+load("@rules_python//python/api:libraries.bzl", "libraries")
+load("@rules_python//python/api:attr_builders.bzl", "attrb")
+
+def _has_snakes_impl(ctx, base):
+    providers = base(ctx)
+
+    out = ctx.actions.declare_file(ctx.label.name + "_snakes.check")
+    ctx.actions.run(
+        inputs = ctx.files.srcs,
+        outputs = [out],
+        executable = ctx.attr._checker[DefaultInfo].files_to_run,
+        args = [out.path] + [f.path for f in ctx.files.srcs],
+    )
+    prior_ogi = None
+    for i, p in enumerate(providers):
+        if type(p) == "OutputGroupInfo":
+            prior_ogi = (i, p)
+            break
+    if prior_ogi:
+        groups = {k: getattr(prior_ogi[1], k) for k in dir(prior_ogi)}
+        if "_validation" in groups:
+            groups["_validation"] = depset([out], transitive=groups["_validation"])
+        else:
+            groups["_validation"] = depset([out])
+        providers[prior_ogi[0]] = OutputGroupInfo(**groups)
+    else:
+        providers.append(OutputGroupInfo(_validation=depset([out])))
+    return providers
+
+def create_has_snakes_rule():
+    r = libraries.py_library_builder()
+    base_impl = r.implementation()
+    r.set_implementation(lambda ctx: _has_snakes_impl(ctx, base_impl))
+    r.attrs["_checker"] = attrb.Label(
+        default="//:checker",
+        executable = True,
+    )
+    return r.build()
+has_snakes_library = create_has_snakes_rule()
+```
+
+## Example: adding transitions
+
+In this example, we derive from `py_binary` to force building for a particular
+platform. We do this by:
+
+* Adding an additional output to the rule's cfg
+* Calling the base transition function
+* Returning the new transition outputs
+
+```starlark
+
+load("@rules_python//python/api:executables.bzl", "executables")
+
+def _force_linux_impl(settings, attr, base_impl):
+    settings = base_impl(settings, attr)
+    settings["//command_line_option:platforms"] = ["//my/platforms:linux"]
+    return settings
+
+def create_rule():
+    r = executables.py_binary_rule_builder()
+    base_impl = r.cfg.implementation()
+    r.cfg.set_implementation(
+        lambda settings, attr: _force_linux_impl(settings, attr, base_impl)
+    )
+    r.cfg.add_output("//command_line_option:platforms")
+    return r.build()
+
+py_linux_binary = create_linux_binary_rule()
+```
+
+Users can then use `py_linux_binary` the same as a regular py_binary. It will
+act as if `--platforms=//my/platforms:linux` was specified when building it.

docs/index.md

@@ -101,6 +101,7 @@ pip
 coverage
 precompiling
 gazelle
+Extending <extending>
 Contributing <contributing>
 support
 Changelog <changelog>

python/api/BUILD.bazel

@@ -25,6 +25,26 @@ bzl_library(
     deps = ["//python/private/api:api_bzl"],
 )
 
+bzl_library(
+    name = "executables_bzl",
+    srcs = ["executables.bzl"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//python/private:py_binary_rule_bzl",
+        "//python/private:py_executable_bzl",
+        "//python/private:py_test_rule_bzl",
+    ],
+)
+
+bzl_library(
+    name = "libraries_bzl",
+    srcs = ["libraries.bzl"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//python/private:py_library_bzl",
+    ],
+)
+
 filegroup(
     name = "distribution",
     srcs = glob(["**"]),

python/api/executables.bzl

@@ -0,0 +1,31 @@
+# Copyright 2025 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+{#python-apis-executables-bzl}
+Loading-phase APIs specific to executables (binaries/tests).
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+"""
+
+load("//python/private:py_binary_rule.bzl", "create_py_binary_rule_builder")
+load("//python/private:py_executable.bzl", "create_executable_rule_builder")
+load("//python/private:py_test_rule.bzl", "create_py_test_rule_builder")
+
+executables = struct(
+    py_binary_rule_builder = create_py_binary_rule_builder,
+    py_test_rule_builder = create_py_test_rule_builder,
+    executable_rule_builder = create_executable_rule_builder,
+)

python/api/libraries.bzl

@@ -0,0 +1,27 @@
+# Copyright 2025 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+{#python-apis-libraries-bzl}
+Loading-phase APIs specific to libraries.
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+"""
+
+load("//python/private:py_library.bzl", "create_py_library_rule_builder")
+
+libraries = struct(
+    py_library_rule_builder = create_py_library_rule_builder,
+)

python/private/BUILD.bazel

@@ -427,6 +427,7 @@ bzl_library(
         ":attributes_bzl",
         ":common_bzl",
         ":flags_bzl",
+        ":precompile_bzl",
         ":py_cc_link_params_info_bzl",
         ":py_internal_bzl",
         ":rule_builders_bzl",
@@ -446,8 +447,6 @@ bzl_library(
     name = "py_library_rule_bzl",
     srcs = ["py_library_rule.bzl"],
     deps = [
-        ":common_bzl",
-        ":precompile_bzl",
         ":py_library_bzl",
     ],
 )

python/private/attr_builders.bzl

@@ -12,7 +12,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Builders for creating attributes et al."""
+"""Builders for creating attributes et al.
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+"""
 
 load("@bazel_skylib//lib:types.bzl", "types")
 load(