Emit RunfilesGroupInfo in py_binary targets

🚀 feature request

Relevant Rules

I want py_binary to return an additional provider, RunfilesGroupInfo.
py_library could contribute by also returning this provider, but that's more of an implementation detail.

I'd be open to work on this if you agree that this is a good idea.

Description

I'd like to make packaging of py_binary targets in container images more efficient. To that end, I have designed a set of providers and described their semantics in a new Bazel module.

Describe the solution you'd like

py_library targets should forward RunfilesGroupInfo and RunfilesGroupMetadataInfo from their deps and add their own runfiles (srcs, data) to a group.
py_binary targets should aggregate runfiles groups from their deps and emit RunfilesGroupInfo and RunfilesGroupMetadataInfo that contain at least:

A group for the interpreter + standard library
One group per third party dependency

Additionally, it could also make sense to produce groups for each py_library target in the transitive dependency graph.

Describe alternatives you've considered

Doing nothing would mean we continue creating (very large) container image layers that contain the full runfiles tree of a py_binary target.

Writing custom packaging rules just for rules_python is unsustainable in my opinion.

I am not against having a generic runfiles grouping, however, I don't fully understand yet how this may clash with our current work that we are doing with the VenvSymlink provider. I also agree that custom packaging rules is not the way forward either. A few observations about packaging of rules_python targets.

If we just take the binary output it will be <package-prefix>/target and <package-prefix>/target.runfiles.
Inside there may be symlinks that we have to preserve within the tar files. At the moment those symlinks may come from either the interpreter or the py_library targets.
Usually those files don't change much and the main variables are: interpreter, 3rd party deps and the 1st party code. If needed, the symlinks could be in a separate layer? They usually don't change.
The dependency whl_library rules have <prefix>_<whl_name>_<sha> type of naming. The prefix in here is an implementation detail and is not stable and may change at any time.

A few questions that I don't understand about your proposal is how would the user use it and what is the maintenance burden put on the ruleset maintainers here. Do you plan to be the maintainer of this particular integration? How much coupling do we get when taking on this dependency?

malt3 posted something with some more details bazelbuild/proposals#417

The basic idea is to provide multiple runfiles objects instead of a single one in DefaultInfo. The sum of these differnet runfiles objects equals what's in DefaultInfo

Insofar as we're concerned, the main technical blocker i see is the proposal only supports plan depsets, but we make use of runfiles.root_symlinks, too.

what about preserving symlinks

This can be automatic for Bazel 8+, which has the File.is_symlink attribute. In the py_zipapp code, it mostly just takes a runfiles object and put it in the zip. Windows and Bazel 7 require special hacks, unfortunately.

what about VenvSymlinks

I don't think it would clash, but it could make for a weird interaction. The VenvSymlinks info is metadata that eventually turns into runfiles at the binary level. It can't turn into runfiles earlier because only a binary knows where its venv is created. A library could give back a RunfilesGroupInfo, but it would just be the "raw" extraction of the library, not the "logical" way it looks in a venv. Not necessarily conflicting but more like incomplete.

If py_binary collects, aggregates, and forwards RGI, that would probably be workable. But if an aspect is traversing to find RGI separately, that will probably be strange. Maybe if, when the binary computes the venv, it groups it with the RGI from the 3rd party dep? IDK.

Insofar as we're concerned, the main technical blocker i see is the proposal only supports plan depsets, but we make use of runfiles.root_symlinks, too.

I think you are right. The current proposal doesn't split root_symlinks or symlinks into groups. We should handle that in RGI as well. Let me revise it.

This can be automatic for Bazel 8+, which has the File.is_symlink attribute. In the py_zipapp code, it mostly just takes a runfiles object and put it in the zip. Windows and Bazel 7 require special hacks, unfortunately.

We should definitely add guidance on what to do as an archiver if you encounter files with File.is_symlink. I don't see this as a hard blocker.

If py_binary collects, aggregates, and forwards RGI, that would probably be workable. But if an aspect is traversing to find RGI separately, that will probably be strange. Maybe if, when the binary computes the venv, it groups it with the RGI from the 3rd party dep? IDK.

The way that I found works best is if each dep is able to contribute its own groups (where this makes sense) and the binary rule (py_binary) collects, aggregates and forwards RGI.
So in this case, third party deps could be in their own groups that are aggregated by the binary rule.

I have now overhauled the provider. Each group now stores a full runfiles object with files, symlinks, root_symlinks, and empty files: bazel-contrib/rules_runfiles_group#16

Nice! As long as the rules_runfiles_group remains minimal (right now it is bazel_features and bazel_skylib) and does not require us adding extra WORKSPACE/bzlmod setup steps, I see no reason why we should hold off from accepting contributions implementing the groups in our py_library, py_binary and py_runtime rules.

@aignas

i think i can bite this off - qq (@aignas / @rickeylev) , do you want this feature behind flags or "just ship it since it's an extra provider and doesn't break other stuff"? to me that is probably the biggest source of potential surface area changes.

Go for it! I'd love to see it happen. Just focusing on other languages right now.
Feel free to reach out on the Bazel Slack with any questions regarding RGI!

Awesome - I'll take a stab once I hear back about the flag since per a sketch I have, that materially impacts the shape of this (and I don't want to hand-write [and also don't want to subject y'all to LLM written documentation] a bunch of documentation for a flag that we then end up saying is not needed :P).

A small word of warning: while trying to add support in rules_java, it became clear that we should make the data structure more memory efficient and add a global flag to control if RGI shall be emitted by rules: bazelbuild/rules_java#368

Since no ruleset is producing RGI yet, this is the best time to still change it.

That's certainly an argument in favor of adding the flag 😓.

I'm not an expert in writing rules - but is the expensive part something like the flatten at the very end? I.E. the way rules_img consumes the depsets? Or is it just expensive to have all the intermediate reps around during analysis even if a rules_img flag controls consumption or not of RGI.

@fmeum

I'm not an expert in writing rules - but is the expensive part something like the flatten at the very end? I.E. the way rules_img consumes the depsets? Or is it just expensive to have all the intermediate reps around during analysis even if a rules_img flag controls consumption or not of RGI.

Before the recent changes, everything was unnecessarily expensive (intermedia providers and the final flattening in rules_img).
Now, the intermediate providers are much more efficient. Most importantly, RGI captures a depset of other RGI providers (collected from deps), instead of accumulating all in a dictionary.
The final flattening in rules_img is still not cheap, but I don't think there's any way around that.
The changes are now merged and a new release (0.1.0) is waiting to be merged in the BCR.

The reason for adding a global flag is that I expect this provider to be rejected by the rules_java maintainers unless there's a simple way to avoid any extra per-target retained heap. @fmeum helped me understand what needs to be changed to optimize for heap usage (thanks!).

feat(rules): emit RunfilesGroupInfo from py_binary, py_test, py_libra…

feat(rules): emit RunfilesGroupInfo from py_binary, py_test, and py_r…

Ok I have something worth looking at ---> #3962

Happy to address any feedback. My rule-writing historically has been in the mode of "how can I quickly unblock something $DAYJOB", but I tried to put on my "this is an upstream thing used by many companies" hat on.

feat(rules): emit RunfilesGroupInfo from py_binary, py_test, and py_r…

@rickeylev

@rickeylev friendly bump on #3962 - @aignas suggested it might interact with your venv work

@rickeylev

@rickeylev / @malt3 doing my once a week friendly bump here to see if I am lucky in catching your attention at the right time :)

I tried #3962 by adding below to my MODULE.bazel

archive_override(
    module_name = "rules_python",
    # https://github.com/bazel-contrib/rules_python/pull/3962
    urls = ["https://github.com/FrankPortman/rules_python/archive/c37ba6ddba786420e3bbf910b3c4d6bf840f9996.tar.gz"], 
    strip_prefix = "rules_python-c37ba6ddba786420e3bbf910b3c4d6bf840f9996",
    # https://github.com/bazel-contrib/rules_python/pull/4054
    patches = ["//build_helper/buildfiles:rules_python_4054.patch"],
    patch_strip = 1,
)

and use the flag --@rules_runfiles_group//runfiles_group:enabled=true but that doesn't seem to work. py_binary and all its dependencies are still in a single layer.

Based on AI:

While the PR successfully adds the RunfilesGroupInfo provider to the py_binary rule, it fails to actually tag the underlying PIP dependencies.

In rules_runfiles_group, files only split into multiple layers if the underlying dependencies are explicitly tagged with a specific group (e.g., using aspect_hints to mark them as "dependencies"). Because the PR hasn't fully wired up the pip_parse / whl_library repository rules to automatically tag external wheels, the dependencies are left completely untagged.

When rules_img's layer_from_binary looks at the runfiles, it sees the new provider, but because no files are tagged, it assumes everything belongs to your default application group and squashes it right back down into a single big layer.

@gfrankliu

@gfrankliu I believe the result you saw but I think the AI explanation is a decent ways off here unfortunately.

I am not aware of any tagging mechanism like this in the rules_runfiles_group protocol, and whl_library generates ordinary py_library targets.

Verified the PR head (c37ba6dd) end to end: py_binary with a pip dep (six), rules_img@0.3.19 image_from_binary, rules_python consumed as an external module:

Setup	Layers
Bazel 9.1.1, `--@rules_runfiles_group//runfiles_group:enabled=true`	6 (22.4MB interpreter+stdlib, `six`, per-library, venv, app, exe)
Bazel 9.1.1, flag unset	1
Bazel 8.5.1, `--@rules_runfiles_group//runfiles_group:enabled=true`	1 (feature requires Bazel 9+; silently off on 7/8, see bazelbuild/bazel#23043)

The silent pass through on Bazel < 9 is potentially a problem and maybe we can error more loudly in the PR or rework the flags as @aignas suggested. Also if you are not using image_from_binary/layer_from_binary, I don't think this works either. Were you on Bazel<9 and/or not using {image, layer}_from_binary?

I'm using Bazel 8.5. Is there a way to turn it on?

@rickeylev

Maybe - I'm not an expert on most of this, but I can imagine a env var like RULES_PYTHON_ENABLE_RUNFILES_GROUPS=1 which in combination with --repositories_without_autoloads=rules_runfiles_group+ makes the shim actually load.

That being said I would prefer to get some engagement on the overall direction from @rickeylev / @malt3 first, before re-working anything around this question. It might also be something that is best filed as a followup issue depending on where this shakes out.

Following up on my Bazel 8.5 question: in the meantime we got per-package layering working on Bazel 8.5 without patching rules_python. We use a small wrapper rule that takes a py_binary, sorts its default_runfiles into groups by path, and returns RunfilesGroupInfo for layer_from_binary:

runtime: interpreter + stdlib (RANK_FOUNDATION)
pypi/<name>: one per wheel (RANK_SHARED_DEPS), with weight = file count
app: first-party code, launcher and venv symlinks, as the executable_group, do_not_merge

We migrated all our images to it, including multi-GB GPU training images. A code change now rebuilds only the app layer (1 of 68 in one image), and bumping a small package only re-pushes that package's layer.

Some things we needed that might inform the design being discussed in bazelbuild/rules_java#368:

Bucketing the ML frameworks. torch, tensorflow, jax, triton, tensorrt and nvidia-* are dozens of multi-GB wheels. We want them in one or two stable layers, not merged at random with small packages under layer_budget. A per-package merge affinity, or a documented RunfilesGroupTransformInfo hint for this, would cover it.
Keeping small, often-bumped packages out of big buckets (do_not_merge). Otherwise one bump re-pushes a multi-GB layer.
Size-aware merging. File count is the only size signal available at analysis time, so we use it as weight; it works reasonably well.
A way to opt in on Bazel 8, or at least an error instead of silently producing one layer, as mentioned above.

Happy to share the wrapper rule if it helps as a reference.

@malt3

New PR against the reworked rules_runfiles_group API: #4254. It's a draft until @malt3 finishes the rework and releases it.

@gfrankliu you should be able to test it in the meantime, including on Bazel 8. You'll need the dev registry and the image opt-in:

# .bazelrc
common --registry=https://bazel-contrib.github.io/rules_runfiles_group --registry=https://bcr.bazel.build

# MODULE.bazel
bazel_dep(name = "rules_img", version = "99.0.0-dev.20261009.1.4a8bfc9")
archive_override(
    module_name = "rules_python",
    urls = ["https://github.com/FrankPortman/rules_python/archive/d291a5b3eb7fc644424a4e90829683ce8d7889b1.tar.gz"],
    strip_prefix = "rules_python-d291a5b3eb7fc644424a4e90829683ce8d7889b1",
)

# BUILD
image_from_binary(
    name = "image",
    binary = ":app",
    use_runfiles_groups = True,
)

WORKSPACE isn't supported yet (or ever?).

@FrankPortman

Thanks @FrankPortman! Tested #4254 (d291a5b) on Bazel 8.5.0 with rules_img 99.0.0-dev.20261009.1.4a8bfc9, in our repo on a real Beam/TF image (~175 wheels). It builds on Bazel 8 🎉 and per-package grouping works. Two issues came up:

Wheel .py files land in two groups. With the current whl_library layout, :pkg is a py_library with srcs = [":srcs"]. :srcs correctly becomes rules_python#pypi/<name>, but _own_files for :pkg also collects :srcs's .py files (via DefaultInfo.files / direct_original_sources). Since :pkg has no METADATA in data, they form a second group named after its label. In our image, 16,510 .py files from 130 of 175 wheels are in two layers each. Maybe _own_files should skip files contributed by rule targets in srcs, which describe themselves? (I suspect the "its library" layer in your six test is this.)
With venvs_site_packages=yes, the executable group gets copies of shared libraries. venv_runfiles goes into the binary's group, including the venv's per-file symlinks to wheel files (mostly .so). rules_img stores these as regular files, so our app layer grew from 0.3MB to 720MB compressed (2.57GB uncompressed; libtensorflow_cc.so.2 alone is 1.18GB), and a one-line code change re-pushes all of it. Our wrapper describes each venv file symlink in the group of the file it points to, so rules_img hardlinks it within that layer. Could the describers do the same?

Image	Total	App layer
our wrapper	995MB	0.3MB
#4254, `venvs_site_packages=yes`	1,753MB	720MB
#4254, `venvs_site_packages=no`	1,034MB	0.3MB

Also: all groups get merge_affinity = "rules_python", so bucketing large ML frameworks together under layer_budget still needs an aspect_hints transform on each py_binary. A way to set it centrally (on layer_from_binary, or a package → affinity mapping) would help.

Happy to re-test new commits.

@malt3

Thanks for taking a look at #4254 @malt3.

Friendly bump to @aignas and @rickeylev - it looks like the rules_java maintainers are curious for y'alls take as well :) bazelbuild/rules_java#368 (comment).

@FrankPortman

@FrankPortman I see #4254 now skips files from Python targets in srcs and splits the venv's links into the PyPI groups, which look like fixes for the two problems from my test report above (duplicated wheel .py files, and copies of shared libraries in the binary's layer). It would have helped to get a reply on that report; I'll re-test.

The third point, though, is still unaddressed: every group is stamped with the same merge_affinity = "rules_python", with no way to change it. For ML images that's the part that matters most. Under layer_budget, torch, tensorflow, jax and the nvidia-* wheels (multiple GB) merge with small, frequently bumped packages purely by weight, so a minor version bump can invalidate a multi-GB layer. The only escape hatch is an aspect_hints transform on every individual py_binary, which doesn't scale across a repo.

Our in-house wrapper handles this with a package → bucket mapping and do_not_merge for small packages, and that's the main reason we can't switch to this yet. Could the describers take a configurable affinity (e.g. a flag or a mapping from package name to affinity), or could packagers like layer_from_binary accept a transform centrally? I'd consider that a blocker for ML use cases rather than a follow-up.

Re-tested #4254 at a87d772 on the same image:

	our wrapper	`d291a5b`	`a87d772`
image total	995MB	1,753MB	999MB
binary's layer	0.3MB	720MB	4.7MB
paths in >1 layer	0	16,510	76

Both fixes work, thanks. What's left:

Namespace packages: the venv's per-file links under site-packages/google/ (shared by google-cloud-*, protobuf, googleapis-common-protos, …) aren't assigned to a package, so 2,390 files (97MB uncompressed) are still copied into the binary's layer and re-pushed on every code change.
76 duplicate paths (1.3MB): mostly protobuf, plus wheels' bin/ console scripts.
Affinity: still the blocker for ML images, as noted above.

feat(rules): describe runfiles groups for rules_runfiles_group

Navigation Menu

Repository navigation

Emit RunfilesGroupInfo in py_binary targets #3757

🚀 feature request

Relevant Rules

Description

Describe the solution you'd like

Describe alternatives you've considered

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Emit RunfilesGroupInfo in py_binary targets #3757

Description

🚀 feature request

Relevant Rules

Description

Describe the solution you'd like

Describe alternatives you've considered

Activity

aignas commented on May 8, 2026

rickeylev commented on May 8, 2026

malt3 commented on May 8, 2026

malt3 commented on May 11, 2026

aignas commented on May 12, 2026

FrankPortman commented on Jul 23, 2026

malt3 commented on Jul 23, 2026

FrankPortman commented on Jul 23, 2026

malt3 commented on Jul 25, 2026

FrankPortman commented on Jul 25, 2026

malt3 commented on Jul 27, 2026

FrankPortman commented on Jul 27, 2026

FrankPortman commented on Aug 10, 2026

FrankPortman commented on Aug 19, 2026

gfrankliu commented on Aug 28, 2026

FrankPortman commented on Aug 28, 2026

gfrankliu commented on Aug 28, 2026

FrankPortman commented on Aug 28, 2026

gfrankliu commented on Sep 30, 2026

FrankPortman commented on Oct 12, 2026

gfrankliu commented on Oct 12, 2026

FrankPortman commented on Oct 23, 2026

gfrankliu commented on Oct 24, 2026

gfrankliu commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions