fix(coverage): prevent issues when packages execute files in tmp dir.

[BurnzZ]

Background

(A good addition to #2597)

This is a proposal to have a workaround for #2575 where an inherent issue from coverage.py when creating the lcov output file.

Proposed Changes

Add an --omit='...' flag pointing to tempdir.

Reproducible steps

We can see how this proposal addresses the issue described from https://github.com/BurnzZ/rules_python.

1. Clone the repo: https://github.com/BurnzZ/rules_python

git clone https://github.com/BurnzZ/rules_python.git rules_python_tmp
cd rules_python_tmp/examples
git checkout fix-omit-tmpdir-coveragepy

2. Clone the reproducible example

git clone https://github.com/BurnzZ/bazel-python-coverage-issue.git
cd bazel-python-coverage-issue
git checkout enable-torchvision

3. Add these lines in line 2 of MODULE.bazel:

local_path_override(
    module_name = "rules_python",
    path = "../..",
)

4. Generate the coverage report:

bazel coverage --combined_report=lcov :test --nocache_test_results --test_output=all
lcov --list "$(bazel info output_path)/_coverage/_coverage_report.dat"
genhtml --branch-coverage --output genhtml "$(bazel info output_path)/_coverage/_coverage_report.dat"

[aignas]

My thinking is that the COVERAGE_RC issue was a similar topic - we need to be able to customize coverage in some way but currently we have no way to do that. The coverage tool right now is coupled to the python toolchain, but maybe we should have a python_test toolchain that can touch/configure the following parts:

• Test runner deps
• Test runner itself
• Coverage deps
• Coverage configuration
• Report generation?

We have a proposal here for this: #2246.

Some thoughts about this change:

• We may want to be able to change coverage configuration per target, similar to the precompile flag.
• rules_python should do the right thing out of the box.

I think there are many other things I am forgetting, but wanted to mention greater design concerns here. I am not too much against this as a temporary fix for the actual problem at hand, but I would like to have a better story for testing and configuration in rules_python in general, so if other people have requirements as to what is needed and/or ideas about design, please share them.

[groodt]

we need to be able to customize coverage in some way but currently we have no way to do that

Yes, indeed. There are also solutions to this which might be solved in userspace instead. Similar to https://pypi.org/project/pytest-bazel/

I do like a userspace solution, but the "batteries included" coverage supporting configuration files makes sense while we are bundling coverage with the rules.

[BurnzZ]

Noting that this proposed PR might not be needed since the PR in #2607 that adds --source to the coverage.py call would also fix the issue in #2575. As per nedbat/coveragepy#1921 (comment), using either --source or --omit would fix the issue.

We can close this (alongside #2575) once #2607 is merged.