fix: scope Codecov report to product code, excluding test files#6034
Merged
Conversation
jam-jee
force-pushed
the
fix/codecov-scope-product-code
branch
from
July 16, 2026 04:29
d261c4a to
7211044
Compare
zhaoqizqwang
previously approved these changes
Jul 16, 2026
CI runs `pytest --cov=sagemaker` per sub-package and uploads to Codecov, but coverage is not scoped to `src/` product code. Executed test-support scripts (notably `*/tests/data/*.py` training entry-scripts and model definitions) get recorded and uploaded. With no `codecov.yml` ignore list, these dominate the report: recent commits on master show ~98% of the measured files/lines are test files rather than product code (e.g. commit 3f810c6: 285 of 290 reported files are under tests/). This inflates the reported project coverage (~88-90%) far above the true product-code coverage and makes the number unreliable and unstable across commits. Add a repo-root `codecov.yml` that scopes the report to product code via an `ignore:` list (tests, conftest, test-data scripts, codegen and workflow_helper utilities), and defines project/patch status targets and `after_n_builds: 4` so the percentage is computed over all four sub-package uploads. Validated with the Codecov config validator. This follows how peer projects configure Codecov (scikit-learn ignores conftest/externals; aws-cdk ignores non-product paths).
jam-jee
force-pushed
the
fix/codecov-scope-product-code
branch
from
July 16, 2026 22:31
7211044 to
6960a62
Compare
aviruthen
approved these changes
Jul 16, 2026
mujtaba1747
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
CI uploads unit-test coverage to Codecov, but the coverage is not scoped to
src/product code, so test code and test-support scripts dominate the report.Root cause (confirmed): the CI unit-test build runs, from inside each sub-package directory:
--cov=.measures the entire submodule directory, which contains bothsrc/andtests/. So executed test files and test-data scripts (e.g.tests/data/*.pytraining entry-scripts and model definitions) are recorded and uploaded alongside product code. (The buildspec lives in the internal CDK infra package, not this repo.)Inspecting the Codecov API for recent commits confirms the effect:
3f810c67(Release 3.16.0, master)d4493939)~98% of the measured files/lines in the uploaded report are test files, not shippable product code. Consequences:
Note: integration tests contribute no coverage (the integ build runs plain
pytestwithIGNORE_COVERAGE=-and no--cov), so this is purely the unit-coverage upload.Verified real product coverage
A verification run in CI (all four unit suites unioned with a corrected
--cov=sagemaker, test files excluded) shows the true product-code coverage:Fix
Add a repo-root
codecov.ymlthat:ignore:list (tests,conftest.py,tests/datascripts,test_script.py, codegen andworkflow_helperutilities, docs, examples). This corrects the report server-side, independent of the CI--covflag.projecttarget 65% (an explicit floor rather thanauto, so the one-time recalibration from the inflated ~90% down to the true ~71% is not read as a regression, while still guarding against real drops) andpatchtarget 70% on changed lines.codecov.notify.after_n_builds: 4so the project % is computed once all four sub-package uploads are in.Validated with the Codecov config validator (
curl --data-binary @codecov.yml https://codecov.io/validate→Valid!).Recommended companion change (separate, in the CI infra package)
The cleanest upstream fix is to change the CI unit command from
--cov=.to--cov=sagemaker(match by import package name, which follows the installed package and excludes tests) so the generatedcoverage.xmlnever contains test files. That change lives in the internal CodeBuild infra package (SageMakerMLFPySDKInfraCDK/lib/buildspecs.ts), not this repo, so it is intentionally not part of this PR. Thecodecov.ymlhere fixes the reported number regardless, and both changes are complementary.Note on the tox
--fail-undergateEach
tox.inidefinescoverage report --fail-under=86under theruncoverageenv, but CI does not invoke that env (it runstox -e py310,py311,py312), so this change does not affect any CI gate. If a future coverage gate is wired up, the threshold should reflect the real ~71% baseline rather than the old inflated number.Testing
python -c "import yaml; yaml.safe_load(open('codecov.yml'))"→ parses.Valid!and compiles the expectedignoreregexes.codecov.ymlis added.