Skip to content

Explore adding a reproducibility test to rust test infrastructure. #139793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Explore adding a reproducibility test to rust test infrastructure. #139793

wants to merge 5 commits into from

Conversation

biabbas
Copy link
Contributor

@biabbas biabbas commented Apr 14, 2025 •
edited by marcoieni
Loading

Fixes #75362
Trying to add a reproducibility check on rust infrastructure itself to detect reproducibility issues.
These were the issues encountered till now:

  1. On ci only nightly build is supported because of an issue here: https://github.com/rust-lang/rust/blob/master/src/build_helper/src/git.rs#L143
  2. Because this is a nightly build, llvm is built from sources. So space was an issue. This was resolved by deleting the copy of source and all build contents except for stage1 artifacts.

Questions:

  1. The check takes about 4hours to complete for now. Is this sustainable for each pr?
  2. Can we add this as a post merge workflow?
    This is also related to issue: Rust reproducibility issue - Finding the proper fix #134589

@rustbot
Copy link
Collaborator

rustbot commented Apr 14, 2025

r? @marcoieni

rustbot has assigned @marcoieni.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

@rustbot rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 14, 2025
@biabbas biabbas force-pushed the reproducible branch 3 times, most recently from e5f3dfb to 95d3d7e Compare April 15, 2025 04:16
@biabbas biabbas force-pushed the reproducible branch 3 times, most recently from fe3b2c7 to 35d08c2 Compare April 15, 2025 09:06
@biabbas biabbas force-pushed the reproducible branch 2 times, most recently from 238555f to 916f799 Compare April 16, 2025 04:04
@biabbas biabbas mentioned this pull request Apr 16, 2025
Open
@biabbas biabbas force-pushed the reproducible branch 4 times, most recently from 4272cba to cfc6ce8 Compare April 17, 2025 06:35
@rustbot rustbot added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Apr 18, 2025
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@biabbas
Copy link
Contributor Author

biabbas commented Apr 22, 2025

@rustbot label -T-bootstrap

@rustbot rustbot removed the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Apr 22, 2025
@biabbas
Copy link
Contributor Author

biabbas commented Apr 22, 2025

@rustbot label +T-infra

@rustbot rustbot added the T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. label Apr 22, 2025
@biabbas biabbas marked this pull request as ready for review April 22, 2025 03:08
@biabbas biabbas marked this pull request as draft April 22, 2025 03:09
@biabbas biabbas marked this pull request as ready for review April 22, 2025 03:09
marcoieni
marcoieni reviewed Apr 22, 2025
View reviewed changes
.github/workflows/repro_check.yml

jobs:
build_and_compare:
runs-on: ubuntu-latest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
runs-on: ubuntu-latest
runs-on: ubuntu-24.04

Let's pin the ubuntu version to avoid surprises

.github/workflows/repro_check.yml
Comment on lines +7 to +10
pull_request:
branches:
- master
- reproducible
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pull_request:
branches:
- master
- reproducible

If this job takes 4 hours we can't run it on pull requests because it would be too slow.

.github/workflows/repro_check.yml

steps:
- name: Checkout repository
uses: actions/checkout@v2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
uses: actions/checkout@v2
uses: actions/checkout@v4

.github/workflows/repro_check.yml
Comment on lines +20 to +24
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.x' # Adjust to the version you need

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.x' # Adjust to the version you need

can't we use the default python preinstalled in github actions runners?
I think this is what we do when we run x.py usually.

actions/setup-python is not used in the codebase

.github/workflows/repro_check.yml
@@ -0,0 +1,79 @@
name: Build and Diff Projects
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a comment explaining what this job does and what problem does it solve.

.github/workflows/repro_check.yml
cp -r "$STAGE2_DIR" .
echo "Contents stage 2 dir : `ls stage2`"
rm -rf build
popd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the code is the same but the directory is different. Can you deduplicate this code?
Also add comments to explain what you are doing.

.github/workflows/repro_check.yml
fi

# Perform a diff between the two builds
diff -r ../buildA/stage2 ../buildA_extended/stage2 || echo "Differences found!"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please explain what the user should do in case of differences and what does it mean that there are differences.

.github/workflows/repro_check.yml
uses: actions/upload-artifact@v4
with:
name: buildA_extended
path: buildA_extended.tar.gz
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add comments:

  • What are those artifacts?
  • Why are the artifacts useful?
  • What should we do with these build artifacts?
  • Should we manually inspect them every time a PR is merged?

@marcoieni
Copy link
Member

2. space was an issue. This was resolved by deleting the copy of source and all build contents except for stage1 artifacts.

please write this as a comment in the code 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marcoieni marcoieni marcoieni left review comments

Assignees

@marcoieni marcoieni

Labels
S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

CI for deterministic / reproducible builds
4 participants
@biabbas @rustbot @rust-log-analyzer @marcoieni