Skip to content

Commit b54075e

Browse files
schneemsschneems
authored
Hybrid Rust (outside) + Bash (inside) (#47)
* Rewrite it in Rust (RiiR) Rewrites the project from Bash to Rust (previously was rewritten from Ruby to Bash). ## Why Rust? I bet you're thinking "Why not something simpler like bash or Ruby?" This library was originally written in Ruby and shelled out. That caused bootstrapping problems, for example when rolling out ARM support, the github action for installing Ruby did not yet support ARM so we had to re-write the logic in Bash (or bootstrap our own version of Ruby with bash). We chose to re-write the library in bash. So why not keep it in bash? Even though bash doesn't need "bootstrapping" authors rely on system tools and packages which may or may not already be installed, and may or may not vary between operating systems. For example GNU grep uses different arguments than BSD (mac) grep. So while there's not a "bash bootstrapping problem" installing dependencies and ensuring scripts work across multiple platforms is tricky. It's easy to write quick scripts, but hard to maintain and do well. Don't you have a Rust bootstrapping problem now? As of Ruby 3.2 YJIT requires a Rust toolchain to support the `--enable-yjit` config flag, so Rust is already a requirement of a full Ruby install. That means that even if we didn't write our scripts in Rust, we still need to have it available on the system when we build Ruby anyway. It's also historically been an easy to install language. * Use proper java properties parsing Turns out, we don't need to re-write java properties parsing logic, an existing crate exists. * Prefer `expect` over `unwrap` * Ensure directory exists before writing tar dir * Fix CI * Split long command to multiple lines * Update build action, introduce jruby_build action * Standardize error reporting * Clean up env var representation * GHA commands on one line * Hybrid workflow Outer scripts are still Rust, inner script is now bash * Clean up bash script - Reduce logic duplication by moving the `./configure` options to an array - Add some minimal comments - Prettier command logging via trap instead of `set -o trace` * Rename `inside_docker` to `shared` Previously this held the `make_ruby.rs` script and a shared set of tools. Now it only holds those tools. * Reduce/remove expect() calls * Workflow commands on one line * Update docs
1 parent 175b5c1 commit b54075e

File tree

100 files changed

+4972
-2150
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

100 files changed

+4972
-2150
lines changed

‎.github/CODEOWNERS

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@
33

44
# However, request review from the language owner instead for files that are
55
# updated by Dependabot, to reduce team review request noise.
6-
Gemfile.lock @schneems
6+
Cargo.lock @schneems

‎.github/workflows/build_jruby.yml

+53
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
name: Build and upload JRuby runtime
2+
run-name: "Build and upload JRuby ${{ inputs.jruby_version }}${{ inputs.dry_run && ' (dry run)' || '' }}"
3+
4+
on:
5+
workflow_dispatch:
6+
inputs:
7+
jruby_version:
8+
description: "The JRuby version to build, specified as X.Y.Z (i.e. 9.4.7.0)"
9+
type: string
10+
required: true
11+
dry_run:
12+
description: "Skip deploying to S3 (e.g. for testing)"
13+
type: boolean
14+
default: false
15+
required: false
16+
17+
permissions:
18+
contents: read
19+
20+
env:
21+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
22+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
23+
AWS_DEFAULT_REGION: "us-east-1"
24+
S3_BUCKET: "heroku-buildpack-ruby"
25+
26+
jobs:
27+
build-and-upload:
28+
runs-on: pub-hk-ubuntu-22.04-xlarge
29+
strategy:
30+
fail-fast: false
31+
matrix:
32+
base_image: ["heroku-20", "heroku-22", "heroku-24"]
33+
steps:
34+
- name: Checkout
35+
uses: actions/checkout@v4
36+
- name: Update Rust toolchain
37+
run: rustup update
38+
- name: Rust Cache
39+
uses: Swatinem/[email protected]
40+
- name: Cargo build
41+
run: cargo build
42+
- name: Output CHANGELOG
43+
run: cargo run --bin jruby_changelog -- --version "${{inputs.jruby_version}}"
44+
- name: Build Ruby
45+
run: cargo run --bin jruby_build -- --version ${{inputs.jruby_version}} --base-image ${{matrix.base_image}} \
46+
- name: Check Ruby
47+
run: cargo run --bin jruby_check -- --version ${{inputs.jruby_version}} --base-image ${{matrix.base_image}} --arch amd64 | tee $GITHUB_STEP_SUMMARY
48+
- name: Upload Ruby runtime archive to S3 dry run
49+
if: (inputs.dry_run)
50+
run: aws s3 sync ./builds "s3://${S3_BUCKET}" --dryrun
51+
- name: Upload Ruby runtime archive to S3 production
52+
if: (!inputs.dry_run)
53+
run: aws s3 sync ./builds "s3://${S3_BUCKET}"

‎.github/workflows/build_ruby.yml

+22-97
Original file line numberDiff line numberDiff line change
@@ -24,107 +24,32 @@ env:
2424
S3_BUCKET: "heroku-buildpack-ruby"
2525

2626
jobs:
27-
build-and-upload-heroku-20:
28-
runs-on: pub-hk-ubuntu-22.04-xlarge
29-
env:
30-
STACK: "heroku-20"
27+
build_ruby:
28+
runs-on: ${{ matrix.arch == 'arm64' && 'pub-hk-ubuntu-22.04-arm-xlarge' || 'pub-hk-ubuntu-22.04-xlarge' }}
29+
strategy:
30+
matrix:
31+
arch: ["amd64", "arm64"]
32+
base_image: ["heroku-20", "heroku-22", "heroku-24"]
33+
exclude:
34+
- base_image: "heroku-20"
35+
arch: "arm64"
36+
- base_image: "heroku-22"
37+
arch: "arm64"
3138
steps:
3239
- name: Checkout
3340
uses: actions/checkout@v4
41+
- name: Update Rust toolchain
42+
run: rustup update
43+
- name: Rust Cache
44+
uses: Swatinem/[email protected]
45+
- name: Cargo build
46+
run: cargo build
3447
- name: Output CHANGELOG
35-
run: bin/print_changelog "${{inputs.ruby_version}}"
36-
- name: Build Docker image
37-
run: bin/activate_docker "$STACK"
38-
- name: Build and package Ruby runtime
39-
run: bin/build_ruby "$STACK" "${{inputs.ruby_version}}"
40-
- name: Verify ruby executable and output rubygems version
41-
run: bin/print_summary "$STACK" "${{inputs.ruby_version}}" | tee $GITHUB_STEP_SUMMARY
42-
- name: Upload Ruby runtime archive to S3 dry run
43-
if: (inputs.dry_run)
44-
run: aws s3 sync ./builds "s3://${S3_BUCKET}" --dryrun
45-
- name: Upload Ruby runtime archive to S3 production
46-
if: (!inputs.dry_run)
47-
run: aws s3 sync ./builds "s3://${S3_BUCKET}"
48-
49-
build-and-upload-heroku-22:
50-
if: (!startsWith(inputs.ruby_version, '3.0')) # https://bugs.ruby-lang.org/issues/18658
51-
runs-on: pub-hk-ubuntu-22.04-xlarge
52-
env:
53-
STACK: "heroku-22"
54-
steps:
55-
- name: Checkout
56-
uses: actions/checkout@v4
57-
- name: Output CHANGELOG
58-
run: bin/print_changelog "${{inputs.ruby_version}}"
59-
- name: Build Docker image
60-
run: bin/activate_docker "$STACK"
61-
- name: Build and package Ruby runtime
62-
run: bin/build_ruby "$STACK" "${{inputs.ruby_version}}"
63-
- name: Verify ruby executable and output rubygems version
64-
run: bin/print_summary "$STACK" "${{inputs.ruby_version}}" | tee $GITHUB_STEP_SUMMARY
65-
- name: Upload Ruby runtime archive to S3 dry run
66-
if: (inputs.dry_run)
67-
run: aws s3 sync ./builds "s3://${S3_BUCKET}" --dryrun
68-
- name: Upload Ruby runtime archive to S3 production
69-
if: (!inputs.dry_run)
70-
run: aws s3 sync ./builds "s3://${S3_BUCKET}"
71-
72-
build-and-upload-heroku-24-amd:
73-
if: (!startsWith(inputs.ruby_version, '3.0')) # https://bugs.ruby-lang.org/issues/18658
74-
runs-on: pub-hk-ubuntu-22.04-xlarge
75-
env:
76-
STACK: "heroku-24"
77-
steps:
78-
- name: Checkout
79-
uses: actions/checkout@v4
80-
- name: Output CHANGELOG
81-
run: bin/print_changelog "${{inputs.ruby_version}}"
82-
- name: Build Docker image
83-
run: bin/activate_docker "$STACK"
84-
- name: Build and package Ruby runtime
85-
run: bin/build_ruby "$STACK" "${{inputs.ruby_version}}"
86-
- name: Verify ruby executable and output rubygems version
87-
run: bin/print_summary "$STACK" "${{inputs.ruby_version}}" amd64 | tee $GITHUB_STEP_SUMMARY
88-
- name: Upload Ruby runtime archive to S3 dry run
89-
if: (inputs.dry_run)
90-
run: aws s3 sync ./builds "s3://${S3_BUCKET}" --dryrun
91-
- name: Upload Ruby runtime archive to S3 production
92-
if: (!inputs.dry_run)
93-
run: aws s3 sync ./builds "s3://${S3_BUCKET}"
94-
95-
build-and-upload-heroku-24-arm:
96-
if: (!startsWith(inputs.ruby_version, '3.0')) # https://bugs.ruby-lang.org/issues/18658
97-
runs-on: pub-hk-ubuntu-22.04-arm-large
98-
env:
99-
STACK: "heroku-24"
100-
steps:
101-
- name: Checkout
102-
uses: actions/checkout@v4
103-
# Docker (and other tools) are not present on the early-access runners.
104-
# We must install them manually: https://github.com/github-early-access/arm-runners-beta
105-
- name: Install docker
106-
run: |
107-
bin/setup_docker_ci
108-
109-
sudo usermod -aG docker $USER
110-
sudo apt-get install acl
111-
sudo setfacl --modify user:$USER:rw /var/run/docker.sock
112-
# AWS CLI (and other tools) are not present on the early-access runners.
113-
# We must install them manually: https://github.com/github-early-access/arm-runners-beta
114-
- name: Install AWS CLI
115-
run: | # https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
116-
sudo apt-get install -y unzip
117-
curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"
118-
unzip awscliv2.zip
119-
sudo ./aws/install
120-
- name: Output CHANGELOG
121-
run: bin/print_changelog "${{inputs.ruby_version}}"
122-
- name: Build Docker image
123-
run: bin/activate_docker "$STACK"
124-
- name: Build and package Ruby runtime
125-
run: bin/build_ruby "$STACK" "${{inputs.ruby_version}}"
126-
- name: Verify ruby executable and output rubygems version
127-
run: bin/print_summary "$STACK" "${{inputs.ruby_version}}" arm64 | tee $GITHUB_STEP_SUMMARY
48+
run: cargo run --bin ruby_changelog -- --version "${{inputs.ruby_version}}"
49+
- name: Build Ruby
50+
run: cargo run --bin ruby_build -- --version ${{inputs.ruby_version}} --base-image ${{matrix.base_image}} --arch ${{matrix.arch}}
51+
- name: Check Ruby
52+
run: cargo run --bin ruby_check -- --version ${{inputs.ruby_version}} --base-image ${{matrix.base_image}} --arch ${{matrix.arch}} | tee $GITHUB_STEP_SUMMARY
12853
- name: Upload Ruby runtime archive to S3 dry run
12954
if: (inputs.dry_run)
13055
run: aws s3 sync ./builds "s3://${S3_BUCKET}" --dryrun

‎.github/workflows/ci.yml

+71-72
Original file line numberDiff line numberDiff line change
@@ -5,101 +5,100 @@ on:
55
- pull_request
66

77
jobs:
8-
lint:
8+
shellcheck:
99
runs-on: ubuntu-latest
1010
steps:
11-
- name: Checkout code
11+
- name: Checkout
1212
uses: actions/checkout@v4
13-
- name: Set up Ruby
14-
uses: ruby/setup-ruby@v1
15-
with:
16-
ruby-version: 3.2
17-
bundler-cache: true
18-
- name: Linting
19-
run: bundle exec standardrb --no-fix
13+
- name: Run ShellCheck
14+
run: shellcheck *.sh -x
2015

21-
test:
16+
lint:
2217
runs-on: ubuntu-latest
2318
steps:
24-
- name: Checkout code
19+
- name: Checkout
2520
uses: actions/checkout@v4
26-
- name: Set up Ruby
27-
uses: ruby/setup-ruby@v1
28-
with:
29-
ruby-version: '3.2'
30-
bundler-cache: true
31-
- name: test
32-
run: bundle exec rspec spec
21+
- name: Update Rust toolchain
22+
# Most of the time this will be a no-op, since GitHub releases new images every week
23+
# which include the latest stable release of Rust, Rustup, Clippy and rustfmt.
24+
run: rustup update
25+
- name: Rust Cache
26+
uses: Swatinem/[email protected]
27+
- name: Clippy
28+
# Using --all-targets so tests are checked and --deny to fail on warnings.
29+
# Not using --locked here and below since Cargo.lock is in .gitignore.
30+
run: cargo clippy --all-targets --all-features -- --deny warnings
31+
- name: rustfmt
32+
run: cargo fmt -- --check
33+
- name: Check docs
34+
# Using RUSTDOCFLAGS until `cargo doc --check` is stabilised:
35+
# https://github.com/rust-lang/cargo/issues/10025
36+
run: RUSTDOCFLAGS="-D warnings" cargo doc --all-features --document-private-items --no-deps
3337

34-
integration_test:
35-
runs-on: pub-hk-ubuntu-22.04-xlarge
36-
strategy:
37-
matrix:
38-
stack: ["heroku-20", "heroku-22"]
39-
version: ["3.1.4"]
38+
unit-test:
39+
runs-on: ubuntu-latest
4040
steps:
4141
- name: Checkout
4242
uses: actions/checkout@v4
43-
- name: Output CHANGELOG
44-
run: bin/print_changelog "${{matrix.version}}"
45-
- name: Build Docker image
46-
run: bin/activate_docker "${{matrix.stack}}"
47-
- name: Build and package Ruby runtime
48-
run: bin/build_ruby "${{matrix.stack}}" "${{matrix.version}}"
49-
- name: Verify ruby executable and output rubygems version
50-
run: bin/print_summary "${{matrix.stack}}" "${{matrix.version}}" | tee $GITHUB_STEP_SUMMARY
43+
- name: Update Rust toolchain
44+
run: rustup update
45+
- name: Rust Cache
46+
uses: Swatinem/[email protected]
47+
- name: Run unit tests
48+
run: cargo test --all-features
5149

52-
integration_test-heroku-24-amd:
53-
runs-on: pub-hk-ubuntu-22.04-xlarge
50+
ruby_integration_test:
51+
runs-on: ${{ matrix.arch == 'arm64' && 'pub-hk-ubuntu-22.04-arm-medium' || 'ubuntu-latest' }}
5452
strategy:
5553
matrix:
56-
stack: ["heroku-24"]
54+
base_image: ["heroku-20", "heroku-22", "heroku-24"]
5755
version: ["3.2.3"]
56+
arch: ["arm64", "amd64"]
57+
exclude:
58+
- base_image: "heroku-20"
59+
arch: "arm64"
60+
- base_image: "heroku-22"
61+
arch: "arm64"
5862
steps:
5963
- name: Checkout
6064
uses: actions/checkout@v4
65+
- name: Update Rust toolchain
66+
run: rustup update
67+
- name: Rust Cache
68+
uses: Swatinem/[email protected]
69+
- name: Cargo build (to make test logs shorter)
70+
run: cargo build
6171
- name: Output CHANGELOG
62-
run: bin/print_changelog "${{matrix.version}}"
63-
- name: Build Docker image
64-
run: bin/activate_docker "${{matrix.stack}}"
65-
- name: Build and package Ruby runtime
66-
run: bin/build_ruby "${{matrix.stack}}" "${{matrix.version}}"
67-
- name: Verify ruby executable and output rubygems version
68-
run: bin/print_summary "${{matrix.stack}}" "${{matrix.version}}" amd64 | tee $GITHUB_STEP_SUMMARY
72+
run: cargo run --bin ruby_changelog -- --version "${{matrix.version}}"
73+
- name: Build Ruby
74+
run: cargo run --bin ruby_build -- --version ${{matrix.version}} --base-image ${{matrix.base_image}} --arch ${{matrix.arch}}
75+
- name: Check Ruby
76+
run: cargo run --bin ruby_check -- --version ${{matrix.version}} --base-image ${{matrix.base_image}} --arch ${{matrix.arch}}
6977

70-
integration_test-heroku-24-arm:
71-
runs-on: pub-hk-ubuntu-22.04-arm-large
78+
jruby_integration_test:
79+
runs-on: ${{ matrix.arch == 'arm64' && 'pub-hk-ubuntu-22.04-arm-medium' || 'ubuntu-latest' }}
7280
strategy:
7381
matrix:
74-
stack: ["heroku-24"]
75-
version: ["3.2.3"]
82+
base_image: ["heroku-20", "heroku-22", "heroku-24"]
83+
version: ["9.4.7.0"]
84+
arch: ["arm64", "amd64"]
85+
exclude:
86+
- base_image: "heroku-20"
87+
arch: "arm64"
88+
- base_image: "heroku-22"
89+
arch: "arm64"
7690
steps:
7791
- name: Checkout
7892
uses: actions/checkout@v4
79-
# Docker (and other tools) are not present on the early-access runners.
80-
# We must install them manually: https://github.com/github-early-access/arm-runners-beta
81-
- name: Install docker
82-
run: |
83-
bin/setup_docker_ci
84-
85-
sudo usermod -aG docker $USER
86-
sudo apt-get install acl
87-
sudo setfacl --modify user:$USER:rw /var/run/docker.sock
88-
# AWS CLI (and other tools) are not present on the early-access runners.
89-
# We must install them manually: https://github.com/github-early-access/arm-runners-beta
90-
- name: Install AWS CLI
91-
run: | # https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
92-
sudo apt-get install -y unzip
93-
curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"
94-
unzip awscliv2.zip
95-
sudo ./aws/install
96-
- name: Verify AWS installed correctly
97-
run: aws --version
93+
- name: Update Rust toolchain
94+
run: rustup update
95+
- name: Rust Cache
96+
uses: Swatinem/[email protected]
97+
- name: Cargo build (to make test logs shorter)
98+
run: cargo build
9899
- name: Output CHANGELOG
99-
run: bin/print_changelog "${{matrix.version}}"
100-
- name: Build Docker image
101-
run: bin/activate_docker "${{matrix.stack}}"
102-
- name: Build and package Ruby runtime
103-
run: bin/build_ruby "${{matrix.stack}}" "${{matrix.version}}"
104-
- name: Verify ruby executable and output rubygems version
105-
run: bin/print_summary "${{matrix.stack}}" "${{matrix.version}}" arm64 | tee $GITHUB_STEP_SUMMARY
100+
run: cargo run --bin jruby_changelog -- --version "${{matrix.version}}"
101+
- name: Build JRuby
102+
run: cargo run --bin jruby_build -- --version ${{matrix.version}} --base-image ${{matrix.base_image}}
103+
- name: Check JRuby
104+
run: cargo run --bin jruby_check -- --version ${{matrix.version}} --base-image ${{matrix.base_image}} --arch ${{matrix.arch}}

‎.gitignore

+6
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,14 @@
11
.bundle/
22
builds/
3+
output/
34
cache/
45
vendor/
56
.DS_Store
67

78
test/fixtures/repos/*
89
Dockerfile
10+
11+
# Added by cargo
12+
target/
13+
cli/target
14+
inside_docker/target

‎.standard.yml

-5
This file was deleted.

0 commit comments

Comments
 (0)