test: create e2e smoke test flow + use cluster for SP1 proofs #405
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| name: e2e-test-matrix | ||
| description: 'Dynamically discovers and generates a test matrix for e2e tests' | ||
|
There was a problem hiding this comment. To avoid having to specify every test every time. There was a problem hiding this comment. Tbh I actually do like listing. The amount of code added to create the list far exceeds the size of the list by several times, and is significantly harder to read. Likely contains weird edge cases. Why would it be bad to have them listed only in one file such as this one? There was a problem hiding this comment. Mostly because it's hard to know if any are missing and why. It's easy to miss if a test was not added to the list, and it seems like an unnecessary maintenance burden to keep around. There was a problem hiding this comment. Now we will never know if a test is missing. I think we can make a practice to add every test we write to the CI lisr, and just have the ones we don't use commented out with a comment explaining why. I don't trust these scripts to not miss some tests eventually. Idk, wdyt? There was a problem hiding this comment. We can do that of course, but I'd rather have less things to worry about than more - hence the want to reduce the maintenance burden. It's not a big deal, but I do prefer this method which we also use in ibc-go (although there it's a go script). Would you feel more comfortable if it was a go script instead of a bash script? One possibility is to take the go script from ibc-go, pull it out into a separate repo, clean it up and publish it as an action we can use in both repos (and maybe others could use too)? |
||
|
|
||
| inputs: | ||
| skip-tests: | ||
| description: 'Comma-separated list of tests to skip' | ||
| required: false | ||
| default: '' | ||
|
|
||
| outputs: | ||
| matrix: | ||
| description: 'JSON string containing the test matrix' | ||
| value: ${{ steps.generate-matrix.outputs.matrix }} | ||
|
|
||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Generate Test Matrix | ||
| id: generate-matrix | ||
| shell: bash | ||
| run: echo "matrix=$(${{ github.workspace }}/scripts/generate-e2e-test-matrix.sh '${{ inputs.skip-tests }}')" >> $GITHUB_OUTPUT | ||
srdtrk marked this conversation as resolved.
Show resolved
Hide resolved
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| name: setup-e2e | ||
|
|
||
| inputs: | ||
| github_token: | ||
| description: 'A Github PAT' | ||
| required: true | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: "Set up environment" | ||
| uses: ./.github/actions/setup | ||
|
|
||
| - name: Setup Go | ||
| uses: actions/setup-go@v4 | ||
| with: | ||
| go-version: "1.23" | ||
| check-latest: true | ||
| cache-dependency-path: e2e/interchaintestv8/go.sum | ||
|
|
||
| - name: "Cache Relayer and Operator" | ||
| id: cache-relayer | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: | | ||
| ~/.cargo/bin/relayer | ||
| ~/.cargo/bin/operator | ||
| key: ${{ runner.os }}-relayer-${{ hashFiles('Cargo.lock', 'packages/**', 'programs/**', 'abi/**') }} | ||
|
|
||
| - name: Install SP1 toolchain | ||
| if: (steps.cache-relayer.outputs.cache-hit != 'true') | ||
| shell: bash | ||
| run: | | ||
| curl -L https://sp1.succinct.xyz | bash | ||
| ~/.sp1/bin/sp1up --token ${{ inputs.github_token }} | ||
| ~/.sp1/bin/cargo-prove prove --version | ||
|
|
||
| - name: Install operator | ||
| if: steps.cache-relayer.outputs.cache-hit != 'true' | ||
| uses: actions-rs/cargo@v1 | ||
| with: | ||
| command: install | ||
| args: --bin operator --path programs/operator --locked | ||
|
|
||
| - name: Install relayer | ||
| if: steps.cache-relayer.outputs.cache-hit != 'true' | ||
| uses: actions-rs/cargo@v1 | ||
| with: | ||
| command: install | ||
| args: --bin relayer --path programs/relayer --locked | ||
|
|
||
| - name: Setup Kurtosis | ||
| if: env.ETH_TESTNET_TYPE == 'pos' | ||
| shell: bash | ||
| run: | | ||
| echo "deb [trusted=yes] https://apt.fury.io/kurtosis-tech/ /" | sudo tee /etc/apt/sources.list.d/kurtosis.list | ||
| sudo apt update | ||
| sudo apt install kurtosis-cli=1.5.0 # Must match the kurtosis library version we use in the e2e tests | ||
| kurtosis engine start | ||
| kurtosis analytics disable | ||
| echo "$(dirname $(which kurtosis))" >> $GITHUB_PATH |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,51 @@ | ||||||||||
| name: e2e-full | ||||||||||
| on: | ||||||||||
| workflow_dispatch: | ||||||||||
| schedule: | ||||||||||
| - cron: "0 0 */3 * *" # Run every 3 days at midnight UTC | ||||||||||
|
|
||||||||||
| concurrency: | ||||||||||
| group: ${{ github.workflow }}-${{ github.ref }} | ||||||||||
| cancel-in-progress: true | ||||||||||
|
|
||||||||||
| env: | ||||||||||
| FOUNDRY_PROFILE: ci | ||||||||||
| ETH_TESTNET_TYPE: pos | ||||||||||
| SP1_PROVER: network | ||||||||||
| ETHEREUM_POS_NETWORK_PRESET: mainnet | ||||||||||
| permissions: | ||||||||||
| contents: read | ||||||||||
|
|
||||||||||
| jobs: | ||||||||||
| generate-matrix: | ||||||||||
| runs-on: ubuntu-latest | ||||||||||
| outputs: | ||||||||||
| matrix: ${{ steps.get-matrix.outputs.matrix }} | ||||||||||
| steps: | ||||||||||
| - uses: actions/checkout@v4 | ||||||||||
| - name: Generate Test Matrix | ||||||||||
| id: get-matrix | ||||||||||
| uses: ./.github/actions/e2e-test-matrix | ||||||||||
| with: | ||||||||||
| skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos' | ||||||||||
|
There was a problem hiding this comment. Can these be listed properly rather than be comma delimited? Like:
Suggested change
There was a problem hiding this comment. Not really. There is no list type for input, so it would have to be either comma separated, or like a json array, but again, as a string |
||||||||||
|
|
||||||||||
| e2e: | ||||||||||
| needs: generate-matrix | ||||||||||
| strategy: | ||||||||||
| fail-fast: false | ||||||||||
| matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }} | ||||||||||
| name: ${{ matrix.test }} | ||||||||||
| runs-on: depot-ubuntu-22.04 | ||||||||||
| steps: | ||||||||||
| - uses: actions/checkout@v4 | ||||||||||
| - name: "Set up E2E environment" | ||||||||||
| uses: ./.github/actions/setup-e2e | ||||||||||
| with: | ||||||||||
| github_token: ${{ secrets.GITHUB_TOKEN }} | ||||||||||
|
|
||||||||||
| - name: Run Tests | ||||||||||
| env: | ||||||||||
| NETWORK_PRIVATE_KEY: ${{ secrets.SP1_PRIVATE_KEY }} | ||||||||||
| run: | | ||||||||||
| cd e2e/interchaintestv8 | ||||||||||
| go test -v -mod=readonly . -run '^${{ matrix.test }}$' -timeout 120m | ||||||||||
|
There was a problem hiding this comment. can just be called There was a problem hiding this comment. The reason I opted for this, was because I realized the naming got a bit confusing with quick and minimal. By adding pos minimal it is more obvious that it refers to the pos setup being minimal, which is what this one is |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| name: e2e-pos-minimal | ||
| on: | ||
| workflow_dispatch: | ||
| push: | ||
| branches: | ||
| - main | ||
| schedule: | ||
| - cron: "0 3 * * *" # Run daily at 3 AM UTC (outside EU and US work hours) | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| FOUNDRY_PROFILE: ci | ||
| ETH_TESTNET_TYPE: pos | ||
| SP1_PROVER: network | ||
| ETHEREUM_POS_NETWORK_PRESET: minimal | ||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| generate-matrix: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| matrix: ${{ steps.get-matrix.outputs.matrix }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Generate Test Matrix | ||
| id: get-matrix | ||
| uses: ./.github/actions/e2e-test-matrix | ||
| with: | ||
| skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16' | ||
|
|
||
| e2e: | ||
| needs: generate-matrix | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }} | ||
| name: ${{ matrix.test }} | ||
| runs-on: depot-ubuntu-22.04 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: "Set up E2E environment" | ||
| uses: ./.github/actions/setup-e2e | ||
| with: | ||
| github_token: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
||
| - name: Run Tests | ||
| env: | ||
| NETWORK_PRIVATE_KEY: ${{ secrets.SP1_PRIVATE_KEY }} | ||
| run: | | ||
| cd e2e/interchaintestv8 | ||
| go test -v -mod=readonly . -run '^${{ matrix.test }}$' -timeout 60m |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| name: e2e-quick | ||
| on: | ||
|
There was a problem hiding this comment. should this also run on push to main? Or any of the others such as minimal There was a problem hiding this comment. Hmm, yeah, I guess it could. Which one would you prefer? |
||
| workflow_dispatch: | ||
| pull_request: | ||
| paths: | ||
| - '**.rs' | ||
| - '**.go' | ||
| - '**.toml' | ||
| - '**.lock' | ||
| - '**.mod' | ||
| - '**.sum' | ||
| - '**.sol' | ||
| - '.github/workflows/quick-e2e.yml' | ||
| - 'bun.lockb' | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| FOUNDRY_PROFILE: ci | ||
| ETH_TESTNET_TYPE: pow | ||
| SP1_PROVER: mock | ||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| golangci: | ||
| name: lint | ||
| runs-on: depot-ubuntu-22.04-4 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: actions/setup-go@v4 | ||
| with: | ||
| go-version: "1.23" | ||
| cache-dependency-path: e2e/interchaintestv8/go.sum | ||
|
|
||
| - name: golangci-lint | ||
| uses: golangci/golangci-lint-action@v6 | ||
| with: | ||
| version: v1.63 | ||
| working-directory: e2e/interchaintestv8 | ||
|
|
||
| generate-matrix: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| matrix: ${{ steps.get-matrix.outputs.matrix }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Generate Test Matrix | ||
| id: get-matrix | ||
| uses: ./.github/actions/e2e-test-matrix | ||
| with: | ||
| skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos' | ||
|
|
||
| e2e: | ||
| needs: generate-matrix | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }} | ||
| name: ${{ matrix.test }} | ||
| runs-on: depot-ubuntu-22.04 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: "Set up E2E environment" | ||
| uses: ./.github/actions/setup-e2e | ||
| with: | ||
| github_token: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
||
| - name: Run Tests | ||
| env: | ||
| NETWORK_PRIVATE_KEY: ${{ secrets.SP1_PRIVATE_KEY }} | ||
| run: | | ||
| cd e2e/interchaintestv8 | ||
| go test -v -mod=readonly . -run '^${{ matrix.test }}$' -timeout 40m | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this in a separate directory called
e2e-test-matrix/? Each action is to have its own dir? If you want to separate these into dirs, just use.github/actions/e2e/imoThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just how github actions are structured. There is one folder per action with a file called action.yml in it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. I really don't like how this turned out. I almost rather prefer we keep what we have now (
e2e.yml) and rename it toe2e-minimal.yml(which supports both pow mock and minimal pos like today). And just add a newe2e-full.ymlwith full code duplication frome2e-minimal.ymlbut only it runs on pos + mainnet. (And not on every PR). Wdyt?I'm fine with you moving the setup action though. But this is super confusing now as we have 3 directories and two of them have different types of setup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather not duplicate this. Would naming this differently help? Like
install-e2e-deps? It seems like this should be solvable with naming. Got any ideas?