test: create e2e smoke test flow + use cluster for SP1 proofs #405

gjermundgaraba · 2025-03-13T20:22:30Z

Description

closes: #402

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

Linked to GitHub issue with discussion and accepted design, OR link to spec that describes this work.
Wrote unit and integration tests.
Added relevant natspec and godoc comments.
Provide a conventional commit message to follow the repository standards.
Re-reviewed Files changed in the GitHub PR explorer.
Review SonarCloud Report in the comment section below once CI passes.

codecov · 2025-03-13T20:30:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.86%. Comparing base (de19ea5) to head (139da65).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #405   +/-   ##
=======================================
  Coverage   99.86%   99.86%           
=======================================
  Files          15       15           
  Lines         744      744           
=======================================
  Hits          743      743           
  Misses          1        1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

srdtrk · 2025-03-14T07:00:19Z

Why do we have a separate smoke test ci pipeline? What's the use case?

gjermundgaraba · 2025-03-14T09:53:12Z

The idea is to have a smaller set of tests that always run in POS mode and that can also be run with the mainnet preset. This set can run a small set of POS tests on PRs (maybe only when changes are made to test files or the light client?) and every week run a couple of tests with the mainnet preset.

I did consider putting this into the e2e workflow itself, but it seemed unnecessarily complex to try to make this fit into the existing workflow.

srdtrk · 2025-03-14T14:07:27Z

@gjermundgaraba I think we should do a full decoupling so that we have 3 sets of separate CI:

quick-e2e.yaml: POW and mock
minimal-e2e.yaml: POS and minimal
full-e2e.yaml: POS and mainnet

These run all the same tests. And we can optimize in which frequency these run later. Maybe initially:

every PR
every 1-3 days
every 5-7 days

We can also perhaps add randomness as well in the future. The reason why I think this will be better is because even one full-e2e is enough to have the CI take a longer time. Since the runtime of the CI is the duration of the longest test.

…rovements

gjermundgaraba

This got a bit bigger than expected, but got in a good cleanup here, I think!

e2e/interchaintestv8/ibc_eureka_test.go

gjermundgaraba · 2025-03-16T23:00:35Z

.github/actions/e2e-test-matrix/action.yml

@@ -0,0 +1,21 @@
+name: 'E2E Test Matrix Generator'
+description: 'Dynamically discovers and generates a test matrix for e2e tests'


To avoid having to specify every test every time.
Instead there is now a skip-list, which will be much shorter and easier to maintain

Tbh I actually do like listing. The amount of code added to create the list far exceeds the size of the list by several times, and is significantly harder to read. Likely contains weird edge cases. Why would it be bad to have them listed only in one file such as this one?

Mostly because it's hard to know if any are missing and why. It's easy to miss if a test was not added to the list, and it seems like an unnecessary maintenance burden to keep around.

Now we will never know if a test is missing. I think we can make a practice to add every test we write to the CI lisr, and just have the ones we don't use commented out with a comment explaining why. I don't trust these scripts to not miss some tests eventually. Idk, wdyt?

We can do that of course, but I'd rather have less things to worry about than more - hence the want to reduce the maintenance burden. It's not a big deal, but I do prefer this method which we also use in ibc-go (although there it's a go script). Would you feel more comfortable if it was a go script instead of a bash script?

One possibility is to take the go script from ibc-go, pull it out into a separate repo, clean it up and publish it as an action we can use in both repos (and maybe others could use too)?

.github/actions/setup/action.yml

e2e/interchaintestv8/relayer/config.go

e2e/interchaintestv8/relayer/relayer.go

programs/relayer/Cargo.toml

programs/relayer/src/core/builder.rs

programs/relayer/src/modules/cosmos_to_cosmos.rs

programs/relayer/src/modules/cosmos_to_eth.rs

scripts/generate-e2e-test-matrix.sh

srdtrk

Did my initial pass

srdtrk · 2025-03-17T13:30:08Z

.github/actions/e2e-test-matrix/action.yml

@@ -0,0 +1,21 @@
+name: 'E2E Test Matrix Generator'
+description: 'Dynamically discovers and generates a test matrix for e2e tests'


Tbh I actually do like listing. The amount of code added to create the list far exceeds the size of the list by several times, and is significantly harder to read. Likely contains weird edge cases. Why would it be bad to have them listed only in one file such as this one?

.github/workflows/docker.yml

srdtrk · 2025-03-17T13:35:42Z

.github/workflows/e2e-full.yml

+        id: get-matrix
+        uses: ./.github/actions/e2e-test-matrix
+        with:
+          skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos'


Can these be listed properly rather than be comma delimited? Like:

Suggested change

skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos'

skip-tests:

- 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16'

- 'TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos'

Not really. There is no list type for input, so it would have to be either comma separated, or like a json array, but again, as a string '["test1", "test2"]'

I think we can keep this for now since you don't think the bash script will live forever. When we move this logic to go fully, perhaps we should run all tests, and some tests just skip themselves based on environment variables or other runtime factors. This would also allow us to implement a 50% run chance per test very easily

srdtrk · 2025-03-17T13:41:13Z

.github/actions/e2e-test-matrix/action.yml

Why is this in a separate directory called e2e-test-matrix/? Each action is to have its own dir? If you want to separate these into dirs, just use .github/actions/e2e/ imo

This is just how github actions are structured. There is one folder per action with a file called action.yml in it.

I see. I really don't like how this turned out. I almost rather prefer we keep what we have now (e2e.yml) and rename it to e2e-minimal.yml (which supports both pow mock and minimal pos like today). And just add a new e2e-full.yml with full code duplication from e2e-minimal.yml but only it runs on pos + mainnet. (And not on every PR). Wdyt?

I'm fine with you moving the setup action though. But this is super confusing now as we have 3 directories and two of them have different types of setup

I'd rather not duplicate this. Would naming this differently help? Like install-e2e-deps? It seems like this should be solvable with naming. Got any ideas?

Directory should probably be called e2e-setup/ so that we have the following directories at the end:

foundry-setup/

e2e-setup/

e2e-matrix/

.github/actions/setup-e2e/action.yml

programs/relayer/src/core/builder.rs

programs/relayer/src/modules/cosmos_to_eth.rs

srdtrk · 2025-03-17T14:43:55Z

scripts/generate-e2e-test-matrix.sh

Seems shorter to just list the tests we want.

It's not about shorter, it's about getting to not have to worry about listing tests and maintaining that. We could easily end up in a situation where one is forgotten at some point and never gets into the CI pipeline.

I just don't wanna maintain this script (or even understand it). It is easy to understand a list of tests.
Then you still need to list the test you don't want to be running. I think we shouldn't introduce bash unless it can be inlined in yaml files.

I'm willing to accept this for now though since I don't think listing is a better way, it just looks much cleaner. The CI is starting to look closer to ibc-go's, which is not necessarily good.

Another option for the tests we don't want to run is to either not have those, or have the skip part of them in code (that they skip on certain conditions, like CI or whatever).

Yes, I want to have skip part in code based on env variables. However, I understand that this might be tough to implement in this PR. I am ok with merging this and addressing it in another PR

…rovements

This reverts commit dc3c72d.

gjermundgaraba · 2025-03-18T22:22:16Z

Added type safety to the sp1 config!

srdtrk

Looking very good. I just have some style suggestions left, we can merge this once we agree on those style decisions

srdtrk · 2025-03-19T15:03:39Z

programs/relayer/src/modules/cosmos_to_eth.rs

The new config looks really good. You should update the example config as well

srdtrk · 2025-03-19T15:07:33Z

.github/workflows/e2e-full.yml

+    runs-on: depot-ubuntu-22.04
+    steps:
+      - uses: actions/checkout@v4
+      - name: "Set up E2E environment"


Some name use quotations and some names don't. I think we should just not use quotations but I'm ok with either for names as long as we are consistent. Ditto everywhere else.

I'll go through and clean up any remaining ones

srdtrk · 2025-03-19T15:09:33Z

.github/workflows/e2e-full.yml

+        id: get-matrix
+        uses: ./.github/actions/e2e-test-matrix
+        with:
+          skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos'


I think we can keep this for now since you don't think the bash script will live forever. When we move this logic to go fully, perhaps we should run all tests, and some tests just skip themselves based on environment variables or other runtime factors. This would also allow us to implement a 50% run chance per test very easily

srdtrk · 2025-03-19T15:11:48Z

.github/workflows/e2e-pos-minimal.yml

I think let's just do e2e-minimal.yml, and we can all the quick one e2e-mock.yml. I like them to be two words. Wdyt:

e2e-full.yml

e2e-minimal.yml

e2e-mock.yml

.github/actions/setup/action.yml

srdtrk · 2025-03-19T15:14:11Z

.github/actions/e2e-test-matrix/action.yml

Directory should probably be called e2e-setup/ so that we have the following directories at the end:

foundry-setup/

e2e-setup/

e2e-matrix/

.github/actions/setup-e2e/action.yml

srdtrk · 2025-03-19T15:16:48Z

scripts/generate-e2e-test-matrix.sh

Yes, I want to have skip part in code based on env variables. However, I understand that this might be tough to implement in this PR. I am ok with merging this and addressing it in another PR

programs/relayer/src/modules/cosmos_to_eth.rs

Co-authored-by: srdtrk <[email protected]>

srdtrk

Lgtm! 🚀

test: create e2e smoke test flow + use cluster for SP1 proofs

c0e045b

gjermundgaraba requested a review from srdtrk as a code owner March 13, 2025 20:22

gjermundgaraba added 2 commits March 13, 2025 21:57

lint

fbf87d5

Merge branch 'main' into gjermund/e2e-test-improvements

6301d74

gjermundgaraba added 12 commits March 14, 2025 16:48

initial setup 3 e2e workflows

fb38ef2

pull out e2e setup

7726fd9

test matrix

f095c40

cleanup

242c96d

ci cleanup

d43edb9

fix newlines

8cf10aa

rename to yaml

f21248a

Merge remote-tracking branch 'origin/main' into gjermund/e2e-test-imp…

8544375

…rovements

fix checkout

f5a0214

single-line matrix output

c4fe1fa

pass github pat to e2e setup

dc39a2d

fix matrix script

b0f88d8

gjermundgaraba marked this pull request as draft March 15, 2025 16:52

gjermundgaraba added 9 commits March 15, 2025 17:53

trying a different approach to the config

b4020b1

lint

bffc470

add missing checkout

88b59e2

fix skip tests

3b76a82

more cleanup

1019cde

fix broken test + stack trace support in relayer

44efbe3

Merge remote-tracking branch 'origin/main' into gjermund/e2e-test-imp…

62a896a

…rovements

some more cleanups

b3d1550

remove unused type

b098839

cleanup after self-review

29d7d90

gjermundgaraba commented Mar 16, 2025

View reviewed changes

gjermundgaraba marked this pull request as ready for review March 16, 2025 23:08

ups

e26d03a

srdtrk reviewed Mar 17, 2025

View reviewed changes

gjermundgaraba added 9 commits March 17, 2025 15:56

removed unecessary quotes

571b89c

code review fixes

883a9a2

Merge remote-tracking branch 'origin/main' into gjermund/e2e-test-imp…

a887d2c

…rovements

push to main run minimal

1cdd6b4

Merge remote-tracking branch 'origin/main' into gjermund/e2e-test-imp…

110a6b1

…rovements

test different tm_client settings

dc3c72d

Merge remote-tracking branch 'origin/main' into gjermund/e2e-test-imp…

dccb743

…rovements

Revert "test different tm_client settings"

0a7260a

This reverts commit dc3c72d.

type safety in sp1 config

f8e1c4e

gjermundgaraba requested a review from srdtrk March 18, 2025 22:21

srdtrk reviewed Mar 19, 2025

View reviewed changes

programs/relayer/src/modules/cosmos_to_eth.rs Outdated Show resolved Hide resolved

programs/relayer/src/modules/cosmos_to_eth.rs Outdated Show resolved Hide resolved

programs/relayer/src/modules/cosmos_to_eth.rs Outdated Show resolved Hide resolved

gjermundgaraba mentioned this pull request Mar 19, 2025

Replace e2e test matrix bash script with go-based action #427

Open

gjermundgaraba and others added 7 commits March 19, 2025 16:57

consistent use of quotes in ci files

cbd441b

fixed paths

db4329f

fixed a bunch of issues

bb24820

fix 2

7ef1698

code review fixes

d3e75b7

Apply suggestions from code review

3638e65

Co-authored-by: srdtrk <[email protected]>

Merge branch 'main' into gjermund/e2e-test-improvements

139da65

srdtrk approved these changes Mar 19, 2025

View reviewed changes

gjermundgaraba merged commit e981ff2 into main Mar 19, 2025
64 checks passed

gjermundgaraba deleted the gjermund/e2e-test-improvements branch March 19, 2025 17:44

		@@ -0,0 +1,21 @@
		name: 'E2E Test Matrix Generator'
		description: 'Dynamically discovers and generates a test matrix for e2e tests'

test: create e2e smoke test flow + use cluster for SP1 proofs #405

test: create e2e smoke test flow + use cluster for SP1 proofs #405

Conversation

gjermundgaraba commented Mar 13, 2025 • edited Loading

Description

codecov bot commented Mar 13, 2025 • edited Loading

Codecov Report

srdtrk commented Mar 14, 2025

gjermundgaraba commented Mar 14, 2025

srdtrk commented Mar 14, 2025 • edited Loading

gjermundgaraba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srdtrk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srdtrk Mar 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gjermundgaraba commented Mar 18, 2025

srdtrk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srdtrk left a comment

Choose a reason for hiding this comment

gjermundgaraba commented Mar 13, 2025 •

edited

Loading

codecov bot commented Mar 13, 2025 •

edited

Loading

srdtrk commented Mar 14, 2025 •

edited

Loading

srdtrk Mar 17, 2025 •

edited

Loading