test: create e2e smoke test flow + use cluster for SP1 proofs #405

gjermundgaraba · 2025-03-13T20:22:30Z

Description

closes: #402

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

Linked to GitHub issue with discussion and accepted design, OR link to spec that describes this work.
Wrote unit and integration tests.
Added relevant natspec and godoc comments.
Provide a conventional commit message to follow the repository standards.
Re-reviewed Files changed in the GitHub PR explorer.
Review SonarCloud Report in the comment section below once CI passes.

codecov · 2025-03-13T20:30:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.86%. Comparing base (08653c0) to head (110a6b1).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #405   +/-   ##
=======================================
  Coverage   99.86%   99.86%           
=======================================
  Files          14       14           
  Lines         744      744           
=======================================
  Hits          743      743           
  Misses          1        1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

srdtrk · 2025-03-14T07:00:19Z

Why do we have a separate smoke test ci pipeline? What's the use case?

gjermundgaraba · 2025-03-14T09:53:12Z

The idea is to have a smaller set of tests that always run in POS mode and that can also be run with the mainnet preset. This set can run a small set of POS tests on PRs (maybe only when changes are made to test files or the light client?) and every week run a couple of tests with the mainnet preset.

I did consider putting this into the e2e workflow itself, but it seemed unnecessarily complex to try to make this fit into the existing workflow.

srdtrk · 2025-03-14T14:07:27Z

@gjermundgaraba I think we should do a full decoupling so that we have 3 sets of separate CI:

quick-e2e.yaml: POW and mock
minimal-e2e.yaml: POS and minimal
full-e2e.yaml: POS and mainnet

These run all the same tests. And we can optimize in which frequency these run later. Maybe initially:

every PR
every 1-3 days
every 5-7 days

We can also perhaps add randomness as well in the future. The reason why I think this will be better is because even one full-e2e is enough to have the CI take a longer time. Since the runtime of the CI is the duration of the longest test.

…rovements

gjermundgaraba

This got a bit bigger than expected, but got in a good cleanup here, I think!

e2e/interchaintestv8/ibc_eureka_test.go

gjermundgaraba · 2025-03-16T23:00:35Z

.github/actions/e2e-test-matrix/action.yml

@@ -0,0 +1,21 @@
+name: 'E2E Test Matrix Generator'
+description: 'Dynamically discovers and generates a test matrix for e2e tests'


To avoid having to specify every test every time.
Instead there is now a skip-list, which will be much shorter and easier to maintain

Tbh I actually do like listing. The amount of code added to create the list far exceeds the size of the list by several times, and is significantly harder to read. Likely contains weird edge cases. Why would it be bad to have them listed only in one file such as this one?

Mostly because it's hard to know if any are missing and why. It's easy to miss if a test was not added to the list, and it seems like an unnecessary maintenance burden to keep around.

Now we will never know if a test is missing. I think we can make a practice to add every test we write to the CI lisr, and just have the ones we don't use commented out with a comment explaining why. I don't trust these scripts to not miss some tests eventually. Idk, wdyt?

We can do that of course, but I'd rather have less things to worry about than more - hence the want to reduce the maintenance burden. It's not a big deal, but I do prefer this method which we also use in ibc-go (although there it's a go script). Would you feel more comfortable if it was a go script instead of a bash script?

One possibility is to take the go script from ibc-go, pull it out into a separate repo, clean it up and publish it as an action we can use in both repos (and maybe others could use too)?

.github/actions/setup/action.yml

e2e/interchaintestv8/relayer/config.go

e2e/interchaintestv8/relayer/relayer.go

programs/relayer/Cargo.toml

programs/relayer/src/core/builder.rs

programs/relayer/src/modules/cosmos_to_cosmos.rs

programs/relayer/src/modules/cosmos_to_eth.rs

scripts/generate-e2e-test-matrix.sh

srdtrk

Did my initial pass

srdtrk · 2025-03-17T13:30:08Z

.github/actions/e2e-test-matrix/action.yml

@@ -0,0 +1,21 @@
+name: 'E2E Test Matrix Generator'
+description: 'Dynamically discovers and generates a test matrix for e2e tests'


Tbh I actually do like listing. The amount of code added to create the list far exceeds the size of the list by several times, and is significantly harder to read. Likely contains weird edge cases. Why would it be bad to have them listed only in one file such as this one?

.github/workflows/docker.yml

srdtrk · 2025-03-17T13:35:42Z

.github/workflows/e2e-full.yml

+        id: get-matrix
+        uses: ./.github/actions/e2e-test-matrix
+        with:
+          skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos'


Can these be listed properly rather than be comma delimited? Like:

Suggested change

skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos'

skip-tests:

- 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16'

- 'TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos'

Not really. There is no list type for input, so it would have to be either comma separated, or like a json array, but again, as a string '["test1", "test2"]'

srdtrk · 2025-03-17T13:41:13Z

.github/actions/e2e-test-matrix/action.yml

Why is this in a separate directory called e2e-test-matrix/? Each action is to have its own dir? If you want to separate these into dirs, just use .github/actions/e2e/ imo

This is just how github actions are structured. There is one folder per action with a file called action.yml in it.

I see. I really don't like how this turned out. I almost rather prefer we keep what we have now (e2e.yml) and rename it to e2e-minimal.yml (which supports both pow mock and minimal pos like today). And just add a new e2e-full.yml with full code duplication from e2e-minimal.yml but only it runs on pos + mainnet. (And not on every PR). Wdyt?

I'm fine with you moving the setup action though. But this is super confusing now as we have 3 directories and two of them have different types of setup

I'd rather not duplicate this. Would naming this differently help? Like install-e2e-deps? It seems like this should be solvable with naming. Got any ideas?

.github/actions/setup-e2e/action.yml

programs/relayer/src/core/builder.rs

srdtrk · 2025-03-17T14:40:25Z

programs/relayer/src/modules/cosmos_to_eth.rs

+    /// The optional private key for the network prover.
+    /// Only used when `prover_type` is "network".
+    /// `NETWORK_PRIVATE_KEY` environment variable is used if not provided.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub network_private_key: Option<String>,
+    /// The optional RPC URL for the network prover.
+    /// Only used when `prover_type` is "network".
+    /// `NETWORK_RPC_URL` environment variable is used if not provided.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub network_rpc_url: Option<String>,
+    /// Whether to use a private cluster.
+    /// Only used when `prover_type` is "network".
+    #[serde(default, skip_serializing_if = "is_default")]
+    pub private_cluster: bool,


The fact that these are ignored when prover type is not network is not enforced by the type system, so I don't really like this pattern even though it might technically be simpler. I'd rather keep what we have and use serde annotations such as untagged (or something like that) on Network. This would basically create a similar user experience while still keeping the previous type system.

The problem with the previous type system was that it created a weird setup where the sp1prover could be either a string or an object, which both looked strange and was much harder to work around when using.

I'm fine with this as long as you can reintroduce type safety (i.e. no entered arguments will be ignored.) And ideally, you should do this without using std::env to set env variables. I believe anything you want should be possible with something like the previous setup but more serde annotations

I think no argument is ignored now? And not sure what you mean with std::env?

programs/relayer/src/modules/cosmos_to_eth.rs

srdtrk · 2025-03-17T14:43:55Z

scripts/generate-e2e-test-matrix.sh

Seems shorter to just list the tests we want.

It's not about shorter, it's about getting to not have to worry about listing tests and maintaining that. We could easily end up in a situation where one is forgotten at some point and never gets into the CI pipeline.

I just don't wanna maintain this script (or even understand it). It is easy to understand a list of tests.
Then you still need to list the test you don't want to be running. I think we shouldn't introduce bash unless it can be inlined in yaml files.

I'm willing to accept this for now though since I don't think listing is a better way, it just looks much cleaner. The CI is starting to look closer to ibc-go's, which is not necessarily good.

Another option for the tests we don't want to run is to either not have those, or have the skip part of them in code (that they skip on certain conditions, like CI or whatever).

…rovements

test: create e2e smoke test flow + use cluster for SP1 proofs

c0e045b

gjermundgaraba requested a review from srdtrk as a code owner March 13, 2025 20:22

gjermundgaraba added 2 commits March 13, 2025 21:57

lint

fbf87d5

Merge branch 'main' into gjermund/e2e-test-improvements

6301d74

gjermundgaraba added 12 commits March 14, 2025 16:48

initial setup 3 e2e workflows

fb38ef2

pull out e2e setup

7726fd9

test matrix

f095c40

cleanup

242c96d

ci cleanup

d43edb9

fix newlines

8cf10aa

rename to yaml

f21248a

Merge remote-tracking branch 'origin/main' into gjermund/e2e-test-imp…

8544375

…rovements

fix checkout

f5a0214

single-line matrix output

c4fe1fa

pass github pat to e2e setup

dc39a2d

fix matrix script

b0f88d8

gjermundgaraba marked this pull request as draft March 15, 2025 16:52

gjermundgaraba added 9 commits March 15, 2025 17:53

trying a different approach to the config

b4020b1

lint

bffc470

add missing checkout

88b59e2

fix skip tests

3b76a82

more cleanup

1019cde

fix broken test + stack trace support in relayer

44efbe3

Merge remote-tracking branch 'origin/main' into gjermund/e2e-test-imp…

62a896a

…rovements

some more cleanups

b3d1550

remove unused type

b098839

cleanup after self-review

29d7d90

gjermundgaraba commented Mar 16, 2025

View reviewed changes

gjermundgaraba marked this pull request as ready for review March 16, 2025 23:08

ups

e26d03a

srdtrk reviewed Mar 17, 2025

View reviewed changes

gjermundgaraba added 5 commits March 17, 2025 15:56

removed unecessary quotes

571b89c

code review fixes

883a9a2

Merge remote-tracking branch 'origin/main' into gjermund/e2e-test-imp…

a887d2c

…rovements

push to main run minimal

1cdd6b4

Merge remote-tracking branch 'origin/main' into gjermund/e2e-test-imp…

110a6b1

…rovements

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: create e2e smoke test flow + use cluster for SP1 proofs #405

test: create e2e smoke test flow + use cluster for SP1 proofs #405

gjermundgaraba commented Mar 13, 2025 •

edited

Loading

codecov bot commented Mar 13, 2025 •

edited

Loading

srdtrk commented Mar 14, 2025

gjermundgaraba commented Mar 14, 2025

srdtrk commented Mar 14, 2025 •

edited

Loading

gjermundgaraba left a comment

gjermundgaraba Mar 16, 2025

srdtrk Mar 17, 2025

gjermundgaraba Mar 17, 2025

srdtrk Mar 17, 2025

gjermundgaraba Mar 18, 2025

srdtrk left a comment

srdtrk Mar 17, 2025

srdtrk Mar 17, 2025

gjermundgaraba Mar 17, 2025

srdtrk Mar 17, 2025

srdtrk Mar 17, 2025

gjermundgaraba Mar 17, 2025

srdtrk Mar 17, 2025

gjermundgaraba Mar 18, 2025

srdtrk Mar 17, 2025

gjermundgaraba Mar 17, 2025

srdtrk Mar 17, 2025

gjermundgaraba Mar 18, 2025

srdtrk Mar 17, 2025

gjermundgaraba Mar 17, 2025

srdtrk Mar 17, 2025 •

edited

Loading

gjermundgaraba Mar 18, 2025

		@@ -0,0 +1,21 @@
		name: 'E2E Test Matrix Generator'
		description: 'Dynamically discovers and generates a test matrix for e2e tests'

test: create e2e smoke test flow + use cluster for SP1 proofs #405

Are you sure you want to change the base?

test: create e2e smoke test flow + use cluster for SP1 proofs #405

Conversation

gjermundgaraba commented Mar 13, 2025 • edited Loading

Description

codecov bot commented Mar 13, 2025 • edited Loading

Codecov Report

srdtrk commented Mar 14, 2025

gjermundgaraba commented Mar 14, 2025

srdtrk commented Mar 14, 2025 • edited Loading

gjermundgaraba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srdtrk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srdtrk Mar 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gjermundgaraba commented Mar 13, 2025 •

edited

Loading

codecov bot commented Mar 13, 2025 •

edited

Loading

srdtrk commented Mar 14, 2025 •

edited

Loading

srdtrk Mar 17, 2025 •

edited

Loading