-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: create e2e smoke test flow + use cluster for SP1 proofs #405
base: main
Are you sure you want to change the base?
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #405 +/- ##
=======================================
Coverage 99.86% 99.86%
=======================================
Files 14 14
Lines 744 744
=======================================
Hits 743 743
Misses 1 1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Why do we have a separate smoke test ci pipeline? What's the use case? |
The idea is to have a smaller set of tests that always run in POS mode and that can also be run with the mainnet preset. This set can run a small set of POS tests on PRs (maybe only when changes are made to test files or the light client?) and every week run a couple of tests with the mainnet preset. I did consider putting this into the e2e workflow itself, but it seemed unnecessarily complex to try to make this fit into the existing workflow. |
@gjermundgaraba I think we should do a full decoupling so that we have 3 sets of separate CI:
These run all the same tests. And we can optimize in which frequency these run later. Maybe initially:
We can also perhaps add randomness as well in the future. The reason why I think this will be better is because even one full-e2e is enough to have the CI take a longer time. Since the runtime of the CI is the duration of the longest test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This got a bit bigger than expected, but got in a good cleanup here, I think!
@@ -0,0 +1,21 @@ | |||
name: 'E2E Test Matrix Generator' | |||
description: 'Dynamically discovers and generates a test matrix for e2e tests' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To avoid having to specify every test every time.
Instead there is now a skip-list, which will be much shorter and easier to maintain
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tbh I actually do like listing. The amount of code added to create the list far exceeds the size of the list by several times, and is significantly harder to read. Likely contains weird edge cases. Why would it be bad to have them listed only in one file such as this one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly because it's hard to know if any are missing and why. It's easy to miss if a test was not added to the list, and it seems like an unnecessary maintenance burden to keep around.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now we will never know if a test is missing. I think we can make a practice to add every test we write to the CI lisr, and just have the ones we don't use commented out with a comment explaining why. I don't trust these scripts to not miss some tests eventually. Idk, wdyt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can do that of course, but I'd rather have less things to worry about than more - hence the want to reduce the maintenance burden. It's not a big deal, but I do prefer this method which we also use in ibc-go (although there it's a go script). Would you feel more comfortable if it was a go script instead of a bash script?
One possibility is to take the go script from ibc-go, pull it out into a separate repo, clean it up and publish it as an action we can use in both repos (and maybe others could use too)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did my initial pass
@@ -0,0 +1,21 @@ | |||
name: 'E2E Test Matrix Generator' | |||
description: 'Dynamically discovers and generates a test matrix for e2e tests' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tbh I actually do like listing. The amount of code added to create the list far exceeds the size of the list by several times, and is significantly harder to read. Likely contains weird edge cases. Why would it be bad to have them listed only in one file such as this one?
id: get-matrix | ||
uses: ./.github/actions/e2e-test-matrix | ||
with: | ||
skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can these be listed properly rather than be comma delimited? Like:
skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos' | |
skip-tests: | |
- 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16' | |
- 'TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really. There is no list type for input, so it would have to be either comma separated, or like a json array, but again, as a string '["test1", "test2"]'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oof
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this in a separate directory called e2e-test-matrix/
? Each action is to have its own dir? If you want to separate these into dirs, just use .github/actions/e2e/
imo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just how github actions are structured. There is one folder per action with a file called action.yml in it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. I really don't like how this turned out. I almost rather prefer we keep what we have now (e2e.yml
) and rename it to e2e-minimal.yml
(which supports both pow mock and minimal pos like today). And just add a new e2e-full.yml
with full code duplication from e2e-minimal.yml
but only it runs on pos + mainnet. (And not on every PR). Wdyt?
I'm fine with you moving the setup action though. But this is super confusing now as we have 3 directories and two of them have different types of setup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather not duplicate this. Would naming this differently help? Like install-e2e-deps
? It seems like this should be solvable with naming. Got any ideas?
/// The optional private key for the network prover. | ||
/// Only used when `prover_type` is "network". | ||
/// `NETWORK_PRIVATE_KEY` environment variable is used if not provided. | ||
#[serde(default, skip_serializing_if = "Option::is_none")] | ||
pub network_private_key: Option<String>, | ||
/// The optional RPC URL for the network prover. | ||
/// Only used when `prover_type` is "network". | ||
/// `NETWORK_RPC_URL` environment variable is used if not provided. | ||
#[serde(default, skip_serializing_if = "Option::is_none")] | ||
pub network_rpc_url: Option<String>, | ||
/// Whether to use a private cluster. | ||
/// Only used when `prover_type` is "network". | ||
#[serde(default, skip_serializing_if = "is_default")] | ||
pub private_cluster: bool, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fact that these are ignored when prover type is not network is not enforced by the type system, so I don't really like this pattern even though it might technically be simpler. I'd rather keep what we have and use serde annotations such as untagged
(or something like that) on Network
. This would basically create a similar user experience while still keeping the previous type system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The problem with the previous type system was that it created a weird setup where the sp1prover could be either a string or an object, which both looked strange and was much harder to work around when using.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm fine with this as long as you can reintroduce type safety (i.e. no entered arguments will be ignored.) And ideally, you should do this without using std::env to set env variables. I believe anything you want should be possible with something like the previous setup but more serde annotations
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think no argument is ignored now? And not sure what you mean with std::env?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems shorter to just list the tests we want.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not about shorter, it's about getting to not have to worry about listing tests and maintaining that. We could easily end up in a situation where one is forgotten at some point and never gets into the CI pipeline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just don't wanna maintain this script (or even understand it). It is easy to understand a list of tests.
Then you still need to list the test you don't want to be running. I think we shouldn't introduce bash unless it can be inlined in yaml files.
I'm willing to accept this for now though since I don't think listing is a better way, it just looks much cleaner. The CI is starting to look closer to ibc-go
's, which is not necessarily good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another option for the tests we don't want to run is to either not have those, or have the skip part of them in code (that they skip on certain conditions, like CI or whatever).
This reverts commit dc3c72d.
Description
closes: #402
Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.
godoc
comments.Files changed
in the GitHub PR explorer.SonarCloud Report
in the comment section below once CI passes.