-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: create e2e smoke test flow + use cluster for SP1 proofs #405
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #405 +/- ##
=======================================
Coverage 99.86% 99.86%
=======================================
Files 15 15
Lines 744 744
=======================================
Hits 743 743
Misses 1 1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Why do we have a separate smoke test ci pipeline? What's the use case? |
The idea is to have a smaller set of tests that always run in POS mode and that can also be run with the mainnet preset. This set can run a small set of POS tests on PRs (maybe only when changes are made to test files or the light client?) and every week run a couple of tests with the mainnet preset. I did consider putting this into the e2e workflow itself, but it seemed unnecessarily complex to try to make this fit into the existing workflow. |
@gjermundgaraba I think we should do a full decoupling so that we have 3 sets of separate CI:
These run all the same tests. And we can optimize in which frequency these run later. Maybe initially:
We can also perhaps add randomness as well in the future. The reason why I think this will be better is because even one full-e2e is enough to have the CI take a longer time. Since the runtime of the CI is the duration of the longest test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This got a bit bigger than expected, but got in a good cleanup here, I think!
@@ -0,0 +1,21 @@ | |||
name: 'E2E Test Matrix Generator' | |||
description: 'Dynamically discovers and generates a test matrix for e2e tests' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To avoid having to specify every test every time.
Instead there is now a skip-list, which will be much shorter and easier to maintain
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tbh I actually do like listing. The amount of code added to create the list far exceeds the size of the list by several times, and is significantly harder to read. Likely contains weird edge cases. Why would it be bad to have them listed only in one file such as this one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly because it's hard to know if any are missing and why. It's easy to miss if a test was not added to the list, and it seems like an unnecessary maintenance burden to keep around.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now we will never know if a test is missing. I think we can make a practice to add every test we write to the CI lisr, and just have the ones we don't use commented out with a comment explaining why. I don't trust these scripts to not miss some tests eventually. Idk, wdyt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can do that of course, but I'd rather have less things to worry about than more - hence the want to reduce the maintenance burden. It's not a big deal, but I do prefer this method which we also use in ibc-go (although there it's a go script). Would you feel more comfortable if it was a go script instead of a bash script?
One possibility is to take the go script from ibc-go, pull it out into a separate repo, clean it up and publish it as an action we can use in both repos (and maybe others could use too)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did my initial pass
@@ -0,0 +1,21 @@ | |||
name: 'E2E Test Matrix Generator' | |||
description: 'Dynamically discovers and generates a test matrix for e2e tests' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tbh I actually do like listing. The amount of code added to create the list far exceeds the size of the list by several times, and is significantly harder to read. Likely contains weird edge cases. Why would it be bad to have them listed only in one file such as this one?
.github/workflows/e2e-full.yml
Outdated
id: get-matrix | ||
uses: ./.github/actions/e2e-test-matrix | ||
with: | ||
skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can these be listed properly rather than be comma delimited? Like:
skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos' | |
skip-tests: | |
- 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16' | |
- 'TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really. There is no list type for input, so it would have to be either comma separated, or like a json array, but again, as a string '["test1", "test2"]'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oof
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can keep this for now since you don't think the bash script will live forever. When we move this logic to go fully, perhaps we should run all tests, and some tests just skip themselves based on environment variables or other runtime factors. This would also allow us to implement a 50% run chance per test very easily
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this in a separate directory called e2e-test-matrix/
? Each action is to have its own dir? If you want to separate these into dirs, just use .github/actions/e2e/
imo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just how github actions are structured. There is one folder per action with a file called action.yml in it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. I really don't like how this turned out. I almost rather prefer we keep what we have now (e2e.yml
) and rename it to e2e-minimal.yml
(which supports both pow mock and minimal pos like today). And just add a new e2e-full.yml
with full code duplication from e2e-minimal.yml
but only it runs on pos + mainnet. (And not on every PR). Wdyt?
I'm fine with you moving the setup action though. But this is super confusing now as we have 3 directories and two of them have different types of setup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather not duplicate this. Would naming this differently help? Like install-e2e-deps
? It seems like this should be solvable with naming. Got any ideas?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Directory should probably be called e2e-setup/
so that we have the following directories at the end:
foundry-setup/
e2e-setup/
e2e-matrix/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems shorter to just list the tests we want.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not about shorter, it's about getting to not have to worry about listing tests and maintaining that. We could easily end up in a situation where one is forgotten at some point and never gets into the CI pipeline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just don't wanna maintain this script (or even understand it). It is easy to understand a list of tests.
Then you still need to list the test you don't want to be running. I think we shouldn't introduce bash unless it can be inlined in yaml files.
I'm willing to accept this for now though since I don't think listing is a better way, it just looks much cleaner. The CI is starting to look closer to ibc-go
's, which is not necessarily good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another option for the tests we don't want to run is to either not have those, or have the skip part of them in code (that they skip on certain conditions, like CI or whatever).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I want to have skip part in code based on env variables. However, I understand that this might be tough to implement in this PR. I am ok with merging this and addressing it in another PR
This reverts commit dc3c72d.
Added type safety to the sp1 config! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking very good. I just have some style suggestions left, we can merge this once we agree on those style decisions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The new config looks really good. You should update the example config as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done!
.github/workflows/e2e-full.yml
Outdated
runs-on: depot-ubuntu-22.04 | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: "Set up E2E environment" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some name use quotations and some names don't. I think we should just not use quotations but I'm ok with either for names as long as we are consistent. Ditto everywhere else.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll go through and clean up any remaining ones
.github/workflows/e2e-full.yml
Outdated
id: get-matrix | ||
uses: ./.github/actions/e2e-test-matrix | ||
with: | ||
skip-tests: 'TestWithRelayerTestSuite/Test_2_ConcurrentRecvPacketToEth_Groth16,TestWithRelayerTestSuite/TestMultiPeriodClientUpdateToCosmos' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can keep this for now since you don't think the bash script will live forever. When we move this logic to go fully, perhaps we should run all tests, and some tests just skip themselves based on environment variables or other runtime factors. This would also allow us to implement a 50% run chance per test very easily
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think let's just do e2e-minimal.yml
, and we can all the quick one e2e-mock.yml
. I like them to be two words. Wdyt:
e2e-full.yml
e2e-minimal.yml
e2e-mock.yml
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Directory should probably be called e2e-setup/
so that we have the following directories at the end:
foundry-setup/
e2e-setup/
e2e-matrix/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I want to have skip part in code based on env variables. However, I understand that this might be tough to implement in this PR. I am ok with merging this and addressing it in another PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lgtm! 🚀
Description
closes: #402
Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.
godoc
comments.Files changed
in the GitHub PR explorer.SonarCloud Report
in the comment section below once CI passes.