-
Notifications
You must be signed in to change notification settings - Fork 34
WIP: Test CI speedups #321
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Okay, this is an obvious win. The largest benefactor is Dask: we had to limit the |
The last commit bumps the number of examples to 500 for NumPy and PyTorch (up from 200) and to 50 for Dask (up from 5). All runs finish in under 5 minutes, which is I think is a sweet spot for the balance between the convenience of iterating on a PR and a reasonable test coverage. |
Using 4 pytest workers, One potential point of concern can be how parallelization affects the failure mode. It's already not always easy to untangle how hypothesis reports errors and replicate a CI failure locally. I suppose we'll try and see if parallelization makes it any more difficult. |
The warning says it's deprecated.
Okay, I think this is it for low-hanging fruits. The slowest tests on torch and numpy spend their time inside hypothesis, so not much we can do about it. |
testing, do not merge: this checks out array-api-tests from a branch of a fork
The matching array-api-tests PR, data-apis/array-api-tests#373, allows to turn xfails into skips.
Apparently, this has a large effect on run time: in fact the vast majority of time on CI was spent in hypothesis working to distill a failure---for things which we know are failing and have explicitly marked as xfails.