Nutest discovers tests by scanning matching files in the path, sourcing that code and collecting test annotations on methods via scope commands. The file patterns detected are test_*.nu, test-*.nu, *_test.nu and *-test.nu to match the most common styles and file groupings.
Each test file, which include multiple tests (a suite) is dispatched to run on a single Nu subshell.
Test results are captured using encoded events written to stdout (one event per line), which carry all the required context indicating what suite and test it is associated with, allowing tests to be run in parallel.
Success and failure events are implied by an error being thrown, such as an assertion failure.
Output from tests is captured by aliasing the print command. This may include structured data, which is preserved in the event data. This is achieved by converting each item sent to a print command as nuon and then encoding as base64, which ensures a single line event is emitted even if it contains multi-line text.
All events for all suites and tests being run are then collated, ready to use to present to the user with flexible output and rendering.
Nutest runs both test suites (a file of tests) and each test in parallel with minimal Nu subshells.
There are two levels of concurrency used in Nutest, leveraging par-each, where the following are run concurrently:
- Suites (file of tests).
- Tests within a suite.
This means that an 8-core CPU would run 8 suites concurrently and within each suite, it would run 8 tests in concurrently. This might suggest Nutest potentially causing excessive CPU context switching, and the run taking longer than is strictly needed. However, this is not necessarily the case as Nushell leverages Rayon for par-test, which purports to be efficient at managing the number of threads and of scheduling work across available CPU cores. For more on this, see Rayon's notion of potential concurrency, the dynamic nature of it's parallel iterators and the underlying use of Rust's available parallelism. However, it#s still not clear how well this works across multiple processes.
Additionally, given the kinds of use-cases Nushell is used for, many tests are likely to be I/O bound.
Feedback on how well this works in practice is very welcome.
Given Nutest runs as much as possible concurrently, this puts an unusual level of pressure on SQLite that collects test results and the output. For this reason, INSERTs sometimes fail and so a retry mechanism has been added to attempt to insert the data again up to a particular maximum tries at which point Nutest may give up and throw an error. The retries have had some stress testing to come to a pragmatic value, but please let us know if you're seeing issues.