Support permanently skipping tests on a specific system

[Nemo157]

Proposal

Problem statement

Sometimes it is not possible to run a specific test on a system because of environmental reasons. These can be skipped by using cargo test -- --skip some_test_name, but this must be specified every time tests are run.

Motivating examples or use cases

cargo has a test cargo_metadata_non_utf8 that tests it can be used while in a non-utf8 directory. That test is already cfg gated because:

Creating non-utf8 path is an OS-specific pain, so let's run this only on linux, where arbitrary bytes work.

But even this claim of working on linux is specific to some linux filesystems, attempting to run this test when the target-dir is on zfs with utf8only=on fails:

failed to mkdir_p /home/nemo157/sources/cargo/target/tmp/cit/t1946/foo/�/./src: Invalid or incomplete multibyte or wide character (os error 84)

This is a fundamental system restriction which means this test will never work here and should always be skipped.

Solution sketch

Extend libtest's --skip CLI arg with a RUST_TEST_SKIP environment variable, a comma separated list of filters that will be added on to those parsed from the args, allowing it to be permanently set in the working tree via utilities like direnv.

Alternatives

Extend libtest's in-process API to allow this test to mark itself as unsupported when it gets this error code while setting up. I believe this would be a good solution to additionally have, and more useful in this specific case, but I think both have their uses and adding the environment variable is easier to get done first.

Links and related work

rust-lang/rust@master...Nemo157:rust:rust-test-skip-env-var

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[the8472]

That seems to be a lot of effort for a single test.
I think it would be easier to change the test to detect system support and bail out (e.g. by calling the mkdir system utility). This pattern is already used in a bunch of tests.

[Nemo157]

The actual code change to support this is one line in libtest, padded out to support being unstable in the linked implementation. It's not the only situation I've encountered this in, just the most recent one. I dislike having tests say they pass when they've actually not run, seeing the

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in $TIME

is a good reminder that you've got some tests not running on your system that you're relying on CI for.

[cuviper]

I dislike having tests say they pass when they've actually not run,

Maybe we need additional capabilities in test return values? We can already return Result<(), E> for pass/fail, but maybe a bool or dedicated enum in the Ok side could distinguish runtime-ignored tests.

[Nemo157]

Yeah, that was the alternative I put in which I think is the better solution for this particular example. I still feel that the ability to skip tests via an env-var is useful in other situations, is far easier to get stabilized, and is easier to work with while unstable (since it's local config, not a feature that needs to be added to the test code).

[cuviper]

I do think an environment variable is reasonable anyway -- there's precedent in other test options like RUST_TEST_THREADS and RUST_TEST_NOCAPTURE.

[the8472]

Citing RUST_TEST_THREADS as an example makes me wary because it has been misused / misunderstood.

[joshtriplett]

I think we should hand this over to the folks currently working on improving libtest and the test infrastructure. Cc @rust-lang/testing-devex.

FWIW, I think we should have some way of marking a test as explicitly "unsupported" or similar, in a fashion that gets counted and listed as such by cargo test. It should be possible to do that both in a fashion that skips compiling the function, as well as in cases where the function gets compiled and runs and returns a special kind of error/panic. (It's possible that those should be two different cases, counted separately.)

[epage]

This is a very understandable problem. We have helpers in Cargo to deal with this and I dislike how limited those helpers are within cargo and the fact that they are limited to cargo. Frustrated with that experience, I originally approached T-libs-api with proposed extensions to libtest to solve that and other problems and the tentative decision we came up with is to feature-freeze libtest and focus on first-class custom test harness support where these things could be more naturally exposed. You can find my notes on this at my blog.

T-testing-devex is working incrementally and has not officially adopted that full plan yet. RUST_TEST_SKIP might work as a short-term solution but as one of the steps towards that long term goal, I've been considering deprecating all existing environment variables, see rust-lang/testing-devex-team#10.

Yes, my blog post was from a while ago. I had to finish up other priorities before rolling over to this work. I had hoped we could recruit others to help in the mean time but it didn't work out. It is now my primary focus (when not distracted...). If someone would want to help with that effort, it'd be much appreciated!

[joshtriplett]

@epage I definitely think an environment variable is not the path I'd want to go here. I'd be much more interested in something like a magic type that you can panic with to indicate "not supported" (and then a macro that panics with that type), and some kind of attribute annotation that says "skip this but leave a stub that panics with that type" (or with a similar type if we want to distinguish the two).

I agree that this would need to wait on a custom test harness.