Introduce --test-results-output flag for libtest

[patskovn]

PR: rust-lang/rust#140805

Libtest currently only writes machine-readable output to stdout, respecting passed --format. This is problematic for CI/build-tools integration and due to possible test results corruption caused by other dependencies writing to stdout. We propose adding a new flag --test-results-output <path>, which writes the chosen --format output to the given file.

Prior attempts to address this gap have not landed: e.g. PRs #96290 and #123365 (to make --logfile use --format) were abandoned, and the testing-devex-team Issue #9 (“Export machine-readable test results to a file”) was closed without merging initially proposed changes.

The recent deprecation of the --logfile flag in libtest was a positive move away from ambiguous reporting behavior toward clearer solutions. However, removing --logfile without introducing a replacement for machine-readable file output left issues not fully addressed.

Motivation

Separating test results from other output avoids contamination. If libtest only writes output to stdout, any non-test output (log messages, debug prints, etc.) may corrupt the stream and break parsers. Rust’s println’s are wrapped by libtest, but anything can (and does, in real world) use libc, or have C code using libc that corrupts stdout. There is no possible workaround for the stdout corruption problem.

Also, in practice, projects often resort to external post-processing to filter test output. As one tracking discussion notes, “due to limitations of Rust libtest formatters, Rust developers often use a separate tool to postprocess the test results output”. By writing test results directly to a file, we can guarantee the test results are isolated and parseable, without third-party noise.

Writing results to a file aligns with established patterns: Google Test (GTest) uses a flag like --gtest_output=json:path to produce test reports in a file.

Proposed Solution

We propose introducing a new option, --test-results-output <file>, which directs libtest to write the structured test report to the given file. This flag would be independent of --logfile; it would capture test results in the specified format. Key points:

• Syntax: --test-results-output path/to/results.ext. The path may be relative or absolute; libtest should fail if file already exists.

• Respecting --format: The output format (JSON, JUnit XML, etc.) is controlled by the existing --format flag. libtest would open the file and use the same formatter logic as for stdout

• Usage Example

cargo test -- -Zunstable-options --test-results-output=tests.json --format=json 

• Unstable Feature: This flag can initially be gated (e.g. behind -Zunstable-options) until its behavior stabilizes

• Error Handling: If the file cannot be written (permissions, etc.), libtest should emit an error to stderr and exit. If multiple binaries produce the same file (or the same test command is executed multiple times), it’s up to the caller to avoid collisions by using a unique file name per invocation or cleaning up the file1

• Exclusivity: Unlike #123365 which refactored libtest so that both stdout and logfile results are written we propose to write only to the file if the argument is passed and do not duplicate outputs. Motivation for that is file outputs are mostly expected to be used by build (and test) tools and not by a user that invokes cargo test from cli and alignment with previous decisions of maintainers.

• Backward Compatibility: This change does not break existing users of --format or --logfile. The change will not alter libtest behavior unless it’s explicitly used.

[epage]

The motivation here seems to be focused on intermixed output. That was acknowledged in the proposal to close at #9 (comment)

(2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming --logfile will fit the needs.

This proposal makes several assumptions that would be aided by gathering requirements and learning from prior art, including

• "Exclusivity": should the output only go to one location or be teed?
• File IO is required for this (iirc pytest allows other options for output capturing which is different but related)

I'd recommend digging more into people's requirements and how prior art works and what we can learn from it.