[feature] Verbosity/output conf options for conan.tools.cmake.CMake.ctest(...)

[puetzk]

What is your suggestion?

Since conan 2.0.15 did did get a new conan.tools.cmake.CMake.ctest helper function in #15282 that no longer goes through make test, I'd like to re-ask for a couple of old feature requests that still seem (to me) like good ideas but were rejected on the (reasonable at the time) basis that the old conans.CMake.test, which used make test, didn't have a way to do them.

The first is something I asked for #12407: a [conf]tools.cmake.ctest:verbosity=quiet|progress|default|failure|verbose|extra that could be set in global.conf (or a profile), to have CI print more output than just the overall pass/fail. These would map straightforwardly to the --quiet --progress, --output-on-failure, --verbose, --extra-verbose switches ctest already has (default would just pass no flags): https://cmake.org/cmake/help/latest/manual/ctest.1.html. You could already (now) pass these with cli_args=, but it would be nice to do so globally instead of modifying recipes (since these wouldn't affect the resulting binaries if the tests do pass).

Similarly, it would be nice to have [conf]tools.cmake.ctest:junit=<path> that would add --output-junit=<path>, so that one could injest the detailed test report for display in UI like github checks or jenkins https://plugins.jenkins.io/junit/

Of course, since these are already things you can do with cli_args, so it would also be viable to just have [conf]tools.cmake.ctest:args or similar that was appended to any cli_args from the recipe. But I think things compose a bit nicer with separate conf settings.

I'd also suggest that the helper should (unconditionally) run ctest with --interactive-debug-mode 0, so you don't ever get hung in the middle of conan create or similar with a WER prompt.

Alternatively, either/both of these verbosity/output conf settings can map into fields of the https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html#test-preset and the helper could pass in a preset, but that would couple the ctest helper to the CMakeToolchain generator and a version of ctest new enough to support presets. And apply the configuration even in a local development/IDE flow. So I think it makes more sense to keep this to just the recipe ctest helper and cli_args.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @puetzk

Thanks for your feedback.

The first is something I asked for #12407: a [conf]tools.cmake.ctest:verbosity=quiet|progress|default|failure|verbose|extra that could be set in global.conf (or a profile), to have CI print more output than just the overall pass/fail. These would map straightforwardly to the --quiet --progress, --output-on-failure, --verbose, --extra-verbose switches ctest already has (default would just pass no flags): https://cmake.org/cmake/help/latest/manual/ctest.1.html. You could already (now) pass these with cli_args=, but it would be nice to do so globally instead of modifying recipes (since these wouldn't affect the resulting binaries if the tests do pass).

About this, this is the current implementation of ctest:

    def ctest(self, cli_args=None, env="", stdout=None, stderr=None):
        """
        Equivalent to running ctest ...
        """
        if self._conanfile.conf.get("tools.build:skip_test", check_type=bool):
            return
..
        verbosity = self._conanfile.conf.get("tools.build:verbosity", choices=("quiet", "verbose"))
        if verbosity:
            arg_list.append(f"--{verbosity}")

So it seems it already supports the tools.build:verbosity you can already use to define the verbosity?
So you mean, the problem would be that it only accepts 2 possible choices, quiet and verbose?

Similarly, it would be nice to have [conf]tools.cmake.ctest:junit= that would add --output-junit=, so that one could injest the detailed test report for display in UI like github checks or jenkins https://plugins.jenkins.io/junit/

Of course, since these are already things you can do with cli_args, so it would also be viable to just have [conf]tools.cmake.ctest:args or similar that was appended to any cli_args from the recipe. But I think things compose a bit nicer with separate conf settings.

Yes, we are now adding confs more carefully, because the thing is exploding too much. So we tend to add general confs that can be used for different things within one use case, in this case, it would be way more likely that we add a ctest:args conf instead of a very specific junit one. I think it might be possible to add such generic conf.

I'd also suggest that the helper should (unconditionally) run ctest with --interactive-debug-mode 0, so you don't ever get hung in the middle of conan create or similar with a WER prompt.

Yes, I see --interactive-debug-mode 0 is the default, non-interactive for dashboard mode only.
I agree, lets define it always.

[puetzk]

So it seems it already supports the tools.build:verbosity you can already use to define the verbosity?

Well, to be honest I I had somehow overlooked that ctest already followed tools.build:verbosity. But yes, I rarely want all or nothing; what I personally want to configure is the combination of --progress --output-junit xxx (for batch runs in a CI/IDE environment where there's UI to display junit-formatted results), or --output-on-failure (in a CLI enviornment where the console is the main UI).

I don't have a problem with a combined ctest:args, it just seemed like you were generally trying to be more structured than that with conf settings.

[memsharded]

I don't have a problem with a combined ctest:args, it just seemed like you were generally trying to be more structured than that with conf settings.

Yes, that is true, but now we have to more carefully handle the explosion of confs, having too many can also be overwhelming, noisy and cluttering a bit too much the UX.

[memsharded]

Doing PR #19198 towards this

[puetzk]

Fine. The other thing that would consolidate the list of confgs a lot (but maybe deviate too much from your current approach) is to lean in harder on the use of confs that are dictionaries, and map the dictionary right into the appropriate parts of the CMakePresets.json schema.

E.g. we could have a [conf]tools.cmake.ctest:output that mapped right onto https://cmake.org/cmake/help/v3.20/manual/cmake-presets.7.html#test-preset

so one would e.g. have [conf]tools.cmake.ctest:output*={verbosity:"quiet", "outputOnFailure": True, "outputJUnitFile": "${fileDir}/ctest-junit.xml"}. But then you'd have to rely on either ctest --preset or teach the build helper how to do lots of preset JSON -> CLI argument expansion.

[conf]tools.cmake:ctest_args is certainly less work, and probably plenty good enough :-)

[memsharded]

But then you'd have to rely on either ctest --preset

Yeah, that would be an issue, the presets aren't used yet (support down to Cmake 3.15) for this purpose (calling cmake.ctest() in recipes), so it would need an extra mapping. I will think about it, but maybe it seems too much for the value, thanks for the suggestion!

[memsharded]

Implemented in #19198 for next 2.23