Feature Request: New dotnet tool command for merging results in standalone mode

[chaoticsoftware]

N.B. Alert! :)
I just started looking into adopting coverlet for our code coverage purpose. We run tests across many different modules not only with parallelism within the test job, but also across several different jobs running in parallel in our pipeline. Having to run tests in serial would be a huge (and unacceptable) blow back to our pipeline latency. Aside from that, I've noticed from several threads in here that MergeWith has its issues right now, even when executing tests in parallel. So I have a feature request, that should be fairly easy to implement. In fact, I just prototyped it for myself:

Why not have a dotnet tool command for merging existing json reports into a single final report. That way one can still do a massively parallel testing, and just run the command to merge all the results together. That command seems to be performing well even if results are merged in a single thread, but ultimately if needed, can be further enhanced to work in a multi-threaded mode using concurrent dictionaries and what not to replace the existing single-threaded data structures that are used for representing the results.

So, the pipeline would look like this (an approximation of the actual syntax just to convey the point):

- job: Build
  steps:
     - dotnet build test.sln
- job: Test
  dependsOn: Build
  strategy:
    - parallel: n
  steps:
    - dotnet test test.sln --no-build --collect:"XPlat Code Coverage" -s ./test.runsettings
    - Copy ./**/coverage.json to $(build.artifactStagingDirectory)/cc
       Flatten: true
    - publish $(build.artifactStagingDirectory)/cc
      artifact: CC_$(System.JobPositionInPhase)
- job: CodeCoverage
  dependsOn: Test
  steps:
    - download
       patterns: 'CC_*'
       path: $(pipeline.workspace)/coverageResults
    - dotnet merge-coverage-reports $(pipeline.workspace)/coverageResults/**/coverage.json -o $(pipeline.workspace)/coverageResults/merged -f json,cobertura
    - publishCodeCoverage: $(pipeline.workspace)/coverageResults/merged

Does this sound reasonable?

[petli]

You can solve this now by configuring Coverlet to output e.g. cobertura or opencover files instead of it's own format, and then use https://github.com/danielpalme/ReportGenerator to merge them into a single result (as well as applying additional filtering of the results if desired).

(It has been discussed to add separate merge steps, but I can't find the issue for that now.)

[MarcoRossignoli]

FYI @jakubch1

[chaoticsoftware]

You can solve this now by configuring Coverlet to output e.g. cobertura or opencover files instead of it's own format, and then use https://github.com/danielpalme/ReportGenerator to merge them into a single result (as well as applying additional filtering of the results if desired).

(It has been discussed to add separate merge steps, but I can't find the issue for that now.)

Thanks for the suggestion. I agree. Certainly it is possible to do so. There are a couple of things that I'd say to that, which in my opinion makes the flow less than ideal:

1. I need to get the report generator from marketplace and depending on the organization that could mean trouble going through approval process. Not an engineering problem, but a very real one in practice.

2. You already have the logic to merge the reports, and it seems like an easy next step to do. Wouldn't it be just cleaner and more efficient if you had to use one tool as opposed to two? I mean if dotnet test, for instance, were to provide the same functionality that coverlet provides, but in a built-in fashion, there is little doubt in my mind that engineers would opt to use that built-in functionality instead of adding a dependency to coverlet. Wouldn't you agree? Same can be said about coverlet being self-sufficient as opposed to relying on an external tool to finish its job for it. In the end all the engineer cares about is having their code coverage. All these different steps that take place under the hood are just the bi-product of our toolset limitations IMHO.

[MarcoRossignoli]

It has been discussed to add separate merge steps, but I can't find the issue for that now.)

This was the discussion #683 (comment)

could mean trouble going through approval process.

Report generator is used also by AzDo https://docs.microsoft.com/en-us/azure/devops/release-notes/2019/sprint-150-update#cobertura-code-coverage-report-updates
FYI if you need some proof in your approval process

[jakubch1]

As part of this: microsoft/vstest#1811 effort we will also add extensibility for other data collectors (for example coverlet). Then we also want to onboard coverlet to this logic.

At the end running dotnet test --collect "XPlat Code Coverage" should generate 1 coverage report with merged information even when running in parallel.

[chaoticsoftware]

Thanks @jakubch1 - what is the current timeline on delivering the work? When is 16.9.0?

Also I'm assuming that it would mean reports from a single dotnet test would be merged into one file, but if I run multiple instances as in parallel Azure Pipeline jobs for instance, I'd still need to merge myself. Any chance that this utility would be made available for external use too?

[jakubch1]

Adding @cvpoienaru who is working on the change.