When benchmarking against multiple TFMs including a Native AOT one, BenchmarkDotNet gets confused about framework identifiers

[akamsteeg]

(I'm using BenchmarkDotNet v0.14. The issue occurs on Windows, I haven't tested it on Linux or Mac OS.)

I have a Program.cs inside a benchmark project that looks like this:

public static class Program
{
  public static void Main(string[] args)
  {
    var config = GetConfig();
    BenchmarkSwitcher
      .FromAssembly(Assembly.GetExecutingAssembly())
      .Run(args, config);
  }

  private static ManualConfig GetConfig()
  {
    var job = Job.Default;

    var config = ManualConfig.Create(DefaultConfig.Instance)
      .AddJob(
        job.WithToolchain(CsProjCoreToolchain.NetCoreApp80).AsBaseline(),
        Job.Default.WithRuntime(NativeAotRuntime.Net80),
        job.WithToolchain(CsProjCoreToolchain.NetCoreApp60)
        )
      .AddDiagnoser(MemoryDiagnoser.Default);

    if (Environment.OSVersion.Platform == PlatformID.Win32NT)
    {
      config.AddJob(job.WithToolchain(CsProjClassicNetToolchain.Net481));
    }

    config.SummaryStyle = SummaryStyle.Default
      .WithRatioStyle(RatioStyle.Percentage);

    config.AddValidator(JitOptimizationsValidator.FailOnError); // Fail when any of the referenced assemblies are not optimized

    config.WithOrderer(new DefaultOrderer(SummaryOrderPolicy.FastestToSlowest));

    return config;
  }
}

So I'm running benchmarks against .NET 8 as my baseline, .NET 8 with Native AOT, .NET 6 and .NET 4.8.1 when on Windows. When running this using dotnet build -c Release && dotnet run -c Release -f net8.0 the final report in the console shows this:

// * Summary *

BenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.3958/23H2/2023Update/SunValley3)
AMD Ryzen 7 7800X3D, 1 CPU, 16 logical and 8 physical cores
.NET SDK 9.0.100-preview.6.24328.19
  [Host]     : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-IRRDUC : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-BCZWNO : .NET 8.0.5, X64 NativeAOT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-EMODDW : .NET 6.0.32 (6.0.3224.31407), X64 RyuJIT AVX2
  Job-LCVGYL : .NET Framework 4.8.1 (4.8.9256.0), X64 RyuJIT VectorSize=256


| Method                        | Runtime       | password             | Mean       | Error    | StdDev   | Ratio    | RatioSD | Gen0   | Gen1   | Allocated | Alloc Ratio |
|------------------------------ |-------------- |--------------------- |-----------:|---------:|---------:|---------:|--------:|-------:|-------:|----------:|------------:|
| GetKAnonimityPartsForPassword | .NET 8.0      | -&HxcB_d             |   277.4 ns |  4.09 ns |  3.83 ns | baseline |         | 0.0043 |      - |     232 B |             |
| GetKAnonimityPartsForPassword | NativeAOT 8.0 | -&HxcB_d             |   284.4 ns |  5.69 ns |  7.20 ns |      +3% |    2.8% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&HxcB_d             |   288.1 ns |  4.49 ns |  4.20 ns |      +4% |    1.9% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&HxcB_d             | 3,249.2 ns | 54.70 ns | 51.16 ns |  +1,072% |    2.0% | 0.7477 | 0.0038 |    4710 B |     +1,930% |
|                               |               |                      |            |          |          |          |         |        |        |           |             |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&Hxc(...)QbuAz [64] |   361.9 ns |  7.22 ns |  6.75 ns | baseline |         | 0.0043 |      - |     232 B |             |
| GetKAnonimityPartsForPassword | NativeAOT 8.0 | -&Hxc(...)QbuAz [64] |   362.2 ns |  4.28 ns |  4.00 ns |      +0% |    2.1% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&Hxc(...)QbuAz [64] |   369.0 ns |  3.93 ns |  3.28 ns |      +2% |    2.0% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&Hxc(...)QbuAz [64] | 3,274.8 ns | 62.43 ns | 61.32 ns |    +805% |    2.6% | 0.7553 | 0.0038 |    4766 B |     +1,954% |
|                               |               |                      |            |          |          |          |         |        |        |           |             |
| GetKAnonimityPartsForPassword | .NET 8.0      | これはパスワードです           |   305.4 ns |  3.88 ns |  3.44 ns | baseline |         | 0.0043 |      - |     232 B |             |
| GetKAnonimityPartsForPassword | NativeAOT 8.0 | これはパスワードです           |   329.3 ns |  6.50 ns |  7.49 ns |      +8% |    2.5% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | これはパスワードです           |   330.8 ns |  3.53 ns |  3.30 ns |      +8% |    1.4% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | これはパスワードです           | 3,248.5 ns | 42.48 ns | 39.73 ns |    +964% |    1.6% | 0.7515 | 0.0038 |    4734 B |     +1,941% |

In the first part of the summary, where the jobs are listed, it correctly identifies the runtimes used. However, in the overview of the metrics per benchmark/runtime combination, it gets confused about it and only lists .NET 8.0 or NativeAOT 8.0 and not .NET 6 and .NET 4.8.1. The metrics are okay though, so it looks like just a display issue for the runtime names.

⚠️ Removing the AOT job (Job.Default.WithRuntime(NativeAotRuntime.Net80)) solves this issue so it seems to be related to having Native AOT benchmarks mixed with 'normal' ones.

For reproduction the code I ran is on Github: akamsteeg/AtleX.HaveIBeenPwned@c6c6cd4. Running it with dotnet build -c Release && dotnet run -c Release -f net8.0 and picking any benchmark should reproduce this. It's consistent for me on two Windows 11 machines.

[filzrev]

in the overview of the metrics per benchmark/runtime combination, it gets confused about it and only lists .NET 8.0 or NativeAOT 8.0 and not .NET 6 and .NET 4.8.1.

It seems occurred when using both WithToolchain and WithRuntime methods are used for jobs configuration.

WithToolchain don't set Runtime characterics to the Job.
If Runtime value is not set. It display [Host] runtime value instead.
And when Runtime column is shown. Toolchain column is automatically hidden.

As a work around.
It can be resolved by explicitly setting runtime by using .WithRuntime(CoreRuntime.CoreNN).

Alternatively it can explicitly specify JobId by using WithId(id) method. and hide Runtime column.

[akamsteeg]

@filzrev , thanks for your reply. I've tested your suggested solution and that seems to work.

The configuration was changed too:

var config = DefaultConfig.Instance
      .AddDiagnoser(MemoryDiagnoser.Default)
      .AddColumn(StatisticColumn.Median, StatisticColumn.Min, StatisticColumn.Max)
      .AddJob(
 job.WithRuntime(CoreRuntime.Core80).AsBaseline(),
        job.WithRuntime(CoreRuntime.Core60),
        job.WithRuntime(NativeAotRuntime.Net80)
      );

    if (Environment.OSVersion.Platform == PlatformID.Win32NT)
    {
      config.AddJob(job.WithRuntime(ClrRuntime.Net481));
    }

That resulted in the correct output:

BenchmarkDotNet v0.14.0, Windows 11 (10.0.26100.4351)
AMD Ryzen 7 7800X3D, 1 CPU, 16 logical and 8 physical cores
.NET SDK 10.0.100-preview.3.25201.16
  [Host]     : .NET 8.0.17 (8.0.1725.26602), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-DAISQD : .NET 8.0.14, X64 NativeAOT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-SZYMSL : .NET 8.0.17 (8.0.1725.26602), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-WNOWAU : .NET 6.0.36 (6.0.3624.51421), X64 RyuJIT AVX2
  Job-AJMMOP : .NET Framework 4.8.1 (4.8.9310.0), X64 RyuJIT VectorSize=256

WarmupCount=16  

Method	Runtime	password	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen0	Gen1	Allocated	Alloc Ratio
GetKAnonimityPartsForPassword	NativeAOT 8.0	-&HxcB_d	262.2 ns	5.15 ns	7.70 ns	259.7 ns	254.2 ns	283.2 ns	-3%	3.8%	0.0043	-	232 B	+0%
GetKAnonimityPartsForPassword	.NET 8.0	-&HxcB_d	271.7 ns	5.26 ns	6.84 ns	270.4 ns	263.3 ns	284.1 ns	baseline		0.0043	-	232 B
GetKAnonimityPartsForPassword	.NET 6.0	-&HxcB_d	294.3 ns	5.75 ns	5.38 ns	293.2 ns	285.8 ns	304.0 ns	+8%	3.0%	0.0043	-	232 B	+0%
GetKAnonimityPartsForPassword	.NET Framework 4.8.1	-&HxcB_d	2,970.3 ns	58.72 ns	80.38 ns	2,928.6 ns	2,887.5 ns	3,100.3 ns	+994%	3.6%	0.7477	0.0038	4710 B	+1,930%
GetKAnonimityPartsForPassword	NativeAOT 8.0	-&Hxc(...)QbuAz [64]	340.1 ns	5.50 ns	4.88 ns	337.9 ns	335.3 ns	351.9 ns	-0%	2.2%	0.0043	-	232 B	+0%
GetKAnonimityPartsForPassword	.NET 8.0	-&Hxc(...)QbuAz [64]	341.8 ns	6.63 ns	5.88 ns	342.5 ns	332.6 ns	352.4 ns	baseline		0.0043	-	232 B
GetKAnonimityPartsForPassword	.NET 6.0	-&Hxc(...)QbuAz [64]	363.3 ns	7.25 ns	8.63 ns	364.6 ns	353.4 ns	381.2 ns	+6%	2.9%	0.0043	-	232 B	+0%
GetKAnonimityPartsForPassword	.NET Framework 4.8.1	-&Hxc(...)QbuAz [64]	3,095.0 ns	59.48 ns	70.80 ns	3,075.1 ns	3,022.9 ns	3,232.2 ns	+806%	2.8%	0.7553	0.0038	4766 B	+1,954%
GetKAnonimityPartsForPassword	NativeAOT 8.0	これはパスワードです	289.4 ns	5.77 ns	5.40 ns	286.5 ns	284.4 ns	300.8 ns	-2%	2.5%	0.0043	-	232 B	+0%
GetKAnonimityPartsForPassword	.NET 8.0	これはパスワードです	293.9 ns	5.57 ns	5.21 ns	291.2 ns	287.6 ns	302.4 ns	baseline		0.0043	-	232 B
GetKAnonimityPartsForPassword	.NET 6.0	これはパスワードです	302.3 ns	5.86 ns	5.75 ns	302.3 ns	295.1 ns	313.5 ns	+3%	2.5%	0.0043	-	232 B	+0%
GetKAnonimityPartsForPassword	.NET Framework 4.8.1	これはパスワードです	2,987.2 ns	47.60 ns	44.53 ns	2,966.0 ns	2,946.5 ns	3,070.6 ns	+917%	2.2%	0.7515	0.0038	4734 B	+1,941%

[filzrev]

BenchmarkDotNet v0.15.2 add runtime validator to raise warning when Runtime column shows incorrect information.

As commented at #2771 (comment). This validator is temporary workaround and it need to be fixed in future.

I think the UX needs to be improved around this, but this is ok as a stop-gap for now.

The root cause of problem is method returns Host runtime if runtime characteristic is not set.
So it need to add additional logics to lookup Toolchain characteristic value.
As far as I've confirmed following methods are relating to this issue.

1. EnvironmentMode::GetRuntime()
2. BenchmarkCase::GetRuntime()
3. BenchmarkPartitioner::GetRuntime()

[timcassell]

I think the proper fix is to make Runtime a property of IToolchain. They are fundamentally coupled concepts, but the job system pretends like they are not (technically a user could use a toolchain with a non-matching runtime, a case that we have not tested). It's a breaking change, but will clean things up quite a bit. Internal code has a lot of places that check both the runtime and the toolchain because of this, and it's brittle (even some places that should do it but don't).