Log a warning when an empty benchmark is run

[AvishaiDotan]

Fix #2712

I've added a warning using the composite logger to notify when an empty benchmark is executed.
Also included a test to ensure the warning is logged correctly.

[AvishaiDotan]

Hi @timcassell,

Apologies for the continuous test runs. I prefer working directly on the branch, and every time I push changes—even when the PR is marked as a draft—it still triggers the tests.

As a side note, it might be a good idea to update the CI/CD configuration so tests only run when the PR is not in draft mode.

[timcassell]

Error is printed when I use

BenchmarkRunner.Run<Benchmark>();

But nothing is printed when I use

BenchmarkRunner.Run(typeof(Program).Assembly);

[AvishaiDotan]

Error is printed when I use

BenchmarkRunner.Run<Benchmark>();

But nothing is printed when I use

BenchmarkRunner.Run(typeof(Program).Assembly);

I've tested running the assembly without any benchmarks using the following methods, and the implementation did not fail in either case:

1. Mocking with AssemblyBuilder
2. Referencing a new project that doesn't contain any benchmarks

In both scenarios, the implementation worked as expected.

Could you please elaborate on how you ran BenchmarkRunner.Run(typeof(Program).Assembly)?

[timcassell]

Just like this. No benchmark types.

public class Program
{
    public static void Main(string[] args)
    {
        BenchmarkRunner.Run(typeof(Program).Assembly);
    }
}

Ran in Visual Studio, got no output at all.

Maybe the validation errors are returned properly, but this particular API isn't logging them?

[AvishaiDotan]

You're right.
When running benchmarks based on an assembly, the BenchmarkRunInfo[] array is created like this:

assembly.GetRunnableBenchmarks()
    .Select(type => BenchmarkConverter.TypeToBenchmarks(type, config))
    .ToArray();

If assembly.GetRunnableBenchmarks() returns an empty collection, TypeToBenchmarks is never called, which means no BenchmarkRunInfo instances are created.

The key point is: when TypeToBenchmarks is called, it sets up the BenchmarkRunInfo with a default logger — BenchmarkDotNet.Loggers.ConsoleLogger. This logger is responsible for displaying validation errors in the console.

However, in the case of an empty assembly (i.e., no runnable benchmarks), TypeToBenchmarks is skipped entirely, and an empty array of benchmarkRunInfos is returned — meaning no logger is configured, and validation error are not being logged.

Possible Solution
I was thinking of adding this check inside CreateCompositeLogger:

if (benchmarkRunInfos.Length == 0)
{
    AddLogger(new ConsoleLogger());
}

This ensures a console logger is present even if no benchmarks are found, so validation errors still get reported.
That said, it feels a bit messy - I Would love your input on this.

[timcassell]

I haven't looked into the implementation details, but I did see that BenchmarkSwitcher does log the error if there are no benchmarks, so perhaps you could check that for how to handle it in BenchmarkRunner?

[timcassell]

Thanks @AvishaiDotan!