Improve dev workflows around benchmarks

[LucaMarconato]

The current workflow for identifying regressions and benchmarking packages is not ideal.

Some comments/ideas for improvement:

• spatialdata and napari-spatialdata both utilize asv, but in different ways: in particular, spatialdata requires manual runs, while napari-spatialdata has workflows enabled.
• The benchmarking machinery in napari-spatialdata is complex and should be simplified.
• We lack a benchmarks results page, making the exploration of benchmark results cumbersome: https://pv.github.io/numpy-bench/#?sort=3&dir=desc
• It appears that with the current workflow, we cannot easily benchmark versions with incompatible installs, which are precisely the scenarios where benchmarks would be most useful: https://github.com/scverse/napari-spatialdata/actions/runs/20647610286/job/59287297836?pr=377

Path forward: Currently, the focus on spatialdata is at the specs and APIs level, not performance. Therefore, I would consciously place less emphasis on systematic benchmarks for the time being and also disable the benchmarks workflow in napari-spatialdata (which can still be used on-demand, similar to spatialdata). When we shift our focus to performance, we should revamp the benchmarks suite, enable it universally (not just in napari-spatialdata), document it for easier onboard for new devs, and enhance it for systematic use.

[LucaMarconato]

In napari-spatialdata, the benchmarks workflow is divided into three parts:

• benchmarks.yml runs the actual benchmarks with asv continuous. This workflow is triggered weekly, on workflow dispatch, or when a PR has the label run-benchmarks.
• benchmarks_report.yml reports the benchmark results and removes the run-benchmarks tag if it was present.
• test.yml performs Python testing and runs some quick benchmarks. Systematic benchmarking of the code is not conducted here; instead, it checks if the benchmarking machinery is functioning. It does not perform systematic benchmarks, as it only uses asv run --quick: from the asv API docs, we read "Do a 'quick' run, where each benchmark function is run only once. This is useful to find basic errors in the benchmark functions faster. The results are unlikely to be useful, and thus are not saved."

I will remove the benchmarks section from test.yml because as it is now the benchmarking machines creates a maintenance burden while offering limited paybacks. In the future, we will need to re-enable it (and improve it), or replace it. Below are some considerations regarding these alternatives.

First, we will need to decide whether to continue using asv. There are some concerns regarding asv, discussed here (for scanpy) scverse/scanpy#3052. Additionally, this older conversation airspeed-velocity/asv#1219 (comment) touches on other important points (though the conversation is older, so some issues may have been resolved by now).

If we choose to keep using asv, we should update the code in napari-spatialdata. The code is derived from the napari benchmarks (since we need to test UI components), and the napari benchmarks have already undergone new changes. Therefore, we should compare the napari-spatialdata benchmark code with the napari one, as well as with the scanpy one, which is much simpler and more maintainable https://github.com/scverse/scanpy/blob/main/.github/workflows/benchmark.yml.

If we decide to drop asv, the main alternative would be codspeed. I have experimented with codspeed in the past, but one of the main limitations was that IO operations and GUI code could not be tested. However, this has been supported for the past three months, making codspeed my preferred option moving forward.