pytest based synthesis tests

[marco66colombo]

Description

This PR introduces synthesis tests into the pytest framework for hls4ml.

The goal of this change is to automate the validation of HLS synthesis reports against predefined baselines to ensure stability and correctness over time, using pytest.

Main Changes

• New submodule in tests/pytests called baselines, which is a Git submodule pointing to an external repository containing synthesis baselines.

  • The structure of that repository follows the format:
    backend_tool/version/test_artifacts.json
  • Note: Currently, this is a temporary personal repo and must be replaced by a proper one under the organization before merging.

• Baseline management:

  • Baselines are organized by backend (e.g., Vivado, Vitis), tool version (e.g., 2020.1, 2023.1), and test case.
  • A manual and recurrent update policy for these baselines should be defined and enforced.

• New conftest.py file to manage shared pytest fixtures:

  • Introduces the synthesis_config fixture, which provides:
    • Whether to run synthesis tests (RUN_SYNTHESIS environment variable).
    • Tool versions (via env vars, e.g., VIVADO_VERSION, defaulting to 2023.1).
    • Build arguments (backend-dependent) for model.build(...).

• Synthesis logic centralized in synthesis_helpers.py:

  • Main method: test_synthesis.
  • If RUN_SYNTHESIS=true, runs synthesis and compares the report to baseline.
  • Skips test if:
    • Tool is not available,
    • Baseline is missing
    • Currently, in the above cases, the whole test is skipped, so it might has to change in future. It is also true that this method is called after any other assert in the test case, so it would still fail in case of a wrong assert in the code before this method is called.

• Tolerances for report values can be specified using get_tolerance.

  • Currently, the tolerance values are placeholders and need to be properly defined.

• CI integration:

  • Synthesis reports are saved as JSON and collected as artifacts via GitLab CI (ci-template.yml updated accordingly).

• Current test coverage:

  • Only test_keras_api.py includes synthesis tests at the moment.
  • Easily extendable to other tests:
    • Add baseline_file_name to the test,
    • Call test_synthesis().

• OneAPI backend is currently skipped due to missing synthesis reports support.

---

Type of change

• Other: improvment of testing infrastrucutre using pytest

---

Tests

Synthesis tests are run conditionally and compared against versioned baselines if RUN_SYNTHESIS=true is set in the environment.

Checklist

• I have read the guidelines for contributing.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have installed and run pre-commit on the files I edited or added.
• I have added tests that prove my feature works.

[jmitrevs]

fpga_emu is actually just C compilation, so this needs to be changed, though to what I am not sure. The descriptions are given in https://github.com/oneapi-src/oneAPI-samples/tree/release/2025.0/DirectProgramming/C%2B%2BSYCL_FPGA/Tutorials/GettingStarted/fpga_compile#four-compilation-options and they correspond to fpga_emu, report, fpga_sim, and fpga. I would guess report unless you want to run cosim, in which case it's fpga_sim. The first two don't require Quartus, while the last two do.

[jmitrevs]

Can we test this to actually run SYNTHESIS tests?

[marco66colombo]

yes, but if the env var 'RUN_SYNTHESIS' is not set, the default is False.
I can add a line in the ci-template.yml and export RUN_SYNTHESIS=True

[vloncar]

What causes this requirement?

[marco66colombo]

I was trying to fix an error that is present only while running the code in the GitLab runner and not if I run the docker image locally.
It happens with oneAPI backend.
It was a test, but it didn't fix the issue and I'll remove it in the next commit.
Failed gitlab job: https://gitlab.cern.ch/fastmachinelearning/hls4ml/-/jobs/54542573

[marco66colombo]

/opt/Xilinx/Vitis/2020.1/tps/lnx64/cmake-3.3.2/bin/cmake
$ cmake --version
cmake: error while loading shared libraries: libidn.so.11: cannot open shared object file: No such file or directory

I found that the difference is that the cmake comes from the Vitis installation with Vitis2020.1.
This is what causes the error.

[marco66colombo]

Do you all agree with the approach of storing all the baselines in a dedicated GitHub repo, and add it as a module under test/pytest/baselines in hls4ml?
If yes, we need to create the repository within Fastmachinelearning and then I can upload as baselines the artifacts generated by the synthesis tests.

[jmitrevs]

I generally agree about storing the baselines in a dedicated repo. Is CERN gitlab or regular github preferred? We create the testing containers in gitlab. I don't really have a preference.