Skip to content

feat(hip-directed): add hip_directed catch2 e2e tests - #57

Draft
y1k2chaitanya wants to merge 3 commits into
mainfrom
users/kyenumul/hip-directed-catch2
Draft

feat(hip-directed): add hip_directed catch2 e2e tests#57
y1k2chaitanya wants to merge 3 commits into
mainfrom
users/kyenumul/hip-directed-catch2

Conversation

@y1k2chaitanya

Copy link
Copy Markdown
Contributor

Summary

Ports the hip_directed catch2 directed-unit-test suite into the public rocm-tests framework, mirroring the qa-forge rocm-func-core-hip-catch2-amd / Rocmtest hip_directed run.

  • New tests/e2e/hip_directed/ area (category profile: hw.gpu, layer.runtime, ci.nightly, e2e.stack, os.linux).
  • Clones public ROCm/rocm-systems at develop (overridable via ROCM_TEST_ROCM_SYSTEMS_REF), builds projects/hip-tests/catch, and runs a curated set of directed Unit_hip* tests via ctest -R (13 parametrized rows).
  • Adds tests/e2e/hip_directed/ to the e2e-tests.yml pytest dir list so nightly CI collects it.

Directed tests (13)

hipSetValidDevices, hipGetDriverEntryPoint, hipStreamGetId, hipStreamSetAttribute, hipStreamGetAttribute, hipMemPrefetchAsync_v2, hipMemAdvise_v2, hipModuleGetFunctionCount, hipModuleLoadFatBinary, hipMemcpy3DBatchAsync, hipMemcpy3DPeer, hipMemcpyBatchAsync, hipMemsetD2D.

Build design

  • Builds only the catch2 executables that hold the directed tests — DeviceTest, StreamTest, MemoryTest1, ModuleTest — instead of the whole build_tests meta-target. This skips modules like coopGrpTest (which need bleeding-edge HIP headers not in every ROCm install) and keeps the build to seconds instead of the full ~4095-test build.
  • Limits GPU visibility during the build (ROCR_VISIBLE_DEVICES=0): catch2's CMake auto-detects the offload arch via rocm_agent_enumerator, which returns one entry per visible GPU — on multi-GPU hosts that duplicates --offload-arch and trips clang-offload-bundler: Duplicate targets. A single visible device yields one arch.
  • Per-test rows: a build/run failure in one module doesn't block the others.

Test plan

Validated on MI325X (gfx942):

  • pytest tests/e2e/hip_directed/ --rock-dir <rocm> --gpu-arch gfx942 -q
  • 13/13 passed (~59s incl. the scoped build)
  • --collect-only shows 13 tests
  • ruff + black

@y1k2chaitanya
y1k2chaitanya requested review from a team, madkasul and mparamas-amd as code owners July 23, 2026 03:33
@y1k2chaitanya

Copy link
Copy Markdown
Contributor Author

Local validation — quanta MI325 (gfx942), ROCm 7.14.0-dev0+22b25e9

Pinned rocm-systems to the exact commit the installed ROCm was built from — read from <rock_dir>/share/therock/therock_manifest.json (pin_sha 10de68988e384c36d5a076a14d7c14f8e49c6a79), with ROCM_TEST_ROCM_SYSTEMS_REF override and develop fallback. This keeps the hip-tests catch2 source in sync with the installed HIP headers (fixes the version skew we hit against develop).

tests/e2e/hip_directed  →  12 passed, 0 failed, 0 skipped   (29.7s, build cached)

All 12 directed tests green on the manifest-pinned commit:
hipSetValidDevices, hipGetDriverEntryPoint, hipStreamGetId, hipStreamSetAttribute, hipStreamGetAttribute, hipMemPrefetchAsync_v2, hipMemAdvise_v2, hipModuleGetFunctionCount, hipMemcpy3DBatchAsync, hipMemcpy3DPeer, hipMemcpyBatchAsync, hipMemsetD2D.

Dropped hipModuleLoadFatBinary — its positive test loads build-generated .code fatbins produced by add_custom_target(... ALL ...) steps that are not dependencies of the scoped ModuleTest executable build (the negative test passes; not a product bug).

Pushed as 4c13820.

@y1k2chaitanya
y1k2chaitanya marked this pull request as draft July 23, 2026 04:15
@y1k2chaitanya

Copy link
Copy Markdown
Contributor Author

CI validation — e2e-tests.yml on linux-gfx94x-MI325 (gfx942)

Ran the suite through the e2e CI workflow (isolated via -k hip_directed):

collected 292 items / 280 deselected / 12 selected
12 passed, 0 failed, 0 skipped in 216.09s

Run: https://github.com/ROCm/rocm-tests/actions/runs/29980833297

Fixes made portable to the OSSCI container (pushed in 3daad92):

  • compiler_mode=cxx_hip — sets CMAKE_HIP_COMPILER to the ROCm clang++ (a bare system clang++ cannot find the ROCm device library).
  • Pin CMAKE_HIP_ARCHITECTURES — stops CMake's HIP ABI check from duplicating --offload-arch per GPU (rocm_agent_enumerator reads sysfs, ignoring ROCR_VISIBLE_DEVICES) on multi-GPU runners.
  • Bundled sysdeps numafind_library(numa)/find_path(numa.h) resolve from <rock_dir>/lib/rocm_sysdeps and MemoryTest1 loads librocm_sysdeps_numa.so.1 via LD_LIBRARY_PATH; no host apt packages required.

y1k2chaitanya and others added 3 commits July 31, 2026 01:34
Port the hip_directed catch2 directed-unit-test suite into the public rocm-tests
framework. Clones public ROCm/rocm-systems, builds only the catch2 executables
that hold the directed tests (DeviceTest, StreamTest, MemoryTest1, ModuleTest)
and runs a curated set of directed Unit_hip* tests via ctest -R (13 rows).

Build notes:
- Builds just the needed executables (not the whole build_tests meta-target),
  which skips modules like coopGrpTest that need bleeding-edge HIP headers and
  keeps the build fast.
- Limits GPU visibility during the build so rocm_agent_enumerator returns a
  single arch (multi-GPU hosts otherwise duplicate --offload-arch -> clang-
  offload-bundler Duplicate targets).

Validated 13/13 on gfx942 (MI325).
Read <rock_dir>/share/therock/therock_manifest.json and clone hip-tests at the
rocm-systems pin_sha the installed ROCm was built from (env override + develop
fallback). Keeps catch2 source in sync with installed HIP headers.

Drop hipModuleLoadFatBinary: its positive test loads build-generated .code
fatbins from ALL custom targets not built by the scoped executable build.
…h, bundled numa)

- compiler_mode=cxx_hip so CMAKE_HIP_COMPILER is the ROCm clang++ (device libs).
- Pin CMAKE_HIP_ARCHITECTURES so the ABI check does not duplicate --offload-arch
  per GPU via rocm_agent_enumerator on multi-GPU runners.
- Resolve libnuma + numa.h from the ROCm-bundled lib/rocm_sysdeps (no host apt);
  MemoryTest1 loads librocm_sysdeps_numa.so.1 via LD_LIBRARY_PATH.

Validated: 12/12 on quanta MI325 and in e2e CI on linux-gfx94x-MI325.
@y1k2chaitanya
y1k2chaitanya force-pushed the users/kyenumul/hip-directed-catch2 branch from 3daad92 to 6a6e66a Compare July 30, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants