Finalise setup of buildbot for RISC-V RVA23 EVL tail folding

[asb]

This requires a builder with:
-march=rva23u64 -mllvm -force-tail-folding-style=data-with-evl -mllvm -prefer-predicate-over-epilogue=predicate-else-scalar-epilogue' and ideally qemu settings rvv_ta_all_1s=true,rvv_ma_all_1s=true,rvv_vl_half_avl=true to maximise the chance of finding bugs. This will be done using the same cross-compile and then execute under qemu-system setup used for the RVA20 bot. Not all items below are specific to the RVA23 bot.

This requires:

• Update to QEMU 9.2.0 and check for no regressions
• Redeploy x86-64 host with appropriate config
• Resolve sporadic failures due to running out of disk space.
  • Bumping the size of llvm-project.img worked. Issues were sporadic seemingly due to varying test order meaning disk size limits were sometimes reached with temporary files but sometimes not.
• Get a working local debug flow for subsets of the LLVM tests (ninja check-llvm-executionengine for instance fails to work due to llvm-lit being invoked from a different subdirectory and lit-on-qemu not handling this)
• Investigate and fix failures for MCJIT/ExecutionEngine tests
  • Issue was a failure to set -DLLVM_HOST_TRIPLE=riscv64-linux-gnu leading to a confusing compilation flow for mcjit/executionengine
• Resolve issues with host python3 path not matching the one under qemu-system (e.g. when using pip on the host)
  • Explicitly passing -DPython3_EXECUTABLE=/usr/bin/python3 resolves this
• Resolve issues with buildbot running under python3.13 on the host
  • Manual fix for pipes.quote usage and depend on legacy-cgi installed via pip
• (non-blocking issue) Document Python 3.13 workarounds in docs on local builder testing
• Resolve test failures for small subset of tests that try to use lit-on-qemu (set through -DLLVM_EXTERNAL_LIT) internally. Seems to primarily be the update_cc_test_checks tests.
  • Worked around with llvm/llvm-zorg@bc1b63c
• (non-blocking issue) Figure out why MCJIT/ExecutionEngine tests aren't running with e.g. ninja check-llvm-executionengine (marked as 'unsupported', even the RISC-V ones).
  • Marked as done as the issue was found and PRs posted [MCJIT][test] Move MCJIT non-interpreter tests to MCJIT subdirectory #124463 [ExecutionEngine][test][RISCV] Don't mark RISCV as unsupported #124464.
• Receive review on PR to switch over rva23 evl builder [RISCV] Move rva23 evl builder over to cross-compile and execute under qemu-system setup llvm-zorg#358
• Finalise x86-64 host deployment for rva23 evl builder once llvm-zorg#358 lands
• Chase up issue with the staging buildmaster seemingly not automatically redeploying (email sent to Galina)
• Test enabling the test suite locally and resolve any issues
• Port builders over to "annotated builder" setup in order to allow easier flexibility of llvm-test-suite compilation and execution
• Enable the running of the test suite on rva23 evl builder
• Evaluate what other LLVM subprojects can/should be enabled in this setup (and expand this list to cover that work)
• (non-blocking issue) Find a way to get ccache to work for the stage2 build in a non-CI configuration
  • ccache for stage2 makes no sense in CI, but can help iteration time a lot if using a fixed stage1 and investigation an issue. My attempts to enable it seem to be ignored right now.
• Address failure in clang/test/Modules/empty.modulemap (the empty modulemap seems to be slightly above the expected 60KB for some reason). PR up for review: [clang][Modules] Raise empty.modulemap expected size to <70KB to fix RISC-V failure #123959
• (non-blocking) Add preservation of the most recent build artefacts in order to make it easier (at least for the bot owner) to investigate a failure without waiting for a multi-stage rebuild.
• Move builder to non-staging buildmaster

[llvmbot]

@llvm/issue-subscribers-backend-risc-v

Author: Alex Bradbury (asb)

Details

This requires a builder with: `-march=rva23u64 -mllvm -force-tail-folding-style=data-with-evl -mllvm -prefer-predicate-over-epilogue=predicate-else-scalar-epilogue'` and ideally qemu settings `rvv_ta_all_1s=true,rvv_ma_all_1s=true,rvv_vl_half_avl=true` to maximise the chance of finding bugs. This will be done using the same cross-compile and then execute under qemu-system setup used for the RVA20 bot. Not all items below are specific to the RVA23 bot.

This requires:

• Update to QEMU 9.2.0 and check for no regressions
• Redeploy x86-64 host with appropriate config
• Resolve sporadic failures due to running out of disk space.
  • Bumping the size of llvm-project.img worked. Issues were sporadic seemingly due to varying test order meaning disk size limits were sometimes reached with temporary files but sometimes not.
• Get a working local debug flow for subsets of the LLVM tests (ninja check-llvm-executionengine for instance fails to work due to llvm-lit being invoked from a different subdirectory and lit-on-qemu not handling this)
• Investigate and fix failures for MCJIT/ExecutionEngine tests
  • Issue was a failure to set -DLLVM_HOST_TRIPLE=riscv64-linux-gnu leading to a confusing compilation flow for mcjit/executionengine
• Resolve issues with host python3 path not matching the one under qemu-system (e.g. when using pip on the host)
  • Explicitly passing -DPython3_EXECUTABLE=/usr/bin/python3 resolves this
• Resolve issues with buildbot running under python3.13 on the host
  • Manual fix for pipes.quote usage and depend on legacy-cgi installed via pip
• (non-blocking issue) Document Python 3.13 workarounds in docs on local builder testing
• Resolve test failures for small subset of tests that try to use lit-on-qemu (set through -DLLVM_EXTERNAL_LIT) internally. Seems to primarily be the update_test_checks tests.
  • Could potentially mask these tests, or alternatively find a way to override the lit path for just these tests, or set up lit-on-qemu in the correct path under qemu-system that just forwards to lit.
• (non-blocking issue) Figure out why MCJIT/ExecutionEngine tests aren't running with e.g. ninja check-llvm-executionengine (marked as 'unsupported', even the RISC-V ones).
• Receive review on PR to switch over rva23 evl builder [RISCV] Move rva23 evl builder over to cross-compile and execute under qemu-system setup llvm-zorg#358
• Finalise x86-64 host deployment for rva23 evl builder once llvm-zorg#358 lands
• Test enabling the test suite locally and resolve any issues
• Enable the running of the test suite on rva23 evl builder
• Evaluate what other LLVM subprojects can/should be enabled in this setup (and expand this list to cover that work)

[llvmbot]

@llvm/issue-subscribers-infrastructure

Author: Alex Bradbury (asb)

Details

This requires a builder with: `-march=rva23u64 -mllvm -force-tail-folding-style=data-with-evl -mllvm -prefer-predicate-over-epilogue=predicate-else-scalar-epilogue'` and ideally qemu settings `rvv_ta_all_1s=true,rvv_ma_all_1s=true,rvv_vl_half_avl=true` to maximise the chance of finding bugs. This will be done using the same cross-compile and then execute under qemu-system setup used for the RVA20 bot. Not all items below are specific to the RVA23 bot.

This requires:

• Update to QEMU 9.2.0 and check for no regressions
• Redeploy x86-64 host with appropriate config
• Resolve sporadic failures due to running out of disk space.
  • Bumping the size of llvm-project.img worked. Issues were sporadic seemingly due to varying test order meaning disk size limits were sometimes reached with temporary files but sometimes not.
• Get a working local debug flow for subsets of the LLVM tests (ninja check-llvm-executionengine for instance fails to work due to llvm-lit being invoked from a different subdirectory and lit-on-qemu not handling this)
• Investigate and fix failures for MCJIT/ExecutionEngine tests
  • Issue was a failure to set -DLLVM_HOST_TRIPLE=riscv64-linux-gnu leading to a confusing compilation flow for mcjit/executionengine
• Resolve issues with host python3 path not matching the one under qemu-system (e.g. when using pip on the host)
  • Explicitly passing -DPython3_EXECUTABLE=/usr/bin/python3 resolves this
• Resolve issues with buildbot running under python3.13 on the host
  • Manual fix for pipes.quote usage and depend on legacy-cgi installed via pip
• (non-blocking issue) Document Python 3.13 workarounds in docs on local builder testing
• Resolve test failures for small subset of tests that try to use lit-on-qemu (set through -DLLVM_EXTERNAL_LIT) internally. Seems to primarily be the update_test_checks tests.
  • Could potentially mask these tests, or alternatively find a way to override the lit path for just these tests, or set up lit-on-qemu in the correct path under qemu-system that just forwards to lit.
• (non-blocking issue) Figure out why MCJIT/ExecutionEngine tests aren't running with e.g. ninja check-llvm-executionengine (marked as 'unsupported', even the RISC-V ones).
• Receive review on PR to switch over rva23 evl builder [RISCV] Move rva23 evl builder over to cross-compile and execute under qemu-system setup llvm-zorg#358
• Finalise x86-64 host deployment for rva23 evl builder once llvm-zorg#358 lands
• Test enabling the test suite locally and resolve any issues
• Enable the running of the test suite on rva23 evl builder
• Evaluate what other LLVM subprojects can/should be enabled in this setup (and expand this list to cover that work)
• (non-blocking issue) Find a way to get ccache to work for the stage2 build in a non-CI configuration
  • ccache for stage2 makes no sense in CI, but can help iteration time a lot if using a fixed stage1 and investigation an issue. My attempts to enable it seem to be ignored right now.
• Address failure in clang/test/Modules/empty.modulemap (the empty modulemap seems to be slightly above the expected 60KB for some reason)

[mshockwave]

(non-blocking issue) Figure out why MCJIT/ExecutionEngine tests aren't running with e.g. ninja check-llvm-executionengine (marked as 'unsupported', even the RISC-V ones).

(I think we're no longer using MCJIT, it's OrcJIT that's enabled by default) If this is a cross-compiling setting, then we probably need to set LLVM_TARGET_ARCH to riscv64 for JIT to emit RISC-V code.

[asb]

(non-blocking issue) Figure out why MCJIT/ExecutionEngine tests aren't running with e.g. ninja check-llvm-executionengine (marked as 'unsupported', even the RISC-V ones).

(I think we're no longer using MCJIT, it's OrcJIT that's enabled by default) If this is a cross-compiling setting, then we probably need to set LLVM_TARGET_ARCH to riscv64 for JIT to emit RISC-V code.

Yes, there were two issues here:

• DLLVM_HOST_TRIPLE=riscv64-linux-gnu wasn't passed which led to all kinds of weirdness for these tests
• The tests were incorrectly masked on RISCV, when after a little cleanup the majority can run [ExecutionEngine][test][RISCV] Don't mark RISCV as unsupported #124464

[asb]

By way of update: I think the only blocker to the initial build config that doesn't include the full test-suite is that the automatic redeploy of llvm-zorg to LLVM's staging buildmaster appears to be broken right now (as far as I can see at least - it used to happen every hour or so, but I've had a change that landed a couple of days a go that's definitely not reflected on the upstream buildmaster, while it works fine with a local checkout of llvm-zorg HEAD and my buildmaster testing mode). I've dropped Galina an email to check on this. I know there were problems with this some months back as well.

UPDATE: Galina has fixed the issue with the staging buildmaster (thanks!).

[asb]

The builder is now running and giving results in a ~1h40-2h cycle time.

Keeping an eye on https://lab.llvm.org/staging/#/builders/16 should let you see if there appears to be a problem.

I'm not sure if the Galina's llvm-zorg staging redeploy was a one-off or if the automated deploy is enabled again - we'll see less than ideal queue merging behaviour until llvm/llvm-zorg@b272d2f is deployed on the staging buildmaster.

[asb]

Both the rva20 and rva23 evl tail folding builders are now running llvm-test-suite alongside check-all for a multi-stage LLVM build. I've updated the original issue to reflect some changes in approach for how to achieve it (ultimately I ported to the AnnotatedBuilder infrastructure rather than further fighting the ClangBuilder.py setup).

Per the process in https://llvm.org/docs/HowToAddABuilder.html#steps-to-add-builder-to-llvm-buildbot I've requested review from Galina on moving both builders to the non-staging buildmaster, meaning email notifications will be produced as normal for failures.

[asb]

The builder is now on the non-staging buildmaster and will generate emails upon failure. There are aspects I'll iterate further on of course (including setting up another companion builder with RA23 but not EVL tail folding), but one of the core goals of this issue is achieved.

[lukel97]

The EVL buildbots have been up and running for a while now, including in the gauntlet. I feel like the remaining subtasks in this PR aren't EVL specific, can we go ahead and close this issue?

[lukel97]

@asb I think all the EVL specific parts of this have been setup and ticking away for a while now, so I'll go ahead and close this issue (Feel free to reopen of course if I missed something!)