GPU stream implementation for ONNX runtime #14117

ChSonnabend · 2025-03-27T14:17:06Z

No description provided.

Please consider the following formatting changes to AliceO2Group#14069

…ce and one initialization

Please consider the following formatting changes to AliceO2Group#14069

github-actions · 2025-03-27T14:17:18Z

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1
async-2024-PbPb-apass2
async-2023-PbPb-apass5

davidrohr

looks ok, just one change needed

davidrohr · 2025-03-27T14:20:28Z

GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu

 #else  // HIP
 void* GPUReconstructionHIP::getGPUPointer(void* ptr)
 {
  void* retVal = nullptr;
  GPUChkErr(hipHostGetDevicePointer(&retVal, ptr, 0));
  return retVal;
 }
+
+#ifdef GPUCA_HAS_ONNX
+int32_t GPUReconstructionCUDA::SetONNXGPUStream(OrtSessionOptions* session_options, int32_t stream)


Hier sollte HIP stehen statt CUDA

davidrohr · 2025-03-27T14:21:24Z

GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu

@@ -16,6 +16,7 @@
 #include "GPUReconstructionCUDAIncludesHost.h"

 #include <cuda_profiler_api.h>
+#include "ML/OrtInterface.h"


Warum brauchst du hier den Interface header?

Please consider the following formatting changes to AliceO2Group#14117

…eO2Group#14069

ChSonnabend · 2025-03-28T09:08:47Z

The actual CCDB fetching and loading of models will be adjusted in the next PR. For now, this will build and run with some (techincally unneccesary) extra processing steps for model initialization.
Triggering the CI now...

alibuild · 2025-03-28T10:28:17Z

Error while checking build/O2/fullCI_slc9 for 46fb1e1 at 2025-03-28 11:28:

## sw/BUILD/O2-latest/log
/sw/SOURCES/O2/14117-slc9_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu(671): error: identifier "CreateCUDAProviderOptions" is undefined
/sw/SOURCES/O2/14117-slc9_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu(678): error: identifier "UpdateCUDAProviderOptionsWithValue" is undefined
/sw/SOURCES/O2/14117-slc9_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu(679): error: identifier "SessionOptionsAppendExecutionProvider_CUDA_V2" is undefined
/sw/SOURCES/O2/14117-slc9_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu(682): error: identifier "ReleaseCUDAProviderOptions" is undefined
ninja: build stopped: subcommand failed.

Full log here.

ChSonnabend · 2025-03-28T12:10:22Z

Marking as draft until shadowProcessors are implemented

… will merge AliceO2Group#14069 to have the changes in GPUChainTrackingClusterizer.

ChSonnabend · 2025-04-11T12:20:03Z

Triggering the CI

alibuild · 2025-04-11T13:15:01Z

Error while checking build/O2/fullCI_slc9 for 5f741fc at 2025-04-11 15:14:

## sw/BUILD/O2-latest/log
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

Full log here.

ChSonnabend · 2025-04-11T18:47:28Z

Interestingly the ORT variables are all OFF / 0 in the slc9 builder even though the ROCM and CUDA builds should be enabled. Are there dependencies missing in the build container?

From the previous build:

+++ export ORT_ROCM_BUILD=0 +++ ORT_ROCM_BUILD=0 +++ export ORT_CUDA_BUILD=OFF +++ ORT_CUDA_BUILD=OFF +++ export ORT_MIGRAPHX_BUILD=0 +++ ORT_MIGRAPHX_BUILD=0 +++ export ORT_TENSORRT_BUILD=0 +++ ORT_TENSORRT_BUILD=0 ++ echo 'ORT_ROCM_BUILD: 0' ORT_ROCM_BUILD: 0 ++ echo 'ORT_CUDA_BUILD: OFF' ORT_CUDA_BUILD: OFF

alibuild · 2025-04-11T19:25:07Z

Error while checking build/O2/fullCI_slc9 for 70907aa at 2025-04-11 21:25:

## sw/BUILD/O2-latest/log
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

Full log here.

davidrohr · 2025-04-13T06:38:39Z

I also saw that the ORT variables are OFF, and I cannot really understand why. However, why do we have the link failure? That seems to be a genuine error if the ORT variables are off, since you cannot expect that all dependencies are available. I.e., the software should also build correctly with GPU stuff available but ONNX ORT stuff not available.

/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/bin/../lib/gcc/x86_64-unknown-linux-gnu/14.2.0/../../../../x86_64-unknown-linux-gnu/bin/ld: GPU/GPUTracking/Base/cuda/CMakeFiles/O2lib-GPUTrackingCUDA.dir/GPUReconstructionCUDA.cu.o:(.data.rel.ro._ZTVN2o23gpu21GPUReconstructionCUDAE[_ZTVN2o23gpu21GPUReconstructionCUDAE]+0x100): undefined reference to `o2::gpu::GPUReconstructionCUDA::SetONNXGPUStream(Ort::SessionOptions&, int, int*)'

That should be fixed in any case. Could you have a look (while the ORT variables are still OFF)?

Then for why the ORT variables are off, I have no idea. The FullCI should use this docker container: registry.cern.ch/alisw/slc9-gpu-builder @singiamtel : correct me if I am wrong. It is created from these files https://github.com/alisw/docks/tree/master/slc9-gpu-builder, so it should contain all dependencies.

alibuild · 2025-04-13T11:16:39Z

Error while checking build/O2/fullCI_slc9 for e46cdfa at 2025-04-13 13:16:

## sw/BUILD/O2Physics-latest/log
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:


## sw/BUILD/O2-full-system-test-latest/log
Detected critical problem in logfile sim.log


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/1324b702156a43154d5a6574c37d878d8a5f4bb6/slc9_x86-64/o2checkcode/1.0-local168/etc/modulefiles
++ cat
--

Full log here.

davidrohr · 2025-04-13T14:44:51Z

Thanks, linker issue is fixed. It now fails due to the FullCI simulation issue, which I just fixed. Next time the FullCI runs, it should pass it.

And I think I understand what happens with the ORT builds
@singiamtel @ktf : I believe we build ONNXRuntime in another container without CUDA/ROCm support, and then we push that into the binary package repository, and then when we run the FullCI, we pick up the binary package from the store that was built without ORT support, so the slc9-gpu-builder container builds O2 with an ONNXRuntime that does not have GPU support.
I asssume to fix this we have to create different package hashes depending on whether GPU support is available or not. We should discuss tomorrow at CERN, if I understand correctly that functionality should already be there.

…int statements and time measurements

Please consider the following formatting changes to AliceO2Group#14117

davidrohr

@ChSonnabend : I have a couple of mostly cosmetic comments, please have a look, besides the PR is fine with me now, and it passes the CI, now also compiling with ORT_ROCM_BUILD=1. So once you have addressed my points and you write that you are satisfied with the current state, we can merge it.

davidrohr · 2025-04-17T09:29:25Z

GPU/GPUTracking/Base/cuda/CMakeLists.txt

    PRIVATE_INCLUDE_DIRECTORIES
      ${CMAKE_SOURCE_DIR}/Detectors/Base/src
      ${CMAKE_SOURCE_DIR}/Detectors/TRD/base/src
      ${CMAKE_SOURCE_DIR}/DataFormats/Reconstruction/src
      ${CMAKE_CURRENT_SOURCE_DIR}
    TARGETVARNAME targetName)

+  message("Compile definitions for ONNX runtime (CUDA):")


I would not put all of this here, CUDA build is only done when compiling with CUDA, so the printout will only be there if we build for CUDA, which makes no sense for the ROCm variable to be honest. Do we actually need this, since we anyway have this printout in the cmake line with -DCMAKE....

In general, I would propose to do the following:

Remove the printouts here.

Create a separate PR, in which:

you move the include(dependencies/FindONNXRuntime.cmake) from the main CMakeLists.txt to dependencies/O2Dependencies.cmake. In there, you create a one-line printout that shows "ONNXRuntime Found: 0/1 ORT_CUDA_BUILD 0/1 ...".

davidrohr · 2025-04-17T09:30:06Z

GPU/GPUTracking/Base/cuda/CMakeLists.txt

+  target_compile_definitions(${targetName} PRIVATE
+    GPUCA_HAS_ONNX=1
+    ORT_ROCM_BUILD=$<BOOL:${ORT_ROCM_BUILD}>
+    ORT_CUDA_BUILD=$<BOOL:${ORT_CUDA_BUILD}>


do we need ROCM and MIGRAPHX compile definitions for CUDA? I assume we can remove them here, and add them only for the ROCm one?

davidrohr · 2025-04-17T09:31:39Z

Common/ML/CMakeLists.txt

 o2_add_library(ML
               SOURCES src/OrtInterface.cxx
               TARGETVARNAME targetName
               PRIVATE_LINK_LIBRARIES O2::Framework ONNXRuntime::ONNXRuntime)
+
+# Pass ORT variables as a preprocessor definition
+target_compile_definitions(${targetName} PRIVATE


Instead of setting 0/1 definitions, I would set only the =1 definition, if the CMake variable is set.
Then, in the code further below you don't need
#if defined(FOO) && FOO=1, but you can simply use #ifdef FOO

davidrohr · 2025-04-17T09:32:06Z

GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu

+
+void GPUReconstructionCUDA::SetONNXGPUStream(Ort::SessionOptions& session_options, int32_t stream, int32_t* deviceId)
+{
+#if defined(ORT_CUDA_BUILD) && ORT_CUDA_BUILD == 1


If you do as I wrote above, you can use a simple #ifdef here.

davidrohr · 2025-04-17T09:32:44Z

GPU/GPUTracking/Base/hip/CMakeLists.txt

    PRIVATE_INCLUDE_DIRECTORIES
      ${CMAKE_SOURCE_DIR}/Detectors/Base/src
      ${CMAKE_SOURCE_DIR}/Detectors/TRD/base/src
      ${CMAKE_SOURCE_DIR}/DataFormats/Reconstruction/src
      ${GPUCA_HIP_SOURCE_DIR}
    TARGETVARNAME targetName)

+  message("Compile definitions for ONNX runtime (HIP / ROCM):")


same as I commented for CUDA applies here

davidrohr · 2025-04-17T09:33:12Z

GPU/GPUTracking/CMakeLists.txt

                 PUBLIC_INCLUDE_DIRECTORIES ${INCDIRS}
                 SOURCES ${SRCS} ${SRCS_NO_CINT} ${SRCS_NO_H})

  target_include_directories(
    ${targetName}
    PRIVATE $<TARGET_PROPERTY:O2::Framework,INTERFACE_INCLUDE_DIRECTORIES>)

-  target_compile_definitions(${targetName} PRIVATE GPUCA_O2_LIB GPUCA_TPC_GEOMETRY_O2 GPUCA_HAS_ONNX=1)
+  target_compile_definitions(${targetName} PRIVATE


please don't change the formatting if not needed.

davidrohr · 2025-04-17T09:34:52Z

GPU/GPUTracking/Global/GPUChainTrackingClusterizer.cxx

@@ -42,6 +42,7 @@
 #ifdef GPUCA_HAS_ONNX
 #include "GPUTPCNNClusterizerKernels.h"
 #include "GPUTPCNNClusterizerHost.h"
+// #include "ML/3rdparty/GPUORTFloat16.h"


why do you add a commented file?

davidrohr · 2025-04-17T09:39:46Z

GPU/Workflow/src/GPUWorkflowSpec.cxx

@@ -118,6 +119,7 @@ GPURecoWorkflowSpec::GPURecoWorkflowSpec(GPURecoWorkflowSpec::CompletionPolicyDa
  mConfig.reset(new GPUO2InterfaceConfiguration);
  mConfParam.reset(new GPUSettingsO2);
  mTFSettings.reset(new GPUSettingsTF);
+  mNNClusterizerSettings.reset(new GPUSettingsProcessingNNclusterizer);


You don't need this here, all these settings should already be made available from mConfig->configProcessing.nn

davidrohr · 2025-04-17T09:41:52Z

GPU/Workflow/src/GPUWorkflowSpec.cxx

+    o2::tpc::NeuralNetworkClusterizer nnClusterizerFetcher;
+    nnClusterizerFetcher.initCcdbApi(mNNClusterizerSettings->nnCCDBURL);
+    std::map<std::string, std::string> ccdbSettings = {
+      {"nnCCDBURL", mNNClusterizerSettings->nnCCDBURL},


Strictly speaking, all CCDB-related settings so far are in GPUSettingsO2, not in GPUSettingsProcessing. Perhaps it would be cleaner to also move the NN CCDB settings there, but since they are anyway encapsulated in your own struct, I don't care much.

alibuild · 2025-04-19T09:44:00Z

Error while checking build/O2/fullCI_slc9 for a67b634 at 2025-04-19 11:43:

## sw/BUILD/O2Physics-latest/log
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/05b665234baab729088cfcd7ed4a6c4f4e367afa/slc9_x86-64/o2checkcode/1.0-local226/etc/modulefiles
++ cat
--

Full log here.

Please consider the following formatting changes to AliceO2Group#14117

ChSonnabend · 2025-04-19T22:42:42Z

Technically, the Ort Session is now capable of reserving arena memory with a custom allocator using the volatile memory allocation (tested via print-outs). Unfortunately, when using IO binding I haven't figured out a way yet to also use volatile allocation when doing the tensor allocation, which is actually the significant part of the total memory. For now this isn't used anyway, so adjusting for comments and good to go from my side (will open a separate PR for that once figured out).

alibuild · 2025-04-19T23:22:10Z

Error while checking build/O2/fullCI_slc9 for 4d3f54d at 2025-04-20 01:22:

## sw/BUILD/O2-latest/log
/sw/SOURCES/O2/14117-slc9_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu(643): error: identifier "CreateCUDAProviderOptions" is undefined
/sw/SOURCES/O2/14117-slc9_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu(650): error: expression must have class type but it has type "OrtCUDAProviderOptionsV2 *"
/sw/SOURCES/O2/14117-slc9_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu(651): error: identifier "UpdateCUDAProviderOptionsWithValue" is undefined
/sw/SOURCES/O2/14117-slc9_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu(652): error: cannot convert to incomplete class "OrtCUDAProviderOptionsV2"
/sw/SOURCES/O2/14117-slc9_x86-64/0/GPU/GPUTracking/Base/cuda/GPUReconstructionCUDA.cu(655): error: identifier "ReleaseCUDAProviderOptions" is undefined
ninja: build stopped: subcommand failed.

Full log here.

davidrohr

looks good now, can be squash-merged once CI is green

ChSonnabend and others added 14 commits March 15, 2025 21:36

Initial set of bug.fixes and cosmetic changes

84eac06

Please consider the following formatting changes

2191649

Merge pull request #18 from alibuild/alibot-cleanup-14069

5be779c

Please consider the following formatting changes to AliceO2Group#14069

Adjusting eval sizes. Makes code neater and avoids some calculations

b742c50

Merge branch 'dev' into gpu_clusterizer_bug_fixes

c0bc918

Adding separate functions. Now the host process only needs one instan…

0c1cfb7

…ce and one initialization

First version of CCDB implementation

83c004f

Working CCDB API calls (tested with test-ccdb)

d767ed1

Improve fetching, but have to pass settings by value, not const ref

ad4b22b

Using const ref and moving CCDB calls to host initialization

81c646b

Simplifications and renaming

566ddb7

Please consider the following formatting changes

a9c33b5

Merge pull request #19 from alibuild/alibot-cleanup-14069

0ed7d25

Please consider the following formatting changes to AliceO2Group#14069

First version of GPU stream implementation. Still needs testing.

9037ea6

alibuild mentioned this pull request Mar 27, 2025

Please consider the following formatting changes to #14117 ChSonnabend/AliceO2#20

Merged

davidrohr reviewed Mar 27, 2025

View reviewed changes

ChSonnabend and others added 4 commits March 27, 2025 15:41

Fixes

64c19d5

Please consider the following formatting changes

8a5bb69

Merge pull request #20 from alibuild/alibot-cleanup-14117

e657928

Please consider the following formatting changes to AliceO2Group#14117

Adding the lane variable. This PR will in any case conflict with Alic…

46fb1e1

…eO2Group#14069

ChSonnabend marked this pull request as ready for review March 28, 2025 09:08

ChSonnabend requested a review from a team as a code owner March 28, 2025 09:08

ChSonnabend marked this pull request as draft March 28, 2025 12:09

Compiles on EPNs. Need to add shadow processors next. But for this, I…

70320c3

… will merge AliceO2Group#14069 to have the changes in GPUChainTrackingClusterizer.

alibuild mentioned this pull request Mar 29, 2025

Please consider the following formatting changes to #14117 ChSonnabend/AliceO2#21

Merged

Merge branch 'gpu_clusterizer_bug_fixes' into onnx_gpu_implementation

3174e39

Fixing warnings (errors due to size_t)

70907aa

Fixing linker issues

e46cdfa

ChSonnabend and others added 3 commits April 15, 2025 09:06

Merge branch 'dev' into onnx_gpu_implementation

37955fa

Adding volatile memory allocation and MockedOrtAllocator. Removing pr…

4b0825a

…int statements and time measurements

Please consider the following formatting changes

497a9d4

alibuild mentioned this pull request Apr 16, 2025

Please consider the following formatting changes to #14117 ChSonnabend/AliceO2#22

Merged

ChSonnabend added 3 commits April 16, 2025 13:55

Merge pull request #22 from alibuild/alibot-cleanup-14117

aabddb7

Please consider the following formatting changes to AliceO2Group#14117

Merge dev + fixes

cfdc15f

Circumvent "unused result" warning and build failure

a67b634

davidrohr requested changes Apr 17, 2025

View reviewed changes

ChSonnabend and others added 2 commits April 20, 2025 00:38

Adjust for comments

938a1ed

Please consider the following formatting changes

7b07496

alibuild mentioned this pull request Apr 19, 2025

Please consider the following formatting changes to #14117 ChSonnabend/AliceO2#23

Merged

Merge pull request #23 from alibuild/alibot-cleanup-14117

4d3f54d

Please consider the following formatting changes to AliceO2Group#14117

Fixing build flags

af89c9a

davidrohr approved these changes Apr 20, 2025

View reviewed changes

davidrohr enabled auto-merge (squash) April 20, 2025 08:02

davidrohr disabled auto-merge April 20, 2025 10:20

davidrohr merged commit 497d53f into AliceO2Group:dev Apr 20, 2025
13 checks passed

ChSonnabend deleted the onnx_gpu_implementation branch April 25, 2025 08:30

GPU stream implementation for ONNX runtime #14117

GPU stream implementation for ONNX runtime #14117

Uh oh!

Conversation

ChSonnabend commented Mar 27, 2025

Uh oh!

github-actions bot commented Mar 27, 2025

Uh oh!

davidrohr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChSonnabend commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alibuild commented Mar 28, 2025

Uh oh!

ChSonnabend commented Mar 28, 2025

Uh oh!

ChSonnabend commented Apr 11, 2025

Uh oh!

alibuild commented Apr 11, 2025

Uh oh!

ChSonnabend commented Apr 11, 2025

Uh oh!

alibuild commented Apr 11, 2025

Uh oh!

davidrohr commented Apr 13, 2025

Uh oh!

alibuild commented Apr 13, 2025

Uh oh!

davidrohr commented Apr 13, 2025

Uh oh!

davidrohr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alibuild commented Apr 19, 2025

Uh oh!

ChSonnabend commented Apr 19, 2025

Uh oh!

alibuild commented Apr 19, 2025

Uh oh!

davidrohr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ChSonnabend commented Mar 28, 2025 •

edited

Loading