CUDA kernel failed : an illegal memory access was encountered

[yanx27]

Hi @meidachen , I am currently run the HAIS model for instance segmentation. After the training process, I meet an error when running python test.py --config config/hais_run_stpls3d.yaml --pretrain exp/Synthetic_v3_InstanceSegmentation/hais/hais_run_stpls3d/hais_run_stpls3d-000000500.pth :

CUDA kernel failed : an illegal memory access was encountered

I traced the problem and found the code breaks around the following lines:

STPLS3D/HAIS/model/hais/hais.py

Line 306 in 8b83424

proposals_idx, proposals_offset = hais_ops.hierarchical_aggregation(

Could you have some suggestions? Looking forwarding to your reply!

[meidachen]

Hi @yanx27 , it seems like you are having the same issue as hustvl/HAIS#21 (comment). The authors suggested to:
"It seems a problem of memory overflow.
If point clouds of your datasets are much denser or larger than ScanNet, I suggest to increase the values of these macros."

https://github.com/meidachen/STPLS3D/blob/8b83424494fd3d1063b4d40d104465b3d6ceb4ae/HAIS/lib/hais_ops/src/hierarchical_aggregation/hierarchical_aggregation.cu#L8toL11

It maybe because you are using a GPU which has less RAM than I have (3090), so changing these parameters may work.