CUDA not able to recognize both cards

[LoSunny]

I have read the documentation and re-compile llama.cpp using cmake -B build -DGGML_CUDA=ON, however it seems that it can only detect one of my GPU cards. Anyone know why?

$ sunny@hpcnode2:~$ ./llama.cpp/build/bin/llama-cli Llama-3.3-70B-Instruct-F16.gguf -ngl 81 --list-devices
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA A800-SXM4-80GB MIG 4g.40gb, compute capability 8.0, VMM: yes
Available devices:
  CUDA0: NVIDIA A800-SXM4-80GB MIG 4g.40gb (40448 MiB, 40196 MiB free)



$ sunny@hpcnode2:~$ nvidia-smi
Tue Feb  4 00:57:51 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07             Driver Version: 535.161.07   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A800-SXM4-80GB          On  | 00000000:B7:00.0 Off |                   On |
| N/A   30C    P0              53W / 400W |     37MiB / 81920MiB |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A800-SXM4-80GB          On  | 00000000:BD:00.0 Off |                   On |
| N/A   33C    P0              55W / 400W |      1MiB / 81920MiB |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| MIG devices:                                                                          |
+------------------+--------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                   Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                     BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG    |
|                  |                                |        ECC|                       |
|==================+================================+===========+=======================|
|  0    2   0   0  |              21MiB / 40448MiB  | 56      0 |  4   0    2    0    0 |
|                  |               0MiB / 65535MiB  |           |                       |
+------------------+--------------------------------+-----------+-----------------------+
|  1    0   0   0  |               1MiB / 81228MiB  | 98      0 |  7   0    5    1    1 |
|                  |               1MiB / 131072MiB |           |                       |
+------------------+--------------------------------+-----------+-----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+