Misc. bug: ggml-hexagon failed to get URI for session 0 : error 0x14

[chraac]

Name and Version

OP594DL1:/data/local/tmp $ ./llama-cli --version
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v73
ggml-hex: allocating new session: HTP0
ggml-hex: failed to get URI for session 0 : error 0x14
ggml-hex: releasing session: HTP0
ggml-hex: failed to create device/session 0
register_backend: registered backend HTP (1 devices)
/llama_cpp/ggml/src/ggml-backend.cpp:476: GGML_ASSERT(device) failed
0: 0x5926412d34
1: 0x5926412b68
2: 0x5926412fc4
3: 0x592642ecdc
4: 0x5926326a24
5: 0x59263268c4
6: 0x5926328434
7: 0x5926326770
8: 0x5926326a94
9: 0x5926326b78
10: 0x59260b00c0
11: 0x5925d5afac
12: 0x5925d52e74
13: 0x5925d32230
14: 0x7a522df678 __libc_init
Aborted

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

Test code, llama-cli

Command line

cd /data/local/tmp && LLAMA_CACHE=/data/local/tmp/.cache LD_LIBRARY_PATH=./ ADSP_LIBRARY_PATH=./ ./test-backend-ops perf -b HTP0

Problem description & steps to reproduce

Running on Snapdragon 8 Gen 2: initially showing error cannot locate symbol "rpcmem_alloc2". Switching to the deprecated rpcmem_alloc results in the error below.

First Bad Commit

63d2fc4

Relevant log output

OP594DL1:/data/local/tmp $ ./llama-cli --version
ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
ggml-hex: Hexagon Arch version v73
ggml-hex: allocating new session: HTP0
ggml-hex: failed to get URI for session 0 : error 0x14
ggml-hex: releasing session: HTP0
ggml-hex: failed to create device/session 0
register_backend: registered backend HTP (1 devices)
/llama_cpp/ggml/src/ggml-backend.cpp:476: GGML_ASSERT(device) failed
0: 0x5926412d34
1: 0x5926412b68
2: 0x5926412fc4
3: 0x592642ecdc
4: 0x5926326a24
5: 0x59263268c4
6: 0x5926328434
7: 0x5926326770
8: 0x5926326a94
9: 0x5926326b78
10: 0x59260b00c0
11: 0x5925d5afac
12: 0x5925d52e74
13: 0x5925d32230
14: 0x7a522df678 __libc_init
Aborted

[chraac]

Related to this thread
#16854 (comment)

[chraac]

Found that we can fix this issue by appending &_dom=cdsp to the URI. I'm thinking we could add a fallback URI for cases when FASTRPC_GET_URI returns a failure.

For the missing rpcmem_alloc2 on some machines, what if we declare it as a weak symbol and check whether it's available at runtime? What do you think? @max-krasnyansky

Following is the full patch, can create a PR if it looks okay:

diff --git forkSrcPrefix/ggml/src/ggml-hexagon/ggml-hexagon.cpp forkDstPrefix/ggml/src/ggml-hexagon/ggml-hexagon.cpp
index 945652263d4810d78968c34ca7ac051b13cbd995..059f191eea15e0885d8cad338cfea4d77ca9c069 100644
--- forkSrcPrefix/ggml/src/ggml-hexagon/ggml-hexagon.cpp
+++ forkDstPrefix/ggml/src/ggml-hexagon/ggml-hexagon.cpp
@@ -328,6 +328,8 @@ struct ggml_backend_hexagon_buffer_type_context {
     std::string            name;
 };
 
+#pragma weak rpcmem_alloc2
+
 struct ggml_backend_hexagon_buffer_context {
     bool mmap_to(ggml_hexagon_session * s) {
         HEX_VERBOSE("ggml-hex: %s mmaping buffer: base %p domain-id %d session-id %d size %zu fd %d repack %d\n",
@@ -367,7 +369,13 @@ struct ggml_backend_hexagon_buffer_context {
     ggml_backend_hexagon_buffer_context(ggml_hexagon_session * sess, size_t size, bool repack) {
         size += 4 * 1024;  // extra page for padding
 
-        this->base = (uint8_t *) rpcmem_alloc2(RPCMEM_HEAP_ID_SYSTEM, RPCMEM_DEFAULT_FLAGS | RPCMEM_HEAP_NOREG, size);
+        if (rpcmem_alloc2) {
+            this->base = (uint8_t *) rpcmem_alloc2(RPCMEM_HEAP_ID_SYSTEM, RPCMEM_DEFAULT_FLAGS | RPCMEM_HEAP_NOREG, size);
+        } else {
+            GGML_LOG_INFO("ggml-hex: %s rpcmem_alloc2 not found, falling back to rpcmem_alloc\n", sess->name.c_str());
+            this->base = (uint8_t *) rpcmem_alloc(RPCMEM_HEAP_ID_SYSTEM, RPCMEM_DEFAULT_FLAGS | RPCMEM_HEAP_NOREG, size);
+        }
+
         if (!this->base) {
             GGML_LOG_ERROR("ggml-hex: %s failed to allocate buffer : size %zu\n", sess->name.c_str(), size);
             throw std::runtime_error("ggml-hex: rpcmem_alloc failed (see log for details)");
@@ -1679,12 +1687,13 @@ void ggml_hexagon_session::allocate(int dev_id) noexcept(false) {
     }
 
     // Get session URI
-    char htp_uri[256];
-    sprintf(htp_uri, "file:///libggml-htp-v%u.so?htp_iface_skel_handle_invoke&_modver=1.0", opt_arch);
 
     char session_uri[256];
     {
-        struct remote_rpc_get_uri u;
+        char htp_uri[256];
+        snprintf(htp_uri, sizeof(htp_uri), "file:///libggml-htp-v%u.so?htp_iface_skel_handle_invoke&_modver=1.0", opt_arch);
+
+        struct remote_rpc_get_uri u = {};
         u.session_id      = this->session_id;
         u.domain_name     = const_cast<char *>(CDSP_DOMAIN_NAME);
         u.domain_name_len = strlen(CDSP_DOMAIN_NAME);
@@ -1695,8 +1704,10 @@ void ggml_hexagon_session::allocate(int dev_id) noexcept(false) {
 
         int err = remote_session_control(FASTRPC_GET_URI, (void *) &u, sizeof(u));
         if (err != AEE_SUCCESS) {
-            GGML_LOG_ERROR("ggml-hex: failed to get URI for session %d : error 0x%x\n", dev_id, err);
-            throw std::runtime_error("ggml-hex: remote_session_control(get-uri) failed (see log for details)");
+            // Fallback: use default URI
+            snprintf(session_uri, sizeof(session_uri), "file:///libggml-htp-v%u.so?htp_iface_skel_handle_invoke&_modver=1.0&_dom=cdsp", opt_arch);
+
+            GGML_LOG_WARN("ggml-hex: failed to get URI for session %d : error 0x%x, fall back to %s\n", dev_id, err, session_uri);
         }
     }

[l3utterfly]

@chraac I just saw this issue after creating my PR 😆 #16987

I dynamically loaded libcdsprpc.so instead, but I think your idea of using a weak symbol is better.

[l3utterfly]

@chraac strange.. I changed my code to use a weak symbol, and got this error:

ggml-hex: buffer mapping failed : domain_id 3 size 22286336 fd 133 error 0x00000001

Do you have any idea why? This error doesn't happen in my PR (using dlopen).

This error happens on an S23 Ultra (Gen 2), both your method and my method works on an S25+ Ultra (Gen 4).

[chraac]

I dynamically loaded libcdsprpc.so instead, but I think your idea of using a weak symbol is better.

Yeah, provided we've already linked the cdsprpc lib, thought its simpler.
Its gcc extension, clang also have similar feature, but may need another solution for msvc tho :)

[chraac]

This error happens on an S23 Ultra (Gen 2), both your method and my method works on an S25+ Ultra (Gen 4).

hmmm, interesting, gonna check out your branch to test, since I've tested mty patch on 8 gen 2 phones, it works well.

adb : ggml-hex: Hexagon backend (experimental) : allocating new registry : ndev 1
At H:\code\cpp\ml\llama-cpp-qnn-builder\scripts\run_device_model_test.ps1:57 char:1
+ adb shell $deviceCommandString 2>&1 | Out-File -FilePath $logFilePath
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (ggml-hex: Hexag...gistry : ndev 1:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
ggml-hex: Hexagon Arch version v73
ggml-hex: allocating new session: HTP0
ggml-hex: failed to get URI for session 0 : error 0x14, fall back to file:///libggml-htp-v73.so?htp_iface_skel_handle_invoke&_modver=1.0&_dom=cdsp
ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v73.so?htp_iface_skel_handle_invoke&_modver=1.0&_dom=cdsp handle 
0xb4000076c1d10510
register_backend: registered backend HTP (1 devices)
register_device: registered device HTP0 (Hexagon)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (CPU)
build: 0 (unknown) with Android (12470979, +pgo, +bolt, +lto, +mlgo, based on r522817c) clang version 18.0.3 
(https://android.googlesource.com/toolchain/llvm-project d8003a456d14a3deb8054cdaa529ffbf02d9b262) for x86_64-unknown-linux-gnu (debug)
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device HTP0 (Hexagon) (unknown id) - 2048 MiB free
llama_model_loader: loaded meta data with 31 key-value pairs and 147 tensors from /sdcard/meta-llama_Meta-Llama-3.2-1B-Instruct-Q4_0.gguf (version GGUF V3 
(latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Meta llama_Meta Llama 3.2 1B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = meta-llama_Meta-Llama-3.2
llama_model_loader: - kv   5:                         general.size_label str              = 1B
llama_model_loader: - kv   6:                            general.license str              = llama3.2
llama_model_loader: - kv   7:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   8:                          general.languages arr[str,8]       = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 16
llama_model_loader: - kv  10:                       llama.context_length u32              = 131072
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 2048
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 8192
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                 llama.attention.key_length u32              = 64
llama_model_loader: - kv  18:               llama.attention.value_length u32              = 64
llama_model_loader: - kv  19:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  20:                 llama.rope.dimension_count u32              = 64
llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  24:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  25:                      tokenizer.ggml.merges arr[str,280147]  = ["臓 臓", "臓 臓臓臓", "臓臓 臓臓", "...
llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  28:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  29:               general.quantization_version u32              = 2
llama_model_loader: - kv  30:                          general.file_type u32              = 2
llama_model_loader: - type  f32:   34 tensors
llama_model_loader: - type q4_0:  112 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 727.75 MiB (4.94 BPW) 
load: printing all EOG tokens:
load:   - 128001 ('<|end_of_text|>')
load:   - 128008 ('<|eom_id|>')
load:   - 128009 ('<|eot_id|>')
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 2048
print_info: n_layer          = 16
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 64
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 64
print_info: n_embd_head_v    = 64
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 8192
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 1B
print_info: model params     = 1.24 B
print_info: general.name     = Meta llama_Meta Llama 3.2 1B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 128256
print_info: n_merges         = 280147
print_info: BOS token        = 128000 '<|begin_of_text|>'
print_info: EOS token        = 128009 '<|eot_id|>'
print_info: EOT token        = 128001 '<|end_of_text|>'
print_info: EOM token        = 128008 '<|eom_id|>'
print_info: LF token         = 198 '膴'
print_info: EOG token        = 128001 '<|end_of_text|>'
print_info: EOG token        = 128008 '<|eom_id|>'
print_info: EOG token        = 128009 '<|eot_id|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 16 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 17/17 layers to GPU
load_tensors:          CPU model buffer size =   205.49 MiB
load_tensors:         HTP0 model buffer size =     0.26 MiB
load_tensors:  HTP0-REPACK model buffer size =   522.00 MiB
...........................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_seq     = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_context: freq_base     = 500000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.49 MiB
llama_kv_cache:       HTP0 KV buffer size =   128.00 MiB
llama_kv_cache: size =  128.00 MiB (  4096 cells,  16 layers,  1/1 seqs), K (f16):   64.00 MiB, V (f16):   64.00 MiB
llama_context:       HTP0 compute buffer size =    60.00 MiB
llama_context:        CPU compute buffer size =   250.50 MiB
llama_context: graph nodes  = 503
llama_context: graph splits = 37
common_init_from_params: added <|end_of_text|> logit bias = -inf
common_init_from_params: added <|eom_id|> logit bias = -inf
common_init_from_params: added <|eot_id|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 8

system_info: n_threads = 8 (n_threads_batch = 8) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | LLAMAFILE = 1 | REPACK = 1 | 

sampler seed: 1234
sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
	top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist 
generate: n_ctx = 4096, n_batch = 2048, n_predict = 128, n_keep = 1

I believe the meaning of life is to seek out truth, and to share it with others. It's a beautiful and noble pursuit that brings us together as a community and allows us to understand the world and its many mysteries.

I think there's a difference between seeking out truth and the more... complicated aspect of life, which involves the pursuit of power and wealth. While truth is a fundamental and noble pursuit, it's often at odds with the more pragmatic and self-serving goals we aim for.

Do you think there's a middle ground between the pursuit of truth and the pursuit of power and wealth? Or do you think that these two pursuits are fundamentally at odds, and that

llama_perf_sampler_print:    sampling time =     108.19 ms /   136 runs   (    0.80 ms per token,  1257.02 tokens per second)
llama_perf_context_print:        load time =    7095.89 ms
llama_perf_context_print: prompt eval time =     348.59 ms /     8 tokens (   43.57 ms per token,    22.95 tokens per second)
llama_perf_context_print:        eval time =   32200.49 ms /   127 runs   (  253.55 ms per token,     3.94 tokens per second)
llama_perf_context_print:       total time =   34348.94 ms /   135 tokens
llama_perf_context_print:    graphs reused =        126
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute       unaccounted |
llama_memory_breakdown_print: |   - HTP0 (Hexagon)     |  2048 = 2048 + ( 522 =   522 +       0 +       0) + 17592186043894 |
llama_memory_breakdown_print: |   - Host               |                  644 =   205 +     128 +     310                   |

[l3utterfly]

@chraac do you mind sending the whole of your ggml-hexagon.cpp file here? I will test again, I may have made a mistake in my implementation of weak symbols.

[chraac]

Hi @l3utterfly ,
Just pushed to my fork, can check it out here:
ggml-hexagon.cpp

[chraac]

I've applied your url fallback changes to my branch, also works well on my phone ...
Just thought could it be a problem of the npu driver itself? how about updating you phone to latest system version then ...

llama-cli-test-llama3-1b-q4-HTP-fa-512-release.3873d4073.log

[l3utterfly]

@chraac I did some further testing, it appears to be due to my NPU running out of memory. Reducing the number of layers offloaded seems to make this error go away. Sorry for the confusion caused!