[executorch][native-ggml] Support GGUF Q6_K PackedQuant weights

[digantdesai]

Stack from ghstack (oldest at bottom):

• [executorch][native-ggml] Split GgmlExecutor.cpp into lowering units #23786
• [executorch][native-ggml] Run integer elementwise arithmetic in F32 #23784
• [executorch][native-ggml-metal] Add Buck targets for the ggml engine on Metal #23783
• [executorch][native-ggml] Lower 1x1 convolutions without per-tap im2col #23782
• [executorch][native-ggml] Fold inference batch norm at load #23781
• [executorch][native-ggml] Attend only the key prefix a mask leaves open #23780
• [executorch][native-ggml] Pass the F32 query to F16-cache attention #23779
• [executorch][native-ggml] Keep strided permutations as views #23778
• [executorch][native-ggml] Lower layer_norm, gelu, constant_pad_nd, log, lt and conv1d #23777
• [executorch][native-vk] Keep constant_pad_nd input and output in one packed layout #23776
• [executorch][vulkan] Read zeros past a resized conv1d input's length #23775
• [executorch][native-vk] Pack convolution activations by channels #23774
• [executorch][native-vk] Run gelu_ and sigmoid_ out of place like relu_ #23773
• [executorch][native-vk] Keep layer_norm input and output in one packed layout #23772
• [executorch][native-vk] Copy views of dynamically shaped sources #23771
• [executorch][native-vk] Run in-place ops on views that nothing else reads #23770
• [executorch][native] Share one decoder-only LLM runner across native engines #23769
• [executorch][native] Test the same PTN across engines end to end #23768
• [executorch][native-ggml] Stage outputs in pinned host memory #23767
• [executorch][native-ggml] Write F32 rows straight into F16 caches #23766
• [executorch][native-ggml] Share storage for no-op copies #23765
• [executorch][native-ggml] Lower SiLU-gated products to SwiGLU #23764
• [executorch][native-ggml] Reuse integer casts and attention masks across ops #23763
• -> [executorch][native-ggml] Support GGUF Q6_K PackedQuant weights #23762
• [executorch][native-ggml] Replay ggml graphs for repeated input shapes #23761
• [executorch][native-ggml] Load ggml constants from package views without an extra copy #23760
• [executorch][native-ggml] Lower MobileNetV2 and ResNet50 ops on the ggml engine #23759
• [executorch][native-ggml] Run the ggml engine on any linked GPU backend, including HIP #23758
• [executorch][native-ggml] Test a tiny HuggingFace Llama end to end on ggml #23757
• [executorch][native-ggml] Lower HuggingFace Llama attention ops on the ggml engine #23756
• [executorch][native-ggml] Run torchao 8da4w weights on the ggml engine #23755
• [executorch][native-ggml] Broadcast the left operand of ggml binary ops #23754
• [executorch][native-ggml] Support rank-0 tensors in the ggml engine #23753
• [executorch][native-ggml] Add end-to-end op tests for the ggml engine #23752
• [executorch][native-ggml] Add a ggml engine #23751
• [executorch][native-vk] Fall back to the reference GGUF K-quant decoder #23750
• [executorch][native] Export HF LLMs with GGUF weights and an FP16 KV cache #23749
• [executorch][native] Map GGUF tensors onto Hugging Face modules #23748
• [executorch][native] Add a reference GGUF K-quant decoder #23747

GGUF Q4_K_M checkpoints store some weights, such as the token embedding and some projections, as Q6_K, which the ggml engine did not accept.

The ggml engine accepts GGUF Q6_K weights next to Q4_K and runs them on ggml's own Q6_K kernels, checking the block layout at load.

Authored with Claude and Codex assistance.

Differential Revision: D123796929

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23762

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job, 1 Unrelated Failure

As of commit 2142395 with merge base 0a4a05e ():

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / test-qnn-python-imports-linux / linux-job (gh)
  ##[error]received SIGINT

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• Propose to merge ghstack orig PRs to main / Try to create a PR with ghstack /orig branch (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.