Master issue: LLM Benchmark

[freedomtan]

What can we do for the default backend?

• Model: Let's start with llama 3.2 1B, preferably quantized ones
• Runtime:
  • MediaPipe and AI-Edge-Torch: our current default backend is tflite based, so if we can continue using TFLite-based solution, it would be good.
  • • new API in LiteRT (https://ai.google.dev/edge/litert)
  • ExecuTorch: https://github.com/pytorch/executorch, https://github.com/pytorch/executorch/blob/main/examples/models/llama/README.md
  • llama.cpp: https://github.com/ggerganov/llama.cpp
  • onnx runtime: https://github.com/microsoft/onnxruntime, let's check if it works on Android platforms.
  • onnx runtime genai: https://github.com/microsoft/onnxruntime-genai

[farook-edev]

I've looked into building each runtime for Android and here's what I found out:

TFLite

Google's warning about AI-Edge-Torch being experimental is quite the understatement. After much struggling I ended up getting the nightly versions of ai-edge-torch and ai-edge-quantizer which were released the the same day as the last edit to the llama example was made. That was the only way to get the thing to actually function.
Once it started it used up all 11GB my GPU had and promptly failed. Trying to get it to use the CPU ended with a C++ protobuf error.
So I gave up and downloaded pre-converted models from HuggingFace, and attempted to use them with MediaPipe. The android app compiled but would give a PERMISSION_DENIED error when attempting to load the model.

Edit

I managed to get a model loaded by using /data/ directory instead of /sdcard/ seems like Android had some protection for internal storage. it worked but the model did not respond to my prompt properly, instead giving python examples...

Executorch

Executorch seemed to have the most support and robust examples behind it. But unfortunately I couldn't even install the library's dependencies because of a cmake error regarding gflags. I want to look further into it to see if I can set it up locally since it's quite promising.

Edit

I was able to get Executorch working. The cmake error was resolved by removing my local gflags package, seems like it was conflicting with executorch's gflags in the venv.

Executorch has quite a bit of steps to get the .aar library compiled for android, which caused me trouble when using an NDK version lower than 25, their main guide for the llama example does mention that they support only version 27, but this isn't mentioned in the android demo app guide. I was also able to compile the library using NDK version 28.

Getting the models wasn't very straightforward either, since they needed to be converted from .pth to .pte. Thankfully executorch's converter worked without issue (unlike AI-Edge-Torch).

Running the app required upgrading gradle to 8.5 which thankfully Android Studio made quite straightforward, once that was done the app compiled and ran without issue.

I attempted to use the unquantized model first. Upon pressing load in the android app, my Galaxy S8 completely froze for 5 minutes until I restarted it. Possibly due to RAM capacity. Loading the quantized model did not cause any issues, and the model worked on my machine flawlessly.

This was done using the xnnpack backend, Executorch also supports Qualcomm and Mediatek AI engines which I did not test.

llama.cpp

working with llama.cpp has been the most straightforward, the app compiled and launched without issue, a couple of line changes allowed me to download whichever custome GGUF model I wanted, and the models actually functioned (They were however extremely slow on my Galaxy S8). There was an issue where manually loading the models into my phone caused them to not be readable, but was fixed by having the phone download the models instead of my PC.

Conclusion

My priority was TFLite because it's what we use. but it seems that both MediaPipe and AI-Edge-Torch are still in their infancy, and could cause problems when attempting to integrate them into the mobile app.
It's a similar story with ExecuTorch but in a different direction, I like the documentation and support, though it is quite complicated to get running, especially if you stray from the guides.
That leaves us with llama.cpp,models for which seem to be available. And while I've not attempted to convert my own (because HuggingFace has GGUF versions of the models), the models actually working without me having to dig through files upon files of code was a welcome change.

I'll look further into installing Executorch for now, but until I can get it running,
llama.cpp was the only runtime I could get to work without struggle. But Executorch's converter and android app (when they didn't crash the entire machine) were quite impressive.
It's worth mentioning while executorch is built around pytorch and its models, llama.cpp supports quite a few LLMs other than llama.

[freedomtan]

For model conversion with AI-Edge-Torch, as I noted at google-ai-edge/litert-torch#269 (comment)

• you don't need GPU
• you need a lot of DRAM, which is supposed to be a bug / feature of current AI-Edge-Torch implementation, (I don't remember the amount needed, supposedly 64 GiB is enough).
• I used Colab to do it.

[farook-edev]

I tried getting AI-Edge-Torch to work again, this time using a docker container. The code seemed to run without issue until it killed itself, seemingly due to lack of RAM (my machine only has 32GB). But with this setup a local CPU can be used to convert the models. It's just extremely finicky and easily prone to breaking.

[freedomtan]

@Mostelk: there is a GPT2 Android tflite app, https://github.com/huggingface/tflite-android-transformers/tree/master/gpt2
@mohitmundhragithub: there is https://github.com/mlcommons/mlperf_client which runs llama 2 7B

[freedomtan]

@freedomtan to check the GPT2 and the https://github.com/mlcommons/mlperf_client.

[freedomtan]

check that input format of huggingface tflite models (supposedly, you can simple cast integer token ids to fp16, setting the input tensors, if the expected input is token ids rather than some embeddeding after feeding token ids into some gather ops).

[freedomtan]

check that input format of huggingface tflite models (supposedly, you can simple cast integer token ids to fp16, setting the input tensors, if the expected input is token ids rather than some embeddeding after feeding token ids into some gather ops).

@farook-edev it turns out it's a bit complicated. If you check the tflite models, you can find that their inputs and outputs are not simple. There are

• input token and input pos: int32, that is token id (from tokenizer) and token pos (0, 1, 2, ...)
• KV cache matrixes: float32

How to inspect the models:

• visualization tools such as https://ai.google.dev/edge/model-explorer
• couple lines of code using python api

import tensorflow as tf
import sys

interpreter = tf.lite.Interpreter(model_path='/tmp/foo.tflite')
print(interpreter.get_input_details())

[farook-edev]

I managed to get input into and output out of the converted model. I used the python version of TFLite but will be moving the code to C++ since the test on python was successful.

The process is as follows:

1. Copy a single token id into the token tensor and set the pos (starting at 0)
2. run inference
3. increment pos by 1 and copy kv caches from output to input
4. repeat until all input tokens are consumed
5. get output logits and convert to token id
6. feed output token id into input
7. run inference
8. repeat step 3
9. repeat from 5 until a limit is hit or 128001 (end of text token) is found

One thing to note is that if you take a list of input tensors and output tensors, and remove any non kv cache tensors (token, pos, and logits), the remaining tensors are ordered in such a way that the output tensors map directly to the corresponding input tensors, for example:
output[0] -> input[0]
output[1] -> input[1]
output[2] -> input[2]
output[3] -> input[3]
...

Aside from that, @freedomtan I was wondering if our benchmark should include tokenization and detokenization, or will the datasets be in the form of tokens?

[freedomtan]

I managed to get input into and output out of the converted model. I used the python version of TFLite but will be moving the code to C++ since the test on python was successful.

The process is as follows:

1. Copy a single token id into the token tensor and set the pos (starting at 0)
2. run inference
3. increment pos by 1 and copy kv caches from output to input
4. repeat until all input tokens are consumed
5. get output logits and convert to token id
6. feed output token id into input
7. run inference
8. repeat step 3
9. repeat from 5 until a limit is hit or 128001 (end of text token) is found

One thing to note is that if you take a list of input tensors and output tensors, and remove any non kv cache tensors (token, pos, and logits), the remaining tensors are ordered in such a way that the output tensors map directly to the corresponding input tensors, for example: output[0] -> input[0] output[1] -> input[1] output[2] -> input[2] output[3] -> input[3] ...

Aside from that, @freedomtan I was wondering if our benchmark should include tokenization and detokenization, or will the datasets be in the form of tokens?

Like what we did for the Stable Diffusion 1.5, we can use tokens directly for benchmark. Which means we don't need tokenizaton/de-tokenization part. We haven't discussed the dataset, or say the inputs to be used for our benchmark, yet.

[farook-edev]

I've attempted to generate an int8 quantized model, but no matter how much I tried, the output was garbled and nothing like the fp16 one.

I should mention that I did not notice any difference in the shape, type, or sorting of the tensors, and used the same steps as the fp16 model. Please let me know if I'm missing something

For now I'll try to get a pipeline of some sort to run the non-quantized model using flutter. Unless there's something else that takes priority, if so, please let me know.

[freedomtan]

I've attempted to generate an int8 quantized model, but no matter how much I tried, the output was garbled and nothing like the fp16 one.

I should mention that I did not notice any difference in the shape, type, or sorting of the tensors, and used the same steps as the fp16 model. Please let me know if I'm missing something

For now I'll try to get a pipeline of some sort to run the non-quantized model using flutter. Unless there's something else that takes priority, if so, please let me know.

Quantization of small dense language models is not an easy task.

[freedomtan]

Gemma 3 could be another candidate https://blog.google/technology/developers/gemma-3/

[freedomtan]

please

1. test the 3b model
2. check out exactly what the quantization by the ai-edge-torch does.
3. use "standard" tflite quantization tool to quantize model.

[freedomtan]

@freedomtan to check if he can make the llama 3 quantized 1b work, too.

[freedomtan]

@farook-edev
I tested the quantized llama 3.2 1b it tflite model just now (March 26th, 2025) on a Mac mini m4. It worked as expected.

1. create a python venv environment for ai-edge-torch, python3.12 -m venv ai-edge-torch
2. source ai-edge-torch
3. pip install ai-edge-torch
4. mkdir work
5. cd work
6. git clone git@github.com:google-ai-edge/ai-edge-torch.git
7. download the 1b model from hugging face
8. cd ai_edge_torch
9. python ai_edge_torch/generative/examples/llama/convert_to_tflite.py --checkpoint_path THE_MODELCHECKPOINT (by default, a weights-only quantized model will be at /tmp/llama_q8_ekv1280.tflite)
10. build the text_generator example, bazel build -c opt //ai_edge_torch/generative/examples/cpp:text_generator_main
11. prepare llama 3.2 1b tokenizer, python ai_edge_torch/generative/tools/tokenizer_to_sentencepiece.py --output_path=llama3_1b.spm.model --checkpoint=meta-llama/Llama-3.2-1B-Instruct
12. use the text_generator_main to run the quantized model,

bazel-bin/ai_edge_torch/generative/examples/cpp/text_generator_main \
  --tflite_model=/tmp/llama_q8_ekv1280.tflite \
  --sentencepiece_model=llama3_1b.spm.model \
  --start_token="<bos>" --stop_token="<eos>" \
  --num_threads=16 \
  --prompt="Tell me something about systolic array."