Llava 1.5 poor output quality in iOS app

[jakmro]

🐛 Describe the bug

Description

I tried running Llava on iOS following this tutorial. However, the model performs poorly inside the app (see results below). When I run the model using the C++ runner with the same image, the results are accurate.

I used a Linux machine to export the model because I encountered this issue on my Mac.

Results

iOS app result

C++ runner result

What's on the image?I 00:00:26.959272 executorch:text_prefiller.cpp:53] Prefill token result numel(): 32064

The image features a large, happy-looking dog sitting in a lush green field. The dog appears to be enjoying the outdoors, possibly on a sunny day. The field is filled with green grass, providing a perfect setting for the dog to relax and take in the surroundings.</s>

Minimal Example

1. Follow this tutorial, to run the model inside the iOS simulator.
2. Build and run the model using c++ runner.
3. I've also used a different image, if you want to use the same one make sure to go inside ./.ci/scripts/test_llava.sh and change this line of code:

curl -o basketball.jpg https://upload.wikimedia.org/wikipedia/commons/7/73/Chicago_Bulls_and_New_Jersey_Nets%2C_March_28%2C_1991.jpg

to this one:

curl -o basketball.jpg https://upload.wikimedia.org/wikipedia/commons/1/18/Dog_Breeds.jpg

Versions

MacBook Pro 14

Collecting environment information...
PyTorch version: 2.7.0.dev20250131
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 15.1.1 (arm64)
GCC version: Could not collect
Clang version: 16.0.0 (clang-1600.0.26.6)
CMake version: version 3.31.6
Libc version: N/A

Python version: 3.10.0 (default, Mar 3 2022, 03:54:28) [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-15.1.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] executorch==0.6.0a0+e2201c5
[pip3] numpy==2.2.3
[pip3] torch==2.7.0.dev20250131
[pip3] torchao==0.10.0+git7d879462
[pip3] torchaudio==2.6.0.dev20250131
[pip3] torchsr==1.0.4
[pip3] torchvision==0.22.0.dev20250131
[conda] executorch 0.6.0a0+e2201c5 pypi_0 pypi
[conda] numpy 2.2.3 pypi_0 pypi
[conda] torch 2.7.0.dev20250131 pypi_0 pypi
[conda] torchao 0.10.0+git7d879462 pypi_0 pypi
[conda] torchaudio 2.6.0.dev20250131 pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchvision 0.22.0.dev20250131 pypi_0 pypi

Linux Machine

Collecting environment information...
PyTorch version: 2.7.0.dev20250131+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: glibc-2.35

Python version: 3.10.0 (default, Mar 3 2022, 09:58:08) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 535.183.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 5900X 12-Core Processor
CPU family: 25
Model: 33
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 1
Stepping: 2
Frequency boost: enabled
CPU max MHz: 4950,1948
CPU min MHz: 2200,0000
BogoMIPS: 7385.70
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap
L1d cache: 384 KiB (12 instances)
L1i cache: 384 KiB (12 instances)
L2 cache: 6 MiB (12 instances)
L3 cache: 64 MiB (2 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-23
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] executorch==0.6.0a0+e2201c5
[pip3] numpy==2.2.3
[pip3] torch==2.7.0.dev20250131+cpu
[pip3] torchao==0.10.0+git7d879462
[pip3] torchaudio==2.6.0.dev20250131+cpu
[pip3] torchsr==1.0.4
[pip3] torchvision==0.22.0.dev20250131+cpu
[conda] executorch 0.6.0a0+e2201c5 pypi_0 pypi
[conda] numpy 2.2.3 pypi_0 pypi
[conda] torch 2.7.0.dev20250131+cpu pypi_0 pypi
[conda] torchao 0.10.0+git7d879462 pypi_0 pypi
[conda] torchaudio 2.6.0.dev20250131+cpu pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchvision 0.22.0.dev20250131+cpu pypi_0 pypi

cc @shoumikhin @cbilgin @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng

[mergennachin]

How performant is it (e.g., token/s) when running the C++ runner on your computer?

Cc @shoumikhin , @larryliu0820

[jakmro]

I updated the issue title to avoid confusion with performance in terms of speed. My original concern was the quality of the model's output.

Here are the complete logs from the C++ runner.

(et_xnnpack) jakubmroz@jm executorch % cmake-out/examples/models/llava/llava_main \
    --model_path=llava.pte                 \
    --tokenizer_path=tokenizer.bin         \
    --image_path=image2.pt                  \
    --prompt="What's on the image?" \
    --seq_len=768                          \
    --temperature=0
I 00:00:00.003718 executorch:cpuinfo_utils.cpp:62] Reading file /sys/devices/soc0/image_version
I 00:00:00.003837 executorch:cpuinfo_utils.cpp:78] Failed to open midr file /sys/devices/soc0/image_version
I 00:00:00.003841 executorch:cpuinfo_utils.cpp:158] Number of efficient cores 4
I 00:00:00.003842 executorch:main.cpp:78] Resetting threadpool with num threads = 4
I 00:00:00.004730 executorch:multimodal_runner.h:49] Creating Multimodal LLM runner: model_path=llava.pte, tokenizer_path=tokenizer.bin
I 00:00:00.025559 executorch:main.cpp:112] image size(0): 3, size(1): 252, size(2): 336
I 00:00:08.649390 executorch:llava_runner.cpp:144] RSS after loading model: 0.000000 MiB (0 if unsupported)
I 00:00:11.367456 executorch:text_prefiller.cpp:53] Prefill token result numel(): 32064
I 00:00:24.638151 executorch:llava_runner.cpp:168] RSS after prompt and image prefill: 0.000000 MiB (0 if unsupported)
What's on the image?I 00:00:26.959272 executorch:text_prefiller.cpp:53] Prefill token result numel(): 32064

The image features a large, happy-looking dog sitting in a lush green field. The dog appears to be enjoying the outdoors, possibly on a sunny day. The field is filled with green grass, providing a perfect setting for the dog to relax and take in the surroundings.</s>
I 00:03:06.608756 executorch:text_token_generator.h:118] 
Reached to the end of generation
PyTorchObserver {"prompt_tokens":618,"generated_tokens":64,"model_load_start_ms":1741788656861,"model_load_end_ms":1741788665484,"inference_start_ms":1741788665484,"inference_end_ms":1741788843446,"prompt_eval_end_ms":1741788683794,"first_token_ms":1741788683794,"aggregate_sampling_time_ms":3,"SCALING_FACTOR_UNITS_PER_SECOND":1000}
I 00:03:06.608818 executorch:stats.h:110] 	Prompt Tokens: 618    Generated Tokens: 64
I 00:03:06.608819 executorch:stats.h:116] 	Model Load Time:		8.623000 (seconds)
I 00:03:06.608827 executorch:stats.h:126] 	Total inference time:		177.962000 (seconds)		 Rate: 	0.359627 (tokens/second)
I 00:03:06.608829 executorch:stats.h:134] 		Prompt evaluation:	18.310000 (seconds)		 Rate: 	33.752048 (tokens/second)
I 00:03:06.608831 executorch:stats.h:145] 		Generated 64 tokens:	159.652000 (seconds)		 Rate: 	0.400872 (tokens/second)
I 00:03:06.608833 executorch:stats.h:153] 	Time to first generated token:	18.310000 (seconds)
I 00:03:06.608834 executorch:stats.h:160] 	Sampling time over 682 tokens:	0.003000 (seconds)
I 00:03:06.608886 executorch:llava_runner.cpp:180] RSS after finishing text generation: 0.000000 MiB (0 if unsupported)

[mergennachin]

I see what you mean

• with C++ runner, it is saying "The image features a large, happy-looking dog sitting in a lush green field" which is correct
• with iOS, it is saying "the image, there's a view of the sky..." which is incorrect

So that means, there's something wrong with the iOS integration somewhere

Cc @shoumikhin - this might be on your plate.

[GregoryComer]

I'm tentatively assigning to @shoumikhin based on Mergen's comment, but feel free to let me know if there is a better owner. It looks like the problem is there is some numerical divergence between iOS and running on host. That could plausibly be due to the iOS integration (as Mergen mentioned), or differences in the kernels.