[Android/XNNPACK] SIGSEGV in XNNWeightsCache::look_up_or_insert during memcmp on MediaTek Dimensity 6100+ (Galaxy M15)

[lordlugo]

🐛 Describe the Bug

1. System Information

Field	Detail
ExecuTorch Version	1.1.0
OS Platform	Android 14 (One UI 6)
Target Architecture	arm64-v8a
Crashing Hardware	Samsung Galaxy M15 5G (MediaTek Dimensity 6100+ / Mali-G57 MC2)
Working Control Hardware	Samsung Galaxy A16 4G (MediaTek Helio G99 / Mali-G57 MC2)
RAM Variants Tested	Issue occurs on 4 GB / 6 GB / 8 GB variants of the M15

---

2. Expected Behavior

When calling module.forward() on a background HandlerThread, the XNNPACK delegate should successfully parse the memory-mapped .pte file, repack the convolution weights into the XNNWeightsCache, and execute the inference graph. This exact behavior succeeds flawlessly over 100 consecutive times on the control device (Galaxy A16).

---

3. Actual Behavior & Crash Log

On the Galaxy M15, the application instantly terminates with a native Segmentation Fault (SIGSEGV) during the XNNPACK backend initialization phase. The crash specifically occurs deep within the C standard library's memory comparison function (__memcmp_aarch64) while XNNPACK is attempting to index the weights cache.

Backtrace

pid: 0, tid: 13533 >>> com.borzai.vu <<<
backtrace:

#00 pc 0x00000000000a398c  /apex/com.android.runtime/lib64/bionic/libc.so (__memcmp_aarch64+12)

#01 pc 0x0000000000362fa8  /data/app/.../split_config.arm64_v8a.apk!libexecutorch_jni.so

(executorch::backends::xnnpack::delegate::XNNWeightsCache::look_up_or_insert(

executorch::backends::xnnpack::delegate::XNNWeightsCache*,

xnn_weights_cache_look_up_key const*, void*, unsigned long)+92)

#02 pc 0x0000000000403dc8  /data/app/.../split_config.arm64_v8a.apk!libexecutorch_jni.so ...

#04 pc 0x00000000004019cc  /data/app/.../split_config.arm64_v8a.apk!libexecutorch_jni.so

(create_convolution2d_nhwc_f32+700)

#05 pc 0x0000000000401b20  /data/app/.../split_config.arm64_v8a.apk!libexecutorch_jni.so

(xnn_create_convolution2d_nhwc_f32+264)

...

#08 pc 0x00000000003603b0  /data/app/.../split_config.arm64_v8a.apk!libexecutorch_jni.so

(executorch::backends::xnnpack::delegate::XNNCompiler::compileModel(

void const*, unsigned long,

executorch::backends::xnnpack::delegate::XNNExecutor*,

executorch::backends::xnnpack::delegate::XNNWeightsCache*,

xnn_workspace*, executorch::runtime::NamedDataMap const*)+1340)

#09 pc 0x0000000000362274  /data/app/.../split_config.arm64_v8a.apk!libexecutorch_jni.so

(executorch::backends::XnnpackBackend::init(

executorch::runtime::BackendInitContext&,

executorch::runtime::FreeableBuffer*,

executorch::runtime::ArrayRef<executorch::runtime::CompileSpec>) const+176)

...

#14 pc 0x0000000000393c50  /data/app/.../split_config.arm64_v8a.apk!libexecutorch_jni.so

(executorch::extension::module::Module::execute(...))

#20 pc 0x000000000044a0fa  /data/app/.../base.apk

(org.pytorch.executorch.Module.execute+58)

#24 pc 0x0000000000d404da  /data/app/.../base.apk

(com.borzai.vu.MainActivity.runInferenceOnThread+90)

---

4. Context and Architectural Analysis

The underlying cause might be hardware/kernel-specific rather than a universal memory leak, shown by the successful execution on the structurally similar Helio G99 (A16). Both processors utilize the identical ARM Cortex-A76/A55 Big.LITTLE architecture and share the 4 GB RAM baseline.

Given that memcmp triggers the fault, a null pointer or misaligned memory read is being passed into look_up_or_insert.

cc @GregoryComer @digantdesai @cbilgin @kirklandsign

[lucylq]

Hey @lordlugo, do you have a repro that we could take a look at - the model and lowering steps you took to generate the .pte file?

[GregoryComer]

I think I know what the issue is here. I can take a look. Like Lucy mentioned, if you have a concrete repro, that would be helpful. Thanks!

[lordlugo]

Hi @lucylq and @GregoryComer, thanks for looking into this!

Here are the requested repro artifacts:

1. Compiled Model

You can download the generated .pte file that triggers the crash here:
[test.pte](https://vuapp.nyc3.cdn.digitaloceanspaces.com/VU/test.pte)

2. Lowering / Conversion Script

Below is the exact code I used to export, lower, and save the model using the XnnpackPartitioner:

import torch
from executorch.exir import to_edge_transform_and_lower
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner

device = "cpu"

# 1. Export PyTorch model
model_path = 'best_model.pth'
model = load_flow_model(model_path)
model.eval().to(device)

example_inputs = (torch.randn(1, 3, 400, 400).to(device),)
exported_program = torch.export.export(model, example_inputs)

# 2. Optimize for target hardware (switch backends with one line)
program = to_edge_transform_and_lower(
    exported_program,
    partitioner=[XnnpackPartitioner()]  # CPU | CoreMLPartitioner() for iOS | QnnPartitioner() for Qualcomm
).to_executorch()

# 3. Save for deployment
with open("test.pte", "wb") as f:
    f.write(program.buffer)

Let me know if you need any additional info

[lucylq]

@lordlugo thanks for the lowering code it looks fairly straightforward - do you also have the definition for load_flow_model, and the artifact for best_model.pth?

[lordlugo]

Thanks @lucylq Here is the architecture definition and the model artifact.

1. Model Artifact

You can download the test.pth weights here: LINK

2. Model Architecture & load_flow_model

Click to expand Model Definition

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn.utils import spectral_norm, remove_spectral_norm
from transformers import AutoModel
import math

# Load your trained model
def load_flow_model(model_path):
    device = "cpu"
    model = SegDINO(2)
    
    # Load state dict
    checkpoint = torch.load(model_path, map_location=device)
    
    # Handle different checkpoint formats
    if "model_state_dict" in checkpoint:
        state_dict = checkpoint["model_state_dict"]
    elif "model" in checkpoint:
        state_dict = checkpoint["model"]
    else:
        state_dict = checkpoint  # Assume it's the raw state_dict
    
    # Fix key mismatches if needed
    fixed_state_dict = {}
    for key, value in state_dict.items():
        new_key = key
        if key.startswith("module."):
            new_key = key[7:]
        elif key.startswith("_orig_mod."):
            new_key = key[len("_orig_mod."):]
        fixed_state_dict[new_key] = value
    
    # Load the state dict with strict=False to allow partial loading
    model.load_state_dict(fixed_state_dict, strict=False)
    model.to(device)
    model.eval()
    
    return model

class RobustGlobalConsistencyBlock(nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.project_stats = nn.Sequential(
            nn.Linear(in_dim * 2, out_dim),
            nn.LayerNorm(out_dim),
            nn.GELU(),
            nn.Linear(out_dim, out_dim),
            nn.Sigmoid() 
        )
        self.fusion_conv = nn.Sequential(
            nn.Conv2d(in_dim + out_dim, out_dim, kernel_size=1),
            nn.GroupNorm(32, out_dim),
            nn.GELU()
        )

    def forward(self, x):
        b, c, h, w = x.shape
        flat_x = x.view(b, c, -1)
        
        center_val = flat_x.mean(dim=2)
        spread_val = flat_x.std(dim=2)
        
        stats = torch.cat([center_val, spread_val], dim=1)
        global_context = self.project_stats(stats) 
        
        global_context = global_context.view(b, -1, 1, 1).expand(-1, -1, h, w)
        out = torch.cat([x, global_context], dim=1)
        return self.fusion_conv(out)


class LearnableDoG(nn.Module):
    def __init__(self, out_dim, semantic_dim=None):
        super().__init__()
        def get_gaussian_kernel(kernel_size=5, sigma=1.0, channels=3):
            x_coord = torch.arange(kernel_size)
            x_grid = x_coord.repeat(kernel_size).view(kernel_size, kernel_size)
            y_grid = x_grid.t()
            xy_grid = torch.stack([x_grid, y_grid], dim=-1).float()
            mean = (kernel_size - 1)/2.
            variance = sigma**2.
            gaussian_kernel = (1./(2.*math.pi*variance)) * torch.exp(
                -torch.sum((xy_grid - mean)**2., dim=-1) / (2*variance)
            )
            gaussian_kernel = gaussian_kernel / torch.sum(gaussian_kernel)
            return gaussian_kernel.view(1, 1, kernel_size, kernel_size).repeat(channels, 1, 1, 1)

        self.conv_narrow = nn.Conv2d(3, 3, kernel_size=5, padding=2, groups=3, bias=False)
        self.conv_narrow.weight.data = get_gaussian_kernel(5, sigma=1.0)
        self.conv_wide = nn.Conv2d(3, 3, kernel_size=9, padding=4, groups=3, bias=False)
        self.conv_wide.weight.data = get_gaussian_kernel(9, sigma=2.0)
        
        self.processor = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3, padding=1, bias=False), 
            nn.BatchNorm2d(16),
            nn.GELU(),
            nn.Conv2d(16, out_dim, kernel_size=1)
        )
        
        if semantic_dim is not None:
            self.semantic_gate = nn.Sequential(
                nn.Conv2d(semantic_dim, out_dim, kernel_size=1),
                nn.Sigmoid()
            )
        else:
            self.semantic_gate = None
        
    def forward(self, raw_img, target_h, target_w, semantic_features=None):
        x = F.interpolate(raw_img, size=(target_h, target_w), mode='bilinear', align_corners=False)
        high_freq = self.conv_narrow(x)
        low_freq = self.conv_wide(x)
        dog_features = self.processor(high_freq - low_freq)
        
        if self.semantic_gate is not None and semantic_features is not None:
            gate = self.semantic_gate(semantic_features)
            dog_features = dog_features * gate
        
        return dog_features


class ContextGatedBlock(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.detail_conv = nn.Sequential(
            nn.Conv2d(dim, dim, kernel_size=3, padding=1, groups=dim),
            nn.BatchNorm2d(dim),
            nn.Conv2d(dim, dim * 2, 1), 
            nn.GELU(),
            nn.Conv2d(dim * 2, dim, 1)
        )
        self.context_gate = nn.Sequential(
            nn.Conv2d(dim, dim, kernel_size=11, padding=5, groups=dim, bias=False),
            nn.BatchNorm2d(dim),
            nn.Conv2d(dim, dim, kernel_size=1), 
            nn.Sigmoid() 
        )
        self.out_proj = nn.Conv2d(dim, dim, 1)

    def forward(self, x):
        residual = x
        details = self.detail_conv(x)
        gate = self.context_gate(x)
        x = details * gate 
        return self.out_proj(x + residual)


class SpectralUpsampleBlock(nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.net = nn.Sequential(
            spectral_norm(nn.Conv2d(in_dim, out_dim * 4, kernel_size=3, padding=1)),
            nn.PixelShuffle(2), 
            nn.GroupNorm(32, out_dim),
            nn.GELU()
        )
        
    def forward(self, x):
        return self.net(x)


class ExpressiveRobustDecoder(nn.Module):
    def __init__(self, embed_dim, num_classes, num_multiscale_inputs=4, decoder_dim=256, dropout=0.1):
        super().__init__()
        self.projectors = nn.ModuleList([
            nn.Sequential(
                nn.Conv2d(embed_dim, decoder_dim, 1),
                nn.GroupNorm(32, decoder_dim),
                nn.GELU()
            ) for _ in range(num_multiscale_inputs)
        ])
        
        self.noise_injector = LearnableDoG(decoder_dim, semantic_dim=decoder_dim)
        self.global_calibrator = RobustGlobalConsistencyBlock(decoder_dim, decoder_dim)
        
        fusion_in_dim = (num_multiscale_inputs + 1) * decoder_dim 
        self.fusion = nn.Sequential(
            nn.Conv2d(fusion_in_dim, decoder_dim, 1),
            nn.GroupNorm(32, decoder_dim),
            nn.GELU()
        )
        
        self.cortex = nn.Sequential(
            ContextGatedBlock(decoder_dim),
            ContextGatedBlock(decoder_dim)
        )
        
        self.up_block1 = SpectralUpsampleBlock(decoder_dim, decoder_dim)
        self.up_block2 = SpectralUpsampleBlock(decoder_dim, decoder_dim)
        
        self.head = nn.Conv2d(decoder_dim, num_classes, kernel_size=1)
        self.dropout = nn.Dropout2d(dropout)

    def forward(self, features_list, original_image):
        projections = [proj(f) for proj, f in zip(self.projectors, features_list)]
        target_H, target_W = projections[0].shape[2:]
        
        resized_projections = []
        for p in projections:
            p_resized = F.interpolate(p, size=(target_H, target_W), mode='bilinear', align_corners=False)
            resized_projections.append(p_resized)

        semantic_context = resized_projections[-1]
        raw_noise = self.noise_injector(original_image, target_H, target_W, semantic_context)
        calibrated_noise = self.global_calibrator(raw_noise)
        resized_projections.append(calibrated_noise)
        
        x = torch.cat(resized_projections, dim=1)
        x = self.fusion(x)
        x = self.cortex(x)
        x = self.dropout(x)
        
        x = self.up_block1(x) 
        x = self.up_block2(x) 
        
        return self.head(x)


class SegDINO(nn.Module):
    def __init__(self, num_classes, model_name="facebook/dinov3-vitb16-pretrain-lvd1689m", freeze_backbone=True, unfreeze_last_n=3, decoder_dim=256, feature_layers=[2, 5, 8, 11], pad_to_patch=True, patch_size=16):
        super().__init__()
        self.pad_to_patch = pad_to_patch
        self.feature_layers = feature_layers
        self.patch_size = patch_size 
        
        self.backbone = AutoModel.from_pretrained(
            model_name, output_hidden_states=True, trust_remote_code=True
        )
        self.config = self.backbone.config
        self.embed_dim = self.config.hidden_size 

        if freeze_backbone:
            for p in self.backbone.parameters(): p.requires_grad = False
            if unfreeze_last_n > 0:
                blocks = self._find_transformer_blocks(self.backbone)
                if blocks:
                    start_idx = max(0, len(blocks) - unfreeze_last_n)
                    for i in range(start_idx, len(blocks)):
                        for p in blocks[i].parameters(): p.requires_grad = True
                    self._unfreeze_final_norm(self.backbone)

        self.decoder = ExpressiveRobustDecoder(
            embed_dim=self.embed_dim, num_classes=num_classes, num_multiscale_inputs=len(feature_layers), decoder_dim=decoder_dim
        )
        
        self.intermediate_norms = nn.ModuleList([
            nn.GroupNorm(32, self.embed_dim) for _ in feature_layers
        ])

    def _find_transformer_blocks(self, model):
        if hasattr(model, "encoder") and hasattr(model.encoder, "layer"): return model.encoder.layer
        if hasattr(model, "layers"): return model.layers
        if hasattr(model, "blocks"): return model.blocks
        return []

    def _unfreeze_final_norm(self, model):
        candidates = ["layernorm", "norm", "ln_f", "final_layer_norm"]
        for name, module in model.named_modules():
            if any(c in name.lower() for c in candidates) and isinstance(module, (nn.LayerNorm, nn.GroupNorm)):
                for p in module.parameters(): p.requires_grad = True

    def _pad_to_multiple(self, x, multiple):
        B, C, H, W = x.shape
        pad_h = (multiple - (H % multiple)) % multiple
        pad_w = (multiple - (W % multiple)) % multiple
        pad = (0, pad_w, 0, pad_h)
        return F.pad(x, pad, mode='reflect'), pad

    def _unpad(self, x, pad):
        pad_w, pad_h = pad[1], pad[3]
        h_end = x.shape[2] - pad_h
        w_end = x.shape[3] - pad_w
        return x[:, :, 0:h_end, 0:w_end]

    def forward(self, x):
        B, C, H, W = x.shape
        
        if self.pad_to_patch:
            x_pad, pad = self._pad_to_multiple(x, self.patch_size)
        else:
            x_pad, pad = x, (0, 0, 0, 0)
            
        Hp, Wp = x_pad.shape[2], x_pad.shape[3]
        
        outputs = self.backbone(x_pad)
        hidden_states = outputs.hidden_states
        
        extracted = []
        num_patches = (Hp // self.patch_size) * (Wp // self.patch_size)
        for i, layer_idx in enumerate(self.feature_layers):
            state = hidden_states[layer_idx] 
            patch_tokens = state[:, -num_patches:, :] 
            feat = patch_tokens.transpose(1, 2).reshape(B, self.embed_dim, Hp//self.patch_size, Wp//self.patch_size)
            feat = self.intermediate_norms[i](feat)
            extracted.append(feat)
            
        logits = self.decoder(extracted, x_pad)
        
        logits = F.interpolate(logits, size=(Hp, Wp), mode='bilinear', align_corners=False)
        logits = self._unpad(logits, pad)
        logits = F.interpolate(logits, size=(H, W), mode='bilinear', align_corners=False)
        
        return logits

Architectural Context & Potential XNNPACK Triggers

While reviewing this architecture against the XNNPACK lowering pipeline, I noticed a few specific mechanisms in my network that might be failing to correctly initialize or map weights into XNNWeightsCache, resulting in the memcmp access violation.

To break this down from first principles, we must look at how XNNPACK accelerates inference. Historically, XNNPACK achieves CPU performance by rigorously repacking standard PyTorch weight tensors into highly optimized, hardware-specific layouts (e.g., interleaving channel data to perfectly fit AArch64 SIMD registers). Because this packing is computationally expensive, XNNPACK uses a global cache.
Think of the XNNWeightsCache like a cryptographic coat check: when the engine encounters a convolution layer, it hashes the raw, unpacked weight buffer. To handle hash collisions, it then executes a raw byte-by-byte memory comparison (memcmp) against the cached tensor. A SIGSEGV during this memcmp indicates that the Operating System's Memory Management Unit (MMU) trapped an illegal read—meaning XNNPACK is either being passed a null pointer, an unmapped memory page, or a size parameter that forces memcmp to read beyond the allocated boundary of the memory-mapped .pte file.

Looking at the model definition, there are three primary culprits that may be tricking the ExecuTorch AOT compiler into emitting malformed weight pointers:

1. Spectral Normalization "Ghosts" (The Most Likely Cause)

In the SpectralUpsampleBlock, the architecture wraps a 2D Convolution in torch.nn.utils.spectral_norm.
By design, spectral normalization removes the standard weight parameter from the module and replaces it with weight_orig, alongside two non-trainable vectors (weight_u and weight_v). During the forward pass, it dynamically computes the true weight using power iteration: $W_{SN} = \frac{W}{\sigma(W)}$.
Crucially, my export script does not call remove_spectral_norm() before tracing. This means the torch.export graph may be capturing the dynamic calculation of the weights rather than a static parameter. If the XNNPACK partitioner incorrectly categorizes this dynamic weight generation graph as a static convolution weight, it may pass an uninitialized intermediate buffer (or a symbolic shape pointer) to XNNWeightsCache::look_up_or_insert. When memcmp attempts to dereference this ghost tensor, it segfaults.

2. Aliased Memory in the Transformer Backbone

The facebook/dinov3... backbone generates feature maps that are subsequently sliced and reshaped:
patch_tokens = state[:, -num_patches:, :] followed by feat = patch_tokens.transpose(1, 2).reshape(...).
Under the hood, these operations create views (aliases) of the original hidden state buffer with highly specific memory strides. If ExecuTorch passes these strided representations down to the delegate without forcing contiguous memory allocation, memcmp—which assumes tightly packed, contiguous C-arrays—might walk entirely off the end of the mapped memory page on certain kernels.

3. Custom Parameter Data Injection

In LearnableDoG, weights are injected directly into the .data attribute: self.conv_narrow.weight.data = get_gaussian_kernel(...). If these custom-initialized weights are not tracked properly by PyTorch's parameter registry during the edge lowering phase, they could be resulting in unbacked allocations within the flatbuffer schema of the .pte file.

---