Aboud XNNStaticSlice op

[barbecacov]

🐛 Describe the bug

when i run my pte file by executorch_runner, it report

I 00:00:00.000624 executorch:executor_runner.cpp:175] Using method forward
I 00:00:00.000627 executorch:executor_runner.cpp:226] Setting up planned buffer 0, size 157344.
E 00:00:00.001586 executorch:XNNCompiler.cpp:1416] Failed to create static slice node 149 with code: xnn_status_invalid_parameter
E 00:00:00.001613 executorch:XNNPACKBackend.cpp:119] XNNCompiler::compileModel failed: 0x1
E 00:00:00.001617 executorch:method.cpp:109] Init failed for backend XnnpackBackend: 0x1
F 00:00:00.001621 executorch:executor_runner.cpp:246] In function main(), assert failed (method.ok()): Loading of method forward failed with status 0x1

details

this is XNNStaticSlice op in my pte file,

num_dims 4
offsets 0, 2, 0, 0
sizes 1, 4, 1, 240
input_id name: 41
flags 0

when i run my pte file by executorch_runner, it report

I 00:00:00.000624 executorch:executor_runner.cpp:175] Using method forward
I 00:00:00.000627 executorch:executor_runner.cpp:226] Setting up planned buffer 0, size 157344.
E 00:00:00.001586 executorch:XNNCompiler.cpp:1416] Failed to create static slice node 149 with code: xnn_status_invalid_parameter
E 00:00:00.001613 executorch:XNNPACKBackend.cpp:119] XNNCompiler::compileModel failed: 0x1
E 00:00:00.001617 executorch:method.cpp:109] Init failed for backend XnnpackBackend: 0x1
F 00:00:00.001621 executorch:executor_runner.cpp:246] In function main(), assert failed (method.ok()): Loading of method forward failed with status 0x1

and i check the executorch/backends/xnnpack/third-party/XNNPACK/src/subgraph/static-slice.c, found

    if (!xnn_datatype_is_byte_addressable(input_value->datatype)) {
      xnn_log_error(
        "failed to define %s operator with input ID #%" PRIu32 ": unsupported Value datatype %s (%d)",
        xnn_node_type_to_string(xnn_node_type_static_slice), input_id,
        xnn_datatype_to_string(input_value->datatype), input_value->datatype);
      return xnn_status_invalid_parameter;
    }


      if (strides[i] != 1) {
        xnn_log_error(
          "failed to define %s operator with input ID #%" PRIu32 ": Illegal stride value %" PRIi64 " in dimension #%zu",
          xnn_node_type_to_string(xnn_node_type_static_slice), input_id,
          strides[i], i);
        return xnn_status_invalid_parameter;
      }

and found in executorch/backends/xnnpack/third-party/XNNPACK/src/datatype.c

bool xnn_datatype_is_byte_addressable(enum xnn_datatype t) {
  switch (t) {
    case xnn_datatype_invalid:
    case xnn_datatype_qcint4:
    case xnn_datatype_qbint4:
    case xnn_datatype_pfp16:
    case xnn_datatype_pfp32:
    case xnn_datatype_qpint8:
      return false;
    case xnn_datatype_fp16:
    case xnn_datatype_bf16:
    case xnn_datatype_qint8:
    case xnn_datatype_pqint8:
    case xnn_datatype_quint8:
    case xnn_datatype_qint32:
    case xnn_datatype_qcint8:
    case xnn_datatype_qcint32:
    case xnn_datatype_qdint8:
    case xnn_datatype_qduint8:
    case xnn_datatype_int32:
    case xnn_datatype_fp32:
      return true;
  }
  XNN_UNREACHABLE;
  return false;
}

so How could i run the StaticSlice op correctly?

Thank you very much

cc @digantdesai @mcr229 @cbilgin

[barbecacov]

this is my python code to quantize model

---

from executorch.exir import (
    EdgeCompileConfig,
    ExecutorchBackendConfig,
    to_edge_transform_and_lower,
)
from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import (
    get_symmetric_quantization_config,
    XNNPACKQuantizer,
)

from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
from executorch.extension.export_util.utils import save_pte_program


def quantize(model, example_inputs):
    """This is the official recommended flow for quantization in pytorch 2.0 export"""
    # print(f"Original model: {model}")
    quantizer = XNNPACKQuantizer()
    # if we set is_per_channel to True, we also need to add out_variant of quantize_per_channel/dequantize_per_channel
    operator_config = get_symmetric_quantization_config(is_dynamic=False, is_per_channel=True)
    quantizer.set_global(operator_config)
    m = prepare_pt2e(model, quantizer)
    # calibration
    # m(*example_inputs)
    m = convert_pt2e(m)
    # print(f"Quantized model: {m}")
    # make sure we can export to flat buffer
    return m

def main():
  model = create_model()
  model.eval()

  ep = torch.export.export_for_training(model, dummy_args, strict=True)
  model = ep.module()
  if 1:
      model = quantize(model, dummy_args)
      ep = torch.export.export_for_training(model, dummy_args, strict=True)

  edge = to_edge_transform_and_lower(
      ep,
      partitioner=[XnnpackPartitioner()],
      compile_config=EdgeCompileConfig(_check_ir_validity=False,
                                       _skip_dim_order=True),
  )

  exec_prog = edge.to_executorch(config=ExecutorchBackendConfig(extract_delegate_segments=False))

  save_pte_program(exec_prog, "my_model_q8.pte", "./")

[barbecacov]

Can i do like this, i am not sure
@digantdesai
operator_config = get_symmetric_quantization_config(is_dynamic=False, is_per_channel=True)
slice_operator_config = get_symmetric_quantization_config(is_dynamic=True, is_per_channel=True)
quantizer.set_global(quantization_config=operator_config)
quantizer.set_operator_type(torch.ops.aten.slice, slice_operator_config)

[digantdesai]

Based on a quick read couple of suggestions.

If you enable logging, you will see xnn_log_error and narrow down to which one of the condition is failing.

Also see relevant unit-test comment about quantized Slice here -

executorch/backends/xnnpack/test/ops/test_slice_copy.py

Lines 143 to 146 in a8d7298

	# Note: Slice ends up as slice_copy later in the process, but during quantization,
	# it's still slice, which isn't supported by the XNNPACK quantizer.
	@unittest.skip("T156004676 - slice isn't propagated")
	def _test_qs8_slice_copy(self):

[barbecacov]

@digantdesai So how can i control the quantization schedule not to quantize slice op, in XNN quantization. how to set operator_config and quantizer in quantize function? thank you very much

[barbecacov]

@digantdesai Thank you for your suggestions, it works!!!

Then, i rebuild executorch with Debug, then run my pte file , the error message is more clearly,
Error in XNNPACK: failed to define Static Slice operator with input ID #109 and output ID #110: mismatching zero point quantization parameter across input (6) and output (-128) (xnn_subgraph_check_quantization_parameter_matches, executorch/backends/xnnpack/third-party/XNNPACK/src/subgraph/validation.c:191)

it means the zero_point of quantization parameter for Slice op doesnt match.

So i find out the

executorch/backends/xnnpack/quantizer/xnnpack_quantizer_utils.py

Line 1045 in fc435fa

def _is_share_obs_or_fq_op(op: Callable) -> bool:

and I add
torch.ops.aten.slice.Tensor, torch.ops.aten.select.int, torch.ops.aten.index_select.default,
to this function

Now it works

[digantdesai]

Please create a PR linking this issue, and a couple of tests.

[mcr229]

@barbecacov Thanks for the debugging and investigation here! If you put up a pr for the fix we would love to stamp and upstream!

[barbecacov]

@digantdesai @mcr229 here, https://github.com/pytorch/executorch/pull/12416
thank you very much